d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
56fb710f1b
freebsd-dev/sys/dev/cxgbe/t4_sge.c

6655 lines
173 KiB
C
Raw Normal View History

cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
/*-
sys/dev: further adoption of SPDX licensing ID tags. Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts.
2017-11-27 14:52:40 +00:00
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
*
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
* Copyright (c) 2011 Chelsio Communications, Inc.
* All rights reserved.
* Written by: Navdeep Parhar <np@FreeBSD.org>
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
#include "opt_inet.h"
cxgbe(4): support for IPv6 TSO and LRO. Submitted by: bz (this is a modified version of that patch)
2012-06-29 19:51:06 +00:00
#include "opt_inet6.h"
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
#include "opt_kern_tls.h"
cxgbe(4): Allocate offload Tx queues when a card has resources provisioned for NIC_ETHOFLD and the kernel has option RATELIMIT. It is possible to use the chip's offload queues for normal NIC Tx and not just TOE Tx. The difference is that these queues support out of order processing of work requests and have a per-"flowid" mechanism for tracking credits between the driver and hardware. This allows Tx for any number of flows bound to different rate limits to be submitted to a single Tx queue and the work requests for slow flows won't cause HOL blocking for the rest. Sponsored by: Chelsio Communications
2018-05-17 01:42:18 +00:00
#include "opt_ratelimit.h"
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
#include <sys/types.h>
Include necessary headers that now are available due to pollution via if_var.h. Sponsored by: Netflix Sponsored by: Nginx, Inc.
2013-10-28 07:29:16 +00:00
#include <sys/eventhandler.h>
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
#include <sys/mbuf.h>
#include <sys/socket.h>
#include <sys/kernel.h>
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
#include <sys/ktls.h>
Resume tx immediately in response to an SGE egress update from the hardware. MFC after: 1 week
2011-03-05 03:06:38 +00:00
#include <sys/malloc.h>
#include <sys/queue.h>
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
#include <sys/sbuf.h>
Resume tx immediately in response to an SGE egress update from the hardware. MFC after: 1 week
2011-03-05 03:06:38 +00:00
#include <sys/taskqueue.h>
Merge r254386 from user/np/cxl_tuning. Add an INET|INET6 check missing in said revision. r254386: Flush inactive LRO entries periodically.
2013-08-29 06:26:22 +00:00
#include <sys/time.h>
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
#include <sys/sglist.h>
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
#include <sys/sysctl.h>
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
#include <sys/smp.h>
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
#include <sys/socketvar.h>
cxgbe(4): Keep track of the clusters that have to be freed by the custom free routine (rxb_free) in the driver. Fail MOD_UNLOAD with EBUSY if any such cluster has been handed up to the kernel but hasn't been freed yet. This prevents a panic later when the cluster finally needs to be freed but rxb_free is gone from the kernel. MFC after: 1 week
2014-07-23 22:29:22 +00:00
#include <sys/counter.h>
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
#include <net/bpf.h>
#include <net/ethernet.h>
#include <net/if.h>
#include <net/if_vlan_var.h>
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
#include <net/if_vxlan.h>
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
#include <netinet/in.h>
#include <netinet/ip.h>
cxgbe(4): support for IPv6 TSO and LRO. Submitted by: bz (this is a modified version of that patch)
2012-06-29 19:51:06 +00:00
#include <netinet/ip6.h>
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
#include <netinet/tcp.h>
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#include <netinet/udp.h>
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
#include <machine/in_cksum.h>
Fix kernel build on amd64 after r256118, since the machine/md_var.h header is not implicitly included there. So include it explicitly. Approved by: re (delphij) Pointy hat to: dim MFC after: 3 days X-MFC-With: r256118
2013-10-07 22:30:03 +00:00
#include <machine/md_var.h>
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
#include <vm/vm.h>
#include <vm/pmap.h>
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
#ifdef DEV_NETMAP
#include <machine/bus.h>
#include <sys/selinfo.h>
#include <net/if_var.h>
#include <net/netmap.h>
#include <dev/netmap/netmap_kern.h>
#endif
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
#include "common/common.h"
#include "common/t4_regs.h"
#include "common/t4_regs_values.h"
#include "common/t4_msg.h"
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
#include "t4_l2t.h"
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
#include "t4_mp_ring.h"
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
#ifdef T4_PKT_TIMESTAMP
#define RX_COPY_THRESHOLD (MINCLSIZE - 8)
#else
#define RX_COPY_THRESHOLD MINCLSIZE
#endif
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
/* Internal mbuf flags stored in PH_loc.eight[1]. */
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
#define MC_NOMAP 0x01
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
#define MC_RAW_WR 0x02
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
#define MC_TLS 0x04
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
Convert some fixed parameters to tunables (with reasonable default values). - cong_drop specifies what to do on congestion: nothing, backpressure, or drop. - fl_pktshift specifies the padding before Ethernet payload. - fl_pad specifies the boundary upto which to pad Ethernet payload. - spg_len controls the length of the status page. MFC after: 2 weeks
2012-08-14 21:47:41 +00:00
/*
* Ethernet frames are DMA'd at this byte offset into the freelist buffer.
* 0-7 are valid values.
*/
cxgbe(4): Set fl_pktshift to 0 by default. Sponsored by: Chelsio Communications
2018-08-09 21:07:32 +00:00
static int fl_pktshift = 0;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, fl_pktshift, CTLFLAG_RDTUN, &fl_pktshift, 0,
"payload DMA offset in rx buffer (bytes)");
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Convert some fixed parameters to tunables (with reasonable default values). - cong_drop specifies what to do on congestion: nothing, backpressure, or drop. - fl_pktshift specifies the padding before Ethernet payload. - fl_pad specifies the boundary upto which to pad Ethernet payload. - spg_len controls the length of the status page. MFC after: 2 weeks
2012-08-14 21:47:41 +00:00
/*
* Pad ethernet payload up to this boundary.
* -1: driver should figure out a good value.
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
* 0: disable padding.
* Any power of 2 from 32 to 4096 (both inclusive) is also a valid value.
Convert some fixed parameters to tunables (with reasonable default values). - cong_drop specifies what to do on congestion: nothing, backpressure, or drop. - fl_pktshift specifies the padding before Ethernet payload. - fl_pad specifies the boundary upto which to pad Ethernet payload. - spg_len controls the length of the status page. MFC after: 2 weeks
2012-08-14 21:47:41 +00:00
*/
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
int fl_pad = -1;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, fl_pad, CTLFLAG_RDTUN, &fl_pad, 0,
"payload pad boundary (bytes)");
Convert some fixed parameters to tunables (with reasonable default values). - cong_drop specifies what to do on congestion: nothing, backpressure, or drop. - fl_pktshift specifies the padding before Ethernet payload. - fl_pad specifies the boundary upto which to pad Ethernet payload. - spg_len controls the length of the status page. MFC after: 2 weeks
2012-08-14 21:47:41 +00:00
/*
* Status page length.
* -1: driver should figure out a good value.
* 64 or 128 are the only other valid values.
*/
Mark spg_len and fl_pktshift static. These variables are no longer exported to t4_netmap.c after r296478.
2016-07-28 17:37:12 +00:00
static int spg_len = -1;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, spg_len, CTLFLAG_RDTUN, &spg_len, 0,
"status page size (bytes)");
Convert some fixed parameters to tunables (with reasonable default values). - cong_drop specifies what to do on congestion: nothing, backpressure, or drop. - fl_pktshift specifies the padding before Ethernet payload. - fl_pad specifies the boundary upto which to pad Ethernet payload. - spg_len controls the length of the status page. MFC after: 2 weeks
2012-08-14 21:47:41 +00:00
/*
* Congestion drops.
* -1: no congestion feedback (not recommended).
* 0: backpressure the channel instead of dropping packets right away.
* 1: no backpressure, drop packets for the congested queue immediately.
*/
static int cong_drop = 0;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, cong_drop, CTLFLAG_RDTUN, &cong_drop, 0,
"Congestion control for RX queues (0 = backpressure, 1 = drop");
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
/*
* Deliver multiple frames in the same free list buffer if they fit.
* -1: let the driver decide whether to enable buffer packing or not.
* 0: disable buffer packing.
* 1: enable buffer packing.
*/
static int buffer_packing = -1;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, buffer_packing, CTLFLAG_RDTUN, &buffer_packing,
0, "Enable buffer packing");
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
/*
* Start next frame in a packed buffer at this boundary.
* -1: driver should figure out a good value.
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
* T4: driver will ignore this and use the same value as fl_pad above.
* T5: 16, or a power of 2 from 64 to 4096 (both inclusive) is a valid value.
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
*/
static int fl_pack = -1;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, fl_pack, CTLFLAG_RDTUN, &fl_pack, 0,
"payload pack boundary (bytes)");
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
/*
* Largest rx cluster size that the driver is allowed to allocate.
*/
static int largest_rx_cluster = MJUM16BYTES;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, largest_rx_cluster, CTLFLAG_RDTUN,
&largest_rx_cluster, 0, "Largest rx cluster (bytes)");
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
/*
* Size of cluster allocation that's most likely to succeed. The driver will
* fall back to this size if it fails to allocate clusters larger than this.
*/
static int safest_rx_cluster = PAGE_SIZE;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, safest_rx_cluster, CTLFLAG_RDTUN,
&safest_rx_cluster, 0, "Safe rx cluster (bytes)");
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#ifdef RATELIMIT
/*
* Knob to control TCP timestamp rewriting, and the granularity of the tick used
* for rewriting. -1 and 0-3 are all valid values.
* -1: hardware should leave the TCP timestamps alone.
* 0: 1ms
* 1: 100us
* 2: 10us
* 3: 1us
*/
static int tsclk = -1;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, tsclk, CTLFLAG_RDTUN, &tsclk, 0,
"Control TCP timestamp rewriting when using pacing");
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
static int eo_max_backlog = 1024 * 1024;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, eo_max_backlog, CTLFLAG_RDTUN, &eo_max_backlog,
0, "Maximum backlog of ratelimited data per flow");
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#endif
cxgbe: Add a tunable to configure the SGE time scaler, which is available starting with T6. The values in the timer holdoff registers are multiplied by the scaling factor before use. dev.<nexus>.<n>.holdoff_timers shows the final values of the timers in microseconds. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-15 17:00:50 +00:00
/*
* The interrupt holdoff timers are multiplied by this value on T6+.
* 1 and 3-17 (both inclusive) are legal values.
*/
static int tscale = 1;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, tscale, CTLFLAG_RDTUN, &tscale, 0,
"Interrupt holdoff timer scale on T6+");
cxgbe: Add a tunable to configure the SGE time scaler, which is available starting with T6. The values in the timer holdoff registers are multiplied by the scaling factor before use. dev.<nexus>.<n>.holdoff_timers shows the final values of the timers in microseconds. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-15 17:00:50 +00:00
cxgbe: Add tunables to control the number of LRO entries and the number of rx mbufs that should be presorted before LRO. There is no change in default behavior. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-17 09:00:20 +00:00
/*
* Number of LRO entries in the lro_ctrl structure per rx queue.
*/
static int lro_entries = TCP_LRO_ENTRIES;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, lro_entries, CTLFLAG_RDTUN, &lro_entries, 0,
"Number of LRO entries per RX queue");
cxgbe: Add tunables to control the number of LRO entries and the number of rx mbufs that should be presorted before LRO. There is no change in default behavior. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-17 09:00:20 +00:00
/*
* This enables presorting of frames before they're fed into tcp_lro_rx.
*/
static int lro_mbufs = 0;
Add read-only sysctls for all tunables in the cxgbe(4) driver. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D18360
2018-11-27 22:02:54 +00:00
SYSCTL_INT(_hw_cxgbe, OID_AUTO, lro_mbufs, CTLFLAG_RDTUN, &lro_mbufs, 0,
"Enable presorting of LRO frames");
cxgbe: Add tunables to control the number of LRO entries and the number of rx mbufs that should be presorted before LRO. There is no change in default behavior. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-17 09:00:20 +00:00
cxgbe(4): add counters for mbuf pullups and defrags. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 03:06:36 +00:00
static counter_u64_t pullups;
SYSCTL_COUNTER_U64(_hw_cxgbe, OID_AUTO, pullups, CTLFLAG_RD, &pullups,
"Number of mbuf pullups performed");
static counter_u64_t defrags;
SYSCTL_COUNTER_U64(_hw_cxgbe, OID_AUTO, defrags, CTLFLAG_RD, &defrags,
"Number of mbuf defrags performed");
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static int service_iq(struct sge_iq *, int);
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
static int service_iq_fl(struct sge_iq *, int);
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
static struct mbuf *get_fl_payload(struct adapter *, struct sge_fl *, uint32_t);
cxgbe(4): Treat NIC rx as special and run its handler directly and not via the t4_cpl_handler dispatch table. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:01:35 +00:00
static int eth_rx(struct adapter *, struct sge_rxq *, const struct iq_desc *,
u_int);
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
static inline void init_iq(struct sge_iq *, struct adapter *, int, int, int);
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
static inline void init_fl(struct adapter *, struct sge_fl *, int, int, char *);
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
static inline void init_eq(struct adapter *, struct sge_eq *, int, int, uint8_t,
uint16_t, char *);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
static int alloc_ring(struct adapter *, size_t, bus_dma_tag_t *, bus_dmamap_t *,
bus_addr_t *, void **);
static int free_ring(struct adapter *, bus_dma_tag_t, bus_dmamap_t, bus_addr_t,
void *);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int alloc_iq_fl(struct vi_info *, struct sge_iq *, struct sge_fl *,
- Enable per-channel congestion notification. - Enable PCIe relaxed ordering for all egress queues and rx data buffers. MFC after: 3 days
2011-05-18 22:09:04 +00:00
int, int);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int free_iq_fl(struct vi_info *, struct sge_iq *, struct sge_fl *);
cxgbe(4): Reduce duplication by consolidating minor variations of the same code into a single routine. Sponsored by: Chelsio Communications
2017-12-29 02:30:21 +00:00
static void add_iq_sysctls(struct sysctl_ctx_list *, struct sysctl_oid *,
struct sge_iq *);
cxgbe(4): Make the location/length of all descriptor rings available in the sysctl MIB.
2016-09-23 20:03:28 +00:00
static void add_fl_sysctls(struct adapter *, struct sysctl_ctx_list *,
struct sysctl_oid *, struct sge_fl *);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static int alloc_fwq(struct adapter *);
static int free_fwq(struct adapter *);
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
static int alloc_ctrlq(struct adapter *, struct sge_wrq *, int,
struct sysctl_oid *);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int alloc_rxq(struct vi_info *, struct sge_rxq *, int, int,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sysctl_oid *);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int free_rxq(struct vi_info *, struct sge_rxq *);
- Updated TOE support in the kernel. - Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs. These are available as t3_tom and t4_tom modules that augment cxgb(4) and cxgbe(4) respectively. The cxgb/cxgbe drivers continue to work as usual with or without these extra features. - iWARP driver for Terminator 3 ASIC (kernel verbs). T4 iWARP in the works and will follow soon. Build-tested with make universe. 30s overview ============ What interfaces support TCP offload? Look for TOE4 and/or TOE6 in the capabilities of an interface: # ifconfig -m | grep TOE Enable/disable TCP offload on an interface (just like any other ifnet capability): # ifconfig cxgbe0 toe # ifconfig cxgbe0 -toe Which connections are offloaded? Look for toe4 and/or toe6 in the output of netstat and sockstat: # netstat -np tcp | grep toe # sockstat -46c | grep toe Reviewed by: bz, gnn Sponsored by: Chelsio communications. MFC after: ~3 months (after 9.1, and after ensuring MFC is feasible)
2012-06-19 07:34:13 +00:00
#ifdef TCP_OFFLOAD
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int alloc_ofld_rxq(struct vi_info *, struct sge_ofld_rxq *, int, int,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sysctl_oid *);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int free_ofld_rxq(struct vi_info *, struct sge_ofld_rxq *);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
#endif
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
#ifdef DEV_NETMAP
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int alloc_nm_rxq(struct vi_info *, struct sge_nm_rxq *, int, int,
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
struct sysctl_oid *);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int free_nm_rxq(struct vi_info *, struct sge_nm_rxq *);
static int alloc_nm_txq(struct vi_info *, struct sge_nm_txq *, int, int,
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
struct sysctl_oid *);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int free_nm_txq(struct vi_info *, struct sge_nm_txq *);
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
#endif
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static int ctrl_eq_alloc(struct adapter *, struct sge_eq *);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int eth_eq_alloc(struct adapter *, struct vi_info *, struct sge_eq *);
cxgbe(4): Allocate offload Tx queues when a card has resources provisioned for NIC_ETHOFLD and the kernel has option RATELIMIT. It is possible to use the chip's offload queues for normal NIC Tx and not just TOE Tx. The difference is that these queues support out of order processing of work requests and have a per-"flowid" mechanism for tracking credits between the driver and hardware. This allows Tx for any number of flows bound to different rate limits to be submitted to a single Tx queue and the work requests for slow flows won't cause HOL blocking for the rest. Sponsored by: Chelsio Communications
2018-05-17 01:42:18 +00:00
#if defined(TCP_OFFLOAD) || defined(RATELIMIT)
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int ofld_eq_alloc(struct adapter *, struct vi_info *, struct sge_eq *);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
#endif
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int alloc_eq(struct adapter *, struct vi_info *, struct sge_eq *);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static int free_eq(struct adapter *, struct sge_eq *);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int alloc_wrq(struct adapter *, struct vi_info *, struct sge_wrq *,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sysctl_oid *);
static int free_wrq(struct adapter *, struct sge_wrq *);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int alloc_txq(struct vi_info *, struct sge_txq *, int,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sysctl_oid *);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
static int free_txq(struct vi_info *, struct sge_txq *);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
static void oneseg_dma_callback(void *, bus_dma_segment_t *, int, int);
static inline void ring_fl_db(struct adapter *, struct sge_fl *);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static int refill_fl(struct adapter *, struct sge_fl *, int);
static void refill_sfl(void *);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
static int alloc_fl_sdesc(struct sge_fl *);
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
static void free_fl_sdesc(struct adapter *, struct sge_fl *);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
static int find_refill_source(struct adapter *, int, bool);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static void add_fl_to_sfl(struct adapter *, struct sge_fl *);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static inline void get_pkt_gl(struct mbuf *, struct sglist *);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
static inline u_int txpkt_len16(u_int, const u_int);
static inline u_int txpkt_vm_len16(u_int, const u_int);
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
static inline void calculate_mbuf_len16(struct mbuf *, bool);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static inline u_int txpkts0_len16(u_int);
static inline u_int txpkts1_len16(void);
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
static u_int write_raw_wr(struct sge_txq *, void *, struct mbuf *, u_int);
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
static u_int write_txpkt_wr(struct adapter *, struct sge_txq *, struct mbuf *,
u_int);
cxgbe(4): Use correct macro for header length with T6 ASICs. This affects the transmit of the VF driver only. Sponsored by: Chelsio Communications
2016-09-11 16:11:51 +00:00
static u_int write_txpkt_vm_wr(struct adapter *, struct sge_txq *,
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
struct mbuf *);
static int add_to_txpkts_vf(struct adapter *, struct sge_txq *, struct mbuf *,
int, bool *);
static int add_to_txpkts_pf(struct adapter *, struct sge_txq *, struct mbuf *,
int, bool *);
static u_int write_txpkts_wr(struct adapter *, struct sge_txq *);
static u_int write_txpkts_vm_wr(struct adapter *, struct sge_txq *);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static void write_gl_to_txd(struct sge_txq *, struct mbuf *, caddr_t *, int);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
static inline void copy_to_txd(struct sge_eq *, caddr_t, caddr_t *, int);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static inline void ring_eq_db(struct adapter *, struct sge_eq *, u_int);
static inline uint16_t read_hw_cidx(struct sge_eq *);
static inline u_int reclaimable_tx_desc(struct sge_eq *);
static inline u_int total_available_tx_desc(struct sge_eq *);
static u_int reclaim_tx_descs(struct sge_txq *, u_int);
static void tx_reclaim(void *, int);
static __be64 get_flit(struct sglist_seg *, int, int);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static int handle_sge_egr_update(struct sge_iq *, const struct rss_header *,
struct mbuf *);
Allow for a different handler for each type of firmware message. MFC after: 2 weeks
2012-08-16 18:31:50 +00:00
static int handle_fw_msg(struct sge_iq *, const struct rss_header *,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct mbuf *);
Install a handler for firmware work request error messages. If a driver sends an malformed or disallowed work request, the firmware responds with a work request error. Previously the driver treated this is as an unexpected message and panicked. Now it decodes the error message to aid in debugging. Reviewed by: np (older version) MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D6950
2016-07-22 21:52:07 +00:00
static int t4_handle_wrerr_rpl(struct adapter *, const __be64 *);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static void wrq_tx_drain(void *, int);
static void drain_wrq_wr_list(struct adapter *, struct sge_wrq *);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
static int sysctl_uint16(SYSCTL_HANDLER_ARGS);
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
static int sysctl_bufsizes(SYSCTL_HANDLER_ARGS);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#ifdef RATELIMIT
static inline u_int txpkt_eo_len16(u_int, u_int, u_int);
static int ethofld_fw4_ack(struct sge_iq *, const struct rss_header *,
struct mbuf *);
#endif
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
cxgbe(4): Keep track of the clusters that have to be freed by the custom free routine (rxb_free) in the driver. Fail MOD_UNLOAD with EBUSY if any such cluster has been handed up to the kernel but hasn't been freed yet. This prevents a panic later when the cluster finally needs to be freed but rxb_free is gone from the kernel. MFC after: 1 week
2014-07-23 22:29:22 +00:00
static counter_u64_t extfree_refs;
static counter_u64_t extfree_rels;
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
an_handler_t t4_an_handler;
fw_msg_handler_t t4_fw_msg_handler[NUM_FW6_TYPES];
cpl_handler_t t4_cpl_handler[NUM_CPL_CMDS];
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
cpl_handler_t set_tcb_rpl_handlers[NUM_CPL_COOKIES];
cpl_handler_t l2t_write_rpl_handlers[NUM_CPL_COOKIES];
cxgbe(4): Convert ACT_OPEN_RPL to a shared CPL. Reserve 3b in the 14b atid to identify the owner and use it to dispatch the CPL. This allows all CPLs that use an atid to be used as shared CPLs, although ACT_OPEN_RPL is the only one being converted in this revision. Sponsored by: Chelsio Communications
2018-04-30 21:47:30 +00:00
cpl_handler_t act_open_rpl_handlers[NUM_CPL_COOKIES];
cxgbe(4): Add support for hash filters. These filters reside in the card's memory instead of its TCAM and can be configured via a new "hashfilter" subcommand in cxgbetool. Hash and normal TCAM filters can be used together. The hardware does an exact-match of packet fields for hash filters, unlike the masked match performed for TCAM filters. Any T5/T6 card with memory can support at least half a million hash filters. The sample config file with the driver configures 512K of these, it is possible to double this to 1 million+ in some cases. The chip does an exact-match of fields of incoming datagrams with hash filters and performs the action configured for the filter if it matches. The fields to match are specified in a "filter mask" in the firmware config file. The filter mask always includes the 5-tuple (sip, dip, sport, dport, ipproto). It can, optionally, also include any subset of the filter mode (see filterMode and filterMask in the firmware config file). For example: filterMode = fragmentation, mpshittype, protocol, vlan, port, fcoe filterMask = protocol, port, vlan Exact values of the 5-tuple, the physical port, and VLAN tag would have to be provided while setting up a hash filter with the chip configuration above. Hash filters support all actions supported by TCAM filters. A packet that hits a hash filter can be dropped, let through (with optional steering to a specific queue or RSS region), switched out of another port (with optional L2 rewrite of DMAC, SMAC, VLAN tag), or get NAT'ed. (Support for some of these will show up in the driver in a follow-up commit very shortly). Sponsored by: Chelsio Communications
2018-05-09 04:09:49 +00:00
cpl_handler_t abort_rpl_rss_handlers[NUM_CPL_COOKIES];
cxgbe(4): Make FW4_ACK a shared CPL. ETHOFLD in the base driver will use it for per-flow rate limiting. Sponsored by: Chelsio Communications
2018-05-24 08:21:43 +00:00
cpl_handler_t fw4_ack_handlers[NUM_CPL_COOKIES];
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
void
t4_register_an_handler(an_handler_t h)
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
{
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
uintptr_t *loc;
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
MPASS(h == NULL || t4_an_handler == NULL);
loc = (uintptr_t *)&t4_an_handler;
atomic_store_rel_ptr(loc, (uintptr_t)h);
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
}
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
void
t4_register_fw_msg_handler(int type, fw_msg_handler_t h)
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
{
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
uintptr_t *loc;
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
MPASS(type < nitems(t4_fw_msg_handler));
MPASS(h == NULL || t4_fw_msg_handler[type] == NULL);
/*
* These are dispatched by the handler for FW{4|6}_CPL_MSG using the CPL
* handler dispatch table. Reject any attempt to install a handler for
* this subtype.
*/
MPASS(type != FW_TYPE_RSSCPL);
MPASS(type != FW6_TYPE_RSSCPL);
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
loc = (uintptr_t *)&t4_fw_msg_handler[type];
atomic_store_rel_ptr(loc, (uintptr_t)h);
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
}
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
void
t4_register_cpl_handler(int opcode, cpl_handler_t h)
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
{
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
uintptr_t *loc;
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
MPASS(opcode < nitems(t4_cpl_handler));
MPASS(h == NULL || t4_cpl_handler[opcode] == NULL);
loc = (uintptr_t *)&t4_cpl_handler[opcode];
atomic_store_rel_ptr(loc, (uintptr_t)h);
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
}
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
static int
set_tcb_rpl_handler(struct sge_iq *iq, const struct rss_header *rss,
struct mbuf *m)
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
{
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
const struct cpl_set_tcb_rpl *cpl = (const void *)(rss + 1);
u_int tid;
int cookie;
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
MPASS(m == NULL);
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
tid = GET_TID(cpl);
cxgbe(4): Add support for high priority filters on T6+. They have their own region in the TCAM starting with T6, unlike previous chips where they were in the same region as normal filters. These filters "hit" before anything else in the LE's lookup. The exact order is: a) High priority filters b) TOE's active region (TCAM and/or hash) c) Servers (TOE hw listeners) d) Normal filters MFC after: 1 week Sponsored by: Chelsio Communications
2018-08-09 14:19:47 +00:00
if (is_hpftid(iq->adapter, tid) || is_ftid(iq->adapter, tid)) {
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
/*
* The return code for filter-write is put in the CPL cookie so
* we have to rely on the hardware tid (is_ftid) to determine
* that this is a response to a filter.
*/
cookie = CPL_COOKIE_FILTER;
} else {
cookie = G_COOKIE(cpl->cookie);
}
MPASS(cookie > CPL_COOKIE_RESERVED);
MPASS(cookie < nitems(set_tcb_rpl_handlers));
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
return (set_tcb_rpl_handlers[cookie](iq, rss, m));
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
}
static int
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
l2t_write_rpl_handler(struct sge_iq *iq, const struct rss_header *rss,
struct mbuf *m)
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
{
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
const struct cpl_l2t_write_rpl *rpl = (const void *)(rss + 1);
unsigned int cookie;
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
MPASS(m == NULL);
cookie = GET_TID(rpl) & F_SYNC_WR ? CPL_COOKIE_TOM : CPL_COOKIE_FILTER;
return (l2t_write_rpl_handlers[cookie](iq, rss, m));
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
}
cxgbe(4): Convert ACT_OPEN_RPL to a shared CPL. Reserve 3b in the 14b atid to identify the owner and use it to dispatch the CPL. This allows all CPLs that use an atid to be used as shared CPLs, although ACT_OPEN_RPL is the only one being converted in this revision. Sponsored by: Chelsio Communications
2018-04-30 21:47:30 +00:00
static int
act_open_rpl_handler(struct sge_iq *iq, const struct rss_header *rss,
struct mbuf *m)
{
const struct cpl_act_open_rpl *cpl = (const void *)(rss + 1);
u_int cookie = G_TID_COOKIE(G_AOPEN_ATID(be32toh(cpl->atid_status)));
MPASS(m == NULL);
MPASS(cookie != CPL_COOKIE_RESERVED);
return (act_open_rpl_handlers[cookie](iq, rss, m));
}
cxgbe(4): Add support for hash filters. These filters reside in the card's memory instead of its TCAM and can be configured via a new "hashfilter" subcommand in cxgbetool. Hash and normal TCAM filters can be used together. The hardware does an exact-match of packet fields for hash filters, unlike the masked match performed for TCAM filters. Any T5/T6 card with memory can support at least half a million hash filters. The sample config file with the driver configures 512K of these, it is possible to double this to 1 million+ in some cases. The chip does an exact-match of fields of incoming datagrams with hash filters and performs the action configured for the filter if it matches. The fields to match are specified in a "filter mask" in the firmware config file. The filter mask always includes the 5-tuple (sip, dip, sport, dport, ipproto). It can, optionally, also include any subset of the filter mode (see filterMode and filterMask in the firmware config file). For example: filterMode = fragmentation, mpshittype, protocol, vlan, port, fcoe filterMask = protocol, port, vlan Exact values of the 5-tuple, the physical port, and VLAN tag would have to be provided while setting up a hash filter with the chip configuration above. Hash filters support all actions supported by TCAM filters. A packet that hits a hash filter can be dropped, let through (with optional steering to a specific queue or RSS region), switched out of another port (with optional L2 rewrite of DMAC, SMAC, VLAN tag), or get NAT'ed. (Support for some of these will show up in the driver in a follow-up commit very shortly). Sponsored by: Chelsio Communications
2018-05-09 04:09:49 +00:00
static int
abort_rpl_rss_handler(struct sge_iq *iq, const struct rss_header *rss,
struct mbuf *m)
{
struct adapter *sc = iq->adapter;
u_int cookie;
MPASS(m == NULL);
if (is_hashfilter(sc))
cookie = CPL_COOKIE_HASHFILTER;
else
cookie = CPL_COOKIE_TOM;
return (abort_rpl_rss_handlers[cookie](iq, rss, m));
}
cxgbe(4): Make FW4_ACK a shared CPL. ETHOFLD in the base driver will use it for per-flow rate limiting. Sponsored by: Chelsio Communications
2018-05-24 08:21:43 +00:00
static int
fw4_ack_handler(struct sge_iq *iq, const struct rss_header *rss, struct mbuf *m)
{
struct adapter *sc = iq->adapter;
const struct cpl_fw4_ack *cpl = (const void *)(rss + 1);
unsigned int tid = G_CPL_FW4_ACK_FLOWID(be32toh(OPCODE_TID(cpl)));
u_int cookie;
MPASS(m == NULL);
if (is_etid(sc, tid))
cookie = CPL_COOKIE_ETHOFLD;
else
cookie = CPL_COOKIE_TOM;
return (fw4_ack_handlers[cookie](iq, rss, m));
}
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
static void
t4_init_shared_cpl_handlers(void)
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
{
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
t4_register_cpl_handler(CPL_SET_TCB_RPL, set_tcb_rpl_handler);
t4_register_cpl_handler(CPL_L2T_WRITE_RPL, l2t_write_rpl_handler);
cxgbe(4): Convert ACT_OPEN_RPL to a shared CPL. Reserve 3b in the 14b atid to identify the owner and use it to dispatch the CPL. This allows all CPLs that use an atid to be used as shared CPLs, although ACT_OPEN_RPL is the only one being converted in this revision. Sponsored by: Chelsio Communications
2018-04-30 21:47:30 +00:00
t4_register_cpl_handler(CPL_ACT_OPEN_RPL, act_open_rpl_handler);
cxgbe(4): Add support for hash filters. These filters reside in the card's memory instead of its TCAM and can be configured via a new "hashfilter" subcommand in cxgbetool. Hash and normal TCAM filters can be used together. The hardware does an exact-match of packet fields for hash filters, unlike the masked match performed for TCAM filters. Any T5/T6 card with memory can support at least half a million hash filters. The sample config file with the driver configures 512K of these, it is possible to double this to 1 million+ in some cases. The chip does an exact-match of fields of incoming datagrams with hash filters and performs the action configured for the filter if it matches. The fields to match are specified in a "filter mask" in the firmware config file. The filter mask always includes the 5-tuple (sip, dip, sport, dport, ipproto). It can, optionally, also include any subset of the filter mode (see filterMode and filterMask in the firmware config file). For example: filterMode = fragmentation, mpshittype, protocol, vlan, port, fcoe filterMask = protocol, port, vlan Exact values of the 5-tuple, the physical port, and VLAN tag would have to be provided while setting up a hash filter with the chip configuration above. Hash filters support all actions supported by TCAM filters. A packet that hits a hash filter can be dropped, let through (with optional steering to a specific queue or RSS region), switched out of another port (with optional L2 rewrite of DMAC, SMAC, VLAN tag), or get NAT'ed. (Support for some of these will show up in the driver in a follow-up commit very shortly). Sponsored by: Chelsio Communications
2018-05-09 04:09:49 +00:00
t4_register_cpl_handler(CPL_ABORT_RPL_RSS, abort_rpl_rss_handler);
cxgbe(4): Make FW4_ACK a shared CPL. ETHOFLD in the base driver will use it for per-flow rate limiting. Sponsored by: Chelsio Communications
2018-05-24 08:21:43 +00:00
t4_register_cpl_handler(CPL_FW4_ACK, fw4_ack_handler);
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
}
void
t4_register_shared_cpl_handler(int opcode, cpl_handler_t h, int cookie)
{
uintptr_t *loc;
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
MPASS(opcode < nitems(t4_cpl_handler));
MPASS(cookie > CPL_COOKIE_RESERVED);
MPASS(cookie < NUM_CPL_COOKIES);
MPASS(t4_cpl_handler[opcode] != NULL);
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
switch (opcode) {
case CPL_SET_TCB_RPL:
loc = (uintptr_t *)&set_tcb_rpl_handlers[cookie];
break;
case CPL_L2T_WRITE_RPL:
loc = (uintptr_t *)&l2t_write_rpl_handlers[cookie];
break;
cxgbe(4): Convert ACT_OPEN_RPL to a shared CPL. Reserve 3b in the 14b atid to identify the owner and use it to dispatch the CPL. This allows all CPLs that use an atid to be used as shared CPLs, although ACT_OPEN_RPL is the only one being converted in this revision. Sponsored by: Chelsio Communications
2018-04-30 21:47:30 +00:00
case CPL_ACT_OPEN_RPL:
loc = (uintptr_t *)&act_open_rpl_handlers[cookie];
break;
cxgbe(4): Add support for hash filters. These filters reside in the card's memory instead of its TCAM and can be configured via a new "hashfilter" subcommand in cxgbetool. Hash and normal TCAM filters can be used together. The hardware does an exact-match of packet fields for hash filters, unlike the masked match performed for TCAM filters. Any T5/T6 card with memory can support at least half a million hash filters. The sample config file with the driver configures 512K of these, it is possible to double this to 1 million+ in some cases. The chip does an exact-match of fields of incoming datagrams with hash filters and performs the action configured for the filter if it matches. The fields to match are specified in a "filter mask" in the firmware config file. The filter mask always includes the 5-tuple (sip, dip, sport, dport, ipproto). It can, optionally, also include any subset of the filter mode (see filterMode and filterMask in the firmware config file). For example: filterMode = fragmentation, mpshittype, protocol, vlan, port, fcoe filterMask = protocol, port, vlan Exact values of the 5-tuple, the physical port, and VLAN tag would have to be provided while setting up a hash filter with the chip configuration above. Hash filters support all actions supported by TCAM filters. A packet that hits a hash filter can be dropped, let through (with optional steering to a specific queue or RSS region), switched out of another port (with optional L2 rewrite of DMAC, SMAC, VLAN tag), or get NAT'ed. (Support for some of these will show up in the driver in a follow-up commit very shortly). Sponsored by: Chelsio Communications
2018-05-09 04:09:49 +00:00
case CPL_ABORT_RPL_RSS:
loc = (uintptr_t *)&abort_rpl_rss_handlers[cookie];
break;
cxgbe(4): Make FW4_ACK a shared CPL. ETHOFLD in the base driver will use it for per-flow rate limiting. Sponsored by: Chelsio Communications
2018-05-24 08:21:43 +00:00
case CPL_FW4_ACK:
loc = (uintptr_t *)&fw4_ack_handlers[cookie];
break;
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
default:
MPASS(0);
return;
}
MPASS(h == NULL || *loc == (uintptr_t)NULL);
atomic_store_rel_ptr(loc, (uintptr_t)h);
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
}
cxgbe shouldn't directly know of the UMA zones where network buffers come from. MFC after: 1 week
2011-03-08 03:04:07 +00:00
/*
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
* Called on MOD_LOAD. Validates and calculates the SGE tunables.
cxgbe shouldn't directly know of the UMA zones where network buffers come from. MFC after: 1 week
2011-03-08 03:04:07 +00:00
*/
void
t4_sge_modload(void)
{
Better way to determine the status page length and rx pad boundary.
2012-06-23 22:12:27 +00:00
Convert some fixed parameters to tunables (with reasonable default values). - cong_drop specifies what to do on congestion: nothing, backpressure, or drop. - fl_pktshift specifies the padding before Ethernet payload. - fl_pad specifies the boundary upto which to pad Ethernet payload. - spg_len controls the length of the status page. MFC after: 2 weeks
2012-08-14 21:47:41 +00:00
if (fl_pktshift < 0 || fl_pktshift > 7) {
printf("Invalid hw.cxgbe.fl_pktshift value (%d),"
cxgbe(4): Set fl_pktshift to 0 by default. Sponsored by: Chelsio Communications
2018-08-09 21:07:32 +00:00
" using 0 instead.\n", fl_pktshift);
fl_pktshift = 0;
Convert some fixed parameters to tunables (with reasonable default values). - cong_drop specifies what to do on congestion: nothing, backpressure, or drop. - fl_pktshift specifies the padding before Ethernet payload. - fl_pad specifies the boundary upto which to pad Ethernet payload. - spg_len controls the length of the status page. MFC after: 2 weeks
2012-08-14 21:47:41 +00:00
}
if (spg_len != 64 && spg_len != 128) {
int len;
#if defined(__i386__) || defined(__amd64__)
len = cpu_clflush_line_size > 64 ? 128 : 64;
#else
len = 64;
#endif
if (spg_len != -1) {
printf("Invalid hw.cxgbe.spg_len value (%d),"
" using %d instead.\n", spg_len, len);
}
spg_len = len;
}
if (cong_drop < -1 || cong_drop > 1) {
printf("Invalid hw.cxgbe.cong_drop value (%d),"
" using 0 instead.\n", cong_drop);
cong_drop = 0;
}
cxgbe(4): Keep track of the clusters that have to be freed by the custom free routine (rxb_free) in the driver. Fail MOD_UNLOAD with EBUSY if any such cluster has been handed up to the kernel but hasn't been freed yet. This prevents a panic later when the cluster finally needs to be freed but rxb_free is gone from the kernel. MFC after: 1 week
2014-07-23 22:29:22 +00:00
cxgbe: Add a tunable to configure the SGE time scaler, which is available starting with T6. The values in the timer holdoff registers are multiplied by the scaling factor before use. dev.<nexus>.<n>.holdoff_timers shows the final values of the timers in microseconds. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-15 17:00:50 +00:00
if (tscale != 1 && (tscale < 3 || tscale > 17)) {
printf("Invalid hw.cxgbe.tscale value (%d),"
" using 1 instead.\n", tscale);
tscale = 1;
}
cxgbe(4): validate largest_rx_cluster and safest_rx_cluster. These tunables can only be set to a valid cluster size (2K, 4K, 9K, or 16K) as documented in the man page. Anything else could lead to a panic on interface up. Reported by: mav@ MFC after: 1 week Sponsored by: Chelsio Communications
2020-10-02 05:59:55 +00:00
if (largest_rx_cluster != MCLBYTES &&
#if MJUMPAGESIZE != MCLBYTES
largest_rx_cluster != MJUMPAGESIZE &&
#endif
largest_rx_cluster != MJUM9BYTES &&
largest_rx_cluster != MJUM16BYTES) {
printf("Invalid hw.cxgbe.largest_rx_cluster value (%d),"
" using %d instead.\n", largest_rx_cluster, MJUM16BYTES);
largest_rx_cluster = MJUM16BYTES;
}
if (safest_rx_cluster != MCLBYTES &&
#if MJUMPAGESIZE != MCLBYTES
safest_rx_cluster != MJUMPAGESIZE &&
#endif
safest_rx_cluster != MJUM9BYTES &&
safest_rx_cluster != MJUM16BYTES) {
printf("Invalid hw.cxgbe.safest_rx_cluster value (%d),"
" using %d instead.\n", safest_rx_cluster, MJUMPAGESIZE);
safest_rx_cluster = MJUMPAGESIZE;
}
cxgbe(4): Keep track of the clusters that have to be freed by the custom free routine (rxb_free) in the driver. Fail MOD_UNLOAD with EBUSY if any such cluster has been handed up to the kernel but hasn't been freed yet. This prevents a panic later when the cluster finally needs to be freed but rxb_free is gone from the kernel. MFC after: 1 week
2014-07-23 22:29:22 +00:00
extfree_refs = counter_u64_alloc(M_WAITOK);
extfree_rels = counter_u64_alloc(M_WAITOK);
cxgbe(4): add counters for mbuf pullups and defrags. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 03:06:36 +00:00
pullups = counter_u64_alloc(M_WAITOK);
defrags = counter_u64_alloc(M_WAITOK);
cxgbe(4): Keep track of the clusters that have to be freed by the custom free routine (rxb_free) in the driver. Fail MOD_UNLOAD with EBUSY if any such cluster has been handed up to the kernel but hasn't been freed yet. This prevents a panic later when the cluster finally needs to be freed but rxb_free is gone from the kernel. MFC after: 1 week
2014-07-23 22:29:22 +00:00
counter_u64_zero(extfree_refs);
counter_u64_zero(extfree_rels);
cxgbe(4): add counters for mbuf pullups and defrags. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 03:06:36 +00:00
counter_u64_zero(pullups);
counter_u64_zero(defrags);
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
t4_init_shared_cpl_handlers();
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
t4_register_cpl_handler(CPL_FW4_MSG, handle_fw_msg);
t4_register_cpl_handler(CPL_FW6_MSG, handle_fw_msg);
t4_register_cpl_handler(CPL_SGE_EGR_UPDATE, handle_sge_egr_update);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#ifdef RATELIMIT
t4_register_shared_cpl_handler(CPL_FW4_ACK, ethofld_fw4_ack,
CPL_COOKIE_ETHOFLD);
#endif
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
t4_register_fw_msg_handler(FW6_TYPE_CMD_RPL, t4_handle_fw_rpl);
Install a handler for firmware work request error messages. If a driver sends an malformed or disallowed work request, the firmware responds with a work request error. Previously the driver treated this is as an unexpected message and panicked. Now it decodes the error message to aid in debugging. Reviewed by: np (older version) MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D6950
2016-07-22 21:52:07 +00:00
t4_register_fw_msg_handler(FW6_TYPE_WRERR_RPL, t4_handle_wrerr_rpl);
cxgbe(4): Keep track of the clusters that have to be freed by the custom free routine (rxb_free) in the driver. Fail MOD_UNLOAD with EBUSY if any such cluster has been handed up to the kernel but hasn't been freed yet. This prevents a panic later when the cluster finally needs to be freed but rxb_free is gone from the kernel. MFC after: 1 week
2014-07-23 22:29:22 +00:00
}
void
t4_sge_modunload(void)
{
counter_u64_free(extfree_refs);
counter_u64_free(extfree_rels);
cxgbe(4): add counters for mbuf pullups and defrags. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 03:06:36 +00:00
counter_u64_free(pullups);
counter_u64_free(defrags);
cxgbe(4): Keep track of the clusters that have to be freed by the custom free routine (rxb_free) in the driver. Fail MOD_UNLOAD with EBUSY if any such cluster has been handed up to the kernel but hasn't been freed yet. This prevents a panic later when the cluster finally needs to be freed but rxb_free is gone from the kernel. MFC after: 1 week
2014-07-23 22:29:22 +00:00
}
uint64_t
t4_sge_extfree_refs(void)
{
uint64_t refs, rels;
rels = counter_u64_fetch(extfree_rels);
refs = counter_u64_fetch(extfree_refs);
return (refs - rels);
cxgbe shouldn't directly know of the UMA zones where network buffers come from. MFC after: 1 week
2011-03-08 03:04:07 +00:00
}
cxgbe(4): Do not use pack boundary > 512B unless it is explicitly requested. This is a tradeoff between PCIe efficiency during large packet rx and packing efficiency during small packet rx. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-03 23:30:39 +00:00
/* max 4096 */
#define MAX_PACK_BOUNDARY 512
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
static inline void
setup_pad_and_pack_boundaries(struct adapter *sc)
{
uint32_t v, m;
cxgbe(4): Update the pad_boundary calculation for T6, which has a different range of boundaries. Sponsored by: Chelsio Communications
2016-09-11 17:22:54 +00:00
int pad, pack, pad_shift;
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
cxgbe(4): Update the pad_boundary calculation for T6, which has a different range of boundaries. Sponsored by: Chelsio Communications
2016-09-11 17:22:54 +00:00
pad_shift = chip_id(sc) > CHELSIO_T5 ? X_T6_INGPADBOUNDARY_SHIFT :
X_INGPADBOUNDARY_SHIFT;
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
pad = fl_pad;
cxgbe(4): Update the pad_boundary calculation for T6, which has a different range of boundaries. Sponsored by: Chelsio Communications
2016-09-11 17:22:54 +00:00
if (fl_pad < (1 << pad_shift) ||
fl_pad > (1 << (pad_shift + M_INGPADBOUNDARY)) ||
!powerof2(fl_pad)) {
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
/*
* If there is any chance that we might use buffer packing and
* the chip is a T4, then pick 64 as the pad/pack boundary. Set
cxgbe(4): Update the pad_boundary calculation for T6, which has a different range of boundaries. Sponsored by: Chelsio Communications
2016-09-11 17:22:54 +00:00
* it to the minimum allowed in all other cases.
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
*/
cxgbe(4): Update the pad_boundary calculation for T6, which has a different range of boundaries. Sponsored by: Chelsio Communications
2016-09-11 17:22:54 +00:00
pad = is_t4(sc) && buffer_packing ? 64 : 1 << pad_shift;
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
/*
* For fl_pad = 0 we'll still write a reasonable value to the
* register but all the freelists will opt out of padding.
* We'll complain here only if the user tried to set it to a
* value greater than 0 that was invalid.
*/
if (fl_pad > 0) {
device_printf(sc->dev, "Invalid hw.cxgbe.fl_pad value"
" (%d), using %d instead.\n", fl_pad, pad);
}
}
m = V_INGPADBOUNDARY(M_INGPADBOUNDARY);
cxgbe(4): Update the pad_boundary calculation for T6, which has a different range of boundaries. Sponsored by: Chelsio Communications
2016-09-11 17:22:54 +00:00
v = V_INGPADBOUNDARY(ilog2(pad) - pad_shift);
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
t4_set_reg_field(sc, A_SGE_CONTROL, m, v);
if (is_t4(sc)) {
if (fl_pack != -1 && fl_pack != pad) {
/* Complain but carry on. */
device_printf(sc->dev, "hw.cxgbe.fl_pack (%d) ignored,"
" using %d instead.\n", fl_pack, pad);
}
return;
}
pack = fl_pack;
if (fl_pack < 16 || fl_pack == 32 || fl_pack > 4096 ||
!powerof2(fl_pack)) {
cxgbe(4): Do not use pack boundary > 512B unless it is explicitly requested. This is a tradeoff between PCIe efficiency during large packet rx and packing efficiency during small packet rx. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-03 23:30:39 +00:00
if (sc->params.pci.mps > MAX_PACK_BOUNDARY)
pack = MAX_PACK_BOUNDARY;
else
pack = max(sc->params.pci.mps, CACHE_LINE_SIZE);
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
MPASS(powerof2(pack));
if (pack < 16)
pack = 16;
if (pack == 32)
pack = 64;
if (pack > 4096)
pack = 4096;
if (fl_pack != -1) {
device_printf(sc->dev, "Invalid hw.cxgbe.fl_pack value"
" (%d), using %d instead.\n", fl_pack, pack);
}
}
m = V_INGPACKBOUNDARY(M_INGPACKBOUNDARY);
if (pack == 16)
v = V_INGPACKBOUNDARY(0);
else
v = V_INGPACKBOUNDARY(ilog2(pack) - 5);
MPASS(!is_t4(sc)); /* T4 doesn't have SGE_CONTROL2 */
t4_set_reg_field(sc, A_SGE_CONTROL2, m, v);
}
Auto-reduce the holdoff timers that are greater than the maximum value allowed by the hardware. MFC after: 3 days
2013-04-11 22:46:39 +00:00
/*
* adap->params.vpd.cclk must be set up before this is called.
*/
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
void
t4_tweak_chip_settings(struct adapter *sc)
{
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
int i, reg;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
uint32_t v, m;
int intr_timer[SGE_NTIMERS] = {1, 5, 10, 50, 100, 200};
Auto-reduce the holdoff timers that are greater than the maximum value allowed by the hardware. MFC after: 3 days
2013-04-11 22:46:39 +00:00
int timer_max = M_TIMERVALUE0 * 1000 / sc->params.vpd.cclk;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
int intr_pktcount[SGE_NCOUNTERS] = {1, 8, 16, 32}; /* 63 max */
uint16_t indsz = min(RX_COPY_THRESHOLD - 1, M_INDICATESIZE);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
static int sw_buf_sizes[] = {
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
MCLBYTES,
#if MJUMPAGESIZE != MCLBYTES
MJUMPAGESIZE,
#endif
MJUM9BYTES,
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
MJUM16BYTES
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
};
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
KASSERT(sc->flags & MASTER_PF,
("%s: trying to change chip settings when not master.", __func__));
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
m = V_PKTSHIFT(M_PKTSHIFT) | F_RXPKTCPLMODE | F_EGRSTATUSPAGESIZE;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
v = V_PKTSHIFT(fl_pktshift) | F_RXPKTCPLMODE |
Better way to determine the status page length and rx pad boundary.
2012-06-23 22:12:27 +00:00
V_EGRSTATUSPAGESIZE(spg_len == 128);
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
t4_set_reg_field(sc, A_SGE_CONTROL, m, v);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
setup_pad_and_pack_boundaries(sc);
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
v = V_HOSTPAGESIZEPF0(PAGE_SHIFT - 10) |
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
V_HOSTPAGESIZEPF1(PAGE_SHIFT - 10) |
V_HOSTPAGESIZEPF2(PAGE_SHIFT - 10) |
V_HOSTPAGESIZEPF3(PAGE_SHIFT - 10) |
V_HOSTPAGESIZEPF4(PAGE_SHIFT - 10) |
V_HOSTPAGESIZEPF5(PAGE_SHIFT - 10) |
V_HOSTPAGESIZEPF6(PAGE_SHIFT - 10) |
V_HOSTPAGESIZEPF7(PAGE_SHIFT - 10);
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
t4_write_reg(sc, A_SGE_HOST_PAGE_SIZE, v);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): Get Linux cxgb4vf working in bhyve VMs with VFs passed through. cxgb4vf doesn't own the buffer size list but still expects the first two entries to be 4K and some power of 2 respectively. The BSD cxgbe doesn't care where its preferred buffer sizes are as long as they're in the list somewhere, so just move its entries towards the end as a workaround. MFC after: 1 month Sponsored by: Chelsio Communicatons
2018-12-06 21:33:08 +00:00
t4_write_reg(sc, A_SGE_FL_BUFFER_SIZE0, 4096);
t4_write_reg(sc, A_SGE_FL_BUFFER_SIZE1, 65536);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
reg = A_SGE_FL_BUFFER_SIZE2;
for (i = 0; i < nitems(sw_buf_sizes); i++) {
MPASS(reg <= A_SGE_FL_BUFFER_SIZE15);
t4_write_reg(sc, reg, sw_buf_sizes[i]);
reg += 4;
MPASS(reg <= A_SGE_FL_BUFFER_SIZE15);
t4_write_reg(sc, reg, sw_buf_sizes[i] - CL_METADATA_SIZE);
reg += 4;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
v = V_THRESHOLD_0(intr_pktcount[0]) | V_THRESHOLD_1(intr_pktcount[1]) |
V_THRESHOLD_2(intr_pktcount[2]) | V_THRESHOLD_3(intr_pktcount[3]);
t4_write_reg(sc, A_SGE_INGRESS_RX_THRESHOLD, v);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
Auto-reduce the holdoff timers that are greater than the maximum value allowed by the hardware. MFC after: 3 days
2013-04-11 22:46:39 +00:00
KASSERT(intr_timer[0] <= timer_max,
("%s: not a single usable timer (%d, %d)", __func__, intr_timer[0],
timer_max));
for (i = 1; i < nitems(intr_timer); i++) {
KASSERT(intr_timer[i] >= intr_timer[i - 1],
("%s: timers not listed in increasing order (%d)",
__func__, i));
while (intr_timer[i] > timer_max) {
if (i == nitems(intr_timer) - 1) {
intr_timer[i] = timer_max;
break;
}
intr_timer[i] += intr_timer[i - 1];
intr_timer[i] /= 2;
}
}
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
v = V_TIMERVALUE0(us_to_core_ticks(sc, intr_timer[0])) |
V_TIMERVALUE1(us_to_core_ticks(sc, intr_timer[1]));
t4_write_reg(sc, A_SGE_TIMER_VALUE_0_AND_1, v);
v = V_TIMERVALUE2(us_to_core_ticks(sc, intr_timer[2])) |
V_TIMERVALUE3(us_to_core_ticks(sc, intr_timer[3]));
t4_write_reg(sc, A_SGE_TIMER_VALUE_2_AND_3, v);
v = V_TIMERVALUE4(us_to_core_ticks(sc, intr_timer[4])) |
V_TIMERVALUE5(us_to_core_ticks(sc, intr_timer[5]));
t4_write_reg(sc, A_SGE_TIMER_VALUE_4_AND_5, v);
cxgbe: Add a tunable to configure the SGE time scaler, which is available starting with T6. The values in the timer holdoff registers are multiplied by the scaling factor before use. dev.<nexus>.<n>.holdoff_timers shows the final values of the timers in microseconds. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-15 17:00:50 +00:00
if (chip_id(sc) >= CHELSIO_T6) {
m = V_TSCALE(M_TSCALE);
if (tscale == 1)
v = 0;
else
v = V_TSCALE(tscale - 2);
t4_set_reg_field(sc, A_SGE_ITP_CONTROL, m, v);
cxgbe(4): Add two new debug flags -- one to allow manual firmware install after full initialization, and another to disable the TCB cache (T6+). The latter works as a tunable only. Note that debug_flags are for debugging only and should not be set normally. MFC after: 1 week Sponsored by: Chelsio Communications
2017-08-30 23:41:04 +00:00
if (sc->debug_flags & DF_DISABLE_TCB_CACHE) {
m = V_RDTHRESHOLD(M_RDTHRESHOLD) | F_WRTHRTHRESHEN |
V_WRTHRTHRESH(M_WRTHRTHRESH);
t4_tp_pio_read(sc, &v, 1, A_TP_CMM_CONFIG, 1);
v &= ~m;
v |= V_RDTHRESHOLD(1) | F_WRTHRTHRESHEN |
V_WRTHRTHRESH(16);
t4_tp_pio_write(sc, &v, 1, A_TP_CMM_CONFIG, 1);
}
cxgbe: Add a tunable to configure the SGE time scaler, which is available starting with T6. The values in the timer holdoff registers are multiplied by the scaling factor before use. dev.<nexus>.<n>.holdoff_timers shows the final values of the timers in microseconds. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-15 17:00:50 +00:00
}
cxgbe/cxgbei: Retire all DDP related code from cxgbei and switch to routines available in t4_tom to manage the iSCSI DDP page pod region. This adds the ability to use multiple DDP page sizes to the iSCSI driver, among other improvements. Sponsored by: Chelsio Communications
2016-09-01 20:43:01 +00:00
/* 4K, 16K, 64K, 256K DDP "page sizes" for TDDP */
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
v = V_HPZ0(0) | V_HPZ1(2) | V_HPZ2(4) | V_HPZ3(6);
t4_write_reg(sc, A_ULP_RX_TDDP_PSZ, v);
cxgbe/cxgbei: Retire all DDP related code from cxgbei and switch to routines available in t4_tom to manage the iSCSI DDP page pod region. This adds the ability to use multiple DDP page sizes to the iSCSI driver, among other improvements. Sponsored by: Chelsio Communications
2016-09-01 20:43:01 +00:00
/*
* 4K, 8K, 16K, 64K DDP "page sizes" for iSCSI DDP. These have been
* chosen with MAXPHYS = 128K in mind. The largest DDP buffer that we
* may have to deal with is MAXPHYS + 1 page.
*/
v = V_HPZ0(0) | V_HPZ1(1) | V_HPZ2(2) | V_HPZ3(4);
t4_write_reg(sc, A_ULP_RX_ISCSI_PSZ, v);
/* We use multiple DDP page sizes both in plain-TOE and ISCSI modes. */
m = v = F_TDDPTAGTCB | F_ISCSITAGTCB;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
t4_set_reg_field(sc, A_ULP_RX_CTL, m, v);
m = V_INDICATESIZE(M_INDICATESIZE) | F_REARMDDPOFFSET |
F_RESETDDPOFFSET;
v = V_INDICATESIZE(indsz) | F_REARMDDPOFFSET | F_RESETDDPOFFSET;
t4_set_reg_field(sc, A_TP_PARA_REG5, m, v);
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
/*
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
* SGE wants the buffer to be at least 64B and then a multiple of 16. Its
* address mut be 16B aligned. If padding is in use the buffer's start and end
* need to be aligned to the pad boundary as well. We'll just make sure that
* the size is a multiple of the pad boundary here, it is up to the buffer
* allocation code to make sure the start of the buffer is aligned.
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
*/
static inline int
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
hwsz_ok(struct adapter *sc, int hwsz)
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
{
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
int mask = fl_pad ? sc->params.sge.pad_boundary - 1 : 16 - 1;
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
cxgbe(4): allow the driver to use rx buffers that do not end on a pack boundary. MFC after: 2 weeks
2014-12-06 01:47:38 +00:00
return (hwsz >= 64 && (hwsz & mask) == 0);
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
}
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
/*
* XXX: driver really should be able to deal with unexpected settings.
*/
int
t4_read_chip_settings(struct adapter *sc)
{
struct sge *s = &sc->sge;
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
struct sge_params *sp = &sc->params.sge;
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
int i, j, n, rc = 0;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
uint32_t m, v, r;
uint16_t indsz = min(RX_COPY_THRESHOLD - 1, M_INDICATESIZE);
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
static int sw_buf_sizes[] = { /* Sorted by size */
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
MCLBYTES,
#if MJUMPAGESIZE != MCLBYTES
MJUMPAGESIZE,
#endif
MJUM9BYTES,
MJUM16BYTES
};
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
struct rx_buf_info *rxb;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
m = F_RXPKTCPLMODE;
v = F_RXPKTCPLMODE;
Make SGE parameter handling more VF-friendly. Add fields to hold the SGE control register and free list buffer sizes to the sge_params structure. Populate these new fields in t4_init_sge_params() for PF devices and change t4_read_chip_settings() to pull these values out of the params structure instead of reading registers directly. This will permit t4_read_chip_settings() to be reused for VF devices which cannot read SGE registers directly. While here, move the call to t4_init_sge_params() to get_params__post_init(). The VF driver will populate the SGE parameters structure via a different method before calling t4_read_chip_settings(). Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7476
2016-08-15 17:40:05 +00:00
r = sc->params.sge.sge_control;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
if ((r & m) != v) {
device_printf(sc->dev, "invalid SGE_CONTROL(0x%x)\n", r);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
rc = EINVAL;
}
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
/*
* If this changes then every single use of PAGE_SHIFT in the driver
* needs to be carefully reviewed for PAGE_SHIFT vs sp->page_shift.
*/
if (sp->page_shift != PAGE_SHIFT) {
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
device_printf(sc->dev, "invalid SGE_HOST_PAGE_SIZE(0x%x)\n", r);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
rc = EINVAL;
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
s->safe_zidx = -1;
rxb = &s->rx_buf_info[0];
for (i = 0; i < SW_ZONE_SIZES; i++, rxb++) {
rxb->size1 = sw_buf_sizes[i];
rxb->zone = m_getzone(rxb->size1);
rxb->type = m_gettype(rxb->size1);
rxb->size2 = 0;
rxb->hwidx1 = -1;
rxb->hwidx2 = -1;
for (j = 0; j < SGE_FLBUF_SIZES; j++) {
int hwsize = sp->sge_fl_buffer_size[j];
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if (!hwsz_ok(sc, hwsize))
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
continue;
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
/* hwidx for size1 */
if (rxb->hwidx1 == -1 && rxb->size1 == hwsize)
rxb->hwidx1 = j;
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
/* hwidx for size2 (buffer packing) */
if (rxb->size1 - CL_METADATA_SIZE < hwsize)
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
continue;
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
n = rxb->size1 - hwsize - CL_METADATA_SIZE;
if (n == 0) {
rxb->hwidx2 = j;
rxb->size2 = hwsize;
break; /* stop looking */
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
}
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if (rxb->hwidx2 != -1) {
if (n < sp->sge_fl_buffer_size[rxb->hwidx2] -
hwsize - CL_METADATA_SIZE) {
rxb->hwidx2 = j;
rxb->size2 = hwsize;
}
} else if (n <= 2 * CL_METADATA_SIZE) {
rxb->hwidx2 = j;
rxb->size2 = hwsize;
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
}
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if (rxb->hwidx2 != -1)
sc->flags |= BUF_PACKING_OK;
if (s->safe_zidx == -1 && rxb->size1 == safest_rx_cluster)
s->safe_zidx = i;
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
if (sc->flags & IS_VF)
return (0);
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
v = V_HPZ0(0) | V_HPZ1(2) | V_HPZ2(4) | V_HPZ3(6);
r = t4_read_reg(sc, A_ULP_RX_TDDP_PSZ);
if (r != v) {
device_printf(sc->dev, "invalid ULP_RX_TDDP_PSZ(0x%x)\n", r);
rc = EINVAL;
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
m = v = F_TDDPTAGTCB;
r = t4_read_reg(sc, A_ULP_RX_CTL);
if ((r & m) != v) {
device_printf(sc->dev, "invalid ULP_RX_CTL(0x%x)\n", r);
rc = EINVAL;
}
Allow lazy fill up of freelists. MFC after: 3 days
2011-06-04 23:31:33 +00:00
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
m = V_INDICATESIZE(M_INDICATESIZE) | F_REARMDDPOFFSET |
F_RESETDDPOFFSET;
v = V_INDICATESIZE(indsz) | F_REARMDDPOFFSET | F_RESETDDPOFFSET;
r = t4_read_reg(sc, A_TP_PARA_REG5);
if ((r & m) != v) {
device_printf(sc->dev, "invalid TP_PARA_REG5(0x%x)\n", r);
rc = EINVAL;
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Some updates to the common code. - Updated register ranges. - Helper routines for access to TP registers. - Updated routine to read flash parameters. Obtained from: Chelsio Communications MFC after: 2 weeks Sponsored by: Chelsio Communications
2017-07-26 20:20:58 +00:00
t4_init_tp_params(sc, 1);
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
t4_read_mtu_tbl(sc, sc->params.mtus, NULL);
t4_load_mtus(sc, sc->params.mtus, sc->params.a_wnd, sc->params.b_wnd);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
return (rc);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
int
t4_create_dma_tag(struct adapter *sc)
{
int rc;
rc = bus_dma_tag_create(bus_get_dma_tag(sc->dev), 1, 0,
BUS_SPACE_MAXADDR, BUS_SPACE_MAXADDR, NULL, NULL, BUS_SPACE_MAXSIZE,
BUS_SPACE_UNRESTRICTED, BUS_SPACE_MAXSIZE, BUS_DMA_ALLOCNOW, NULL,
NULL, &sc->dmat);
if (rc != 0) {
device_printf(sc->dev,
"failed to create main DMA tag: %d\n", rc);
}
return (rc);
}
Display SGE tunables in the sysctl tree. dev.t5nex.0.fl_pktshift: payload DMA offset in rx buffer (bytes) dev.t5nex.0.fl_pad: payload pad boundary (bytes) dev.t5nex.0.spg_len: status page size (bytes) dev.t5nex.0.cong_drop: congestion drop setting Discussed with: scottl
2013-07-31 05:12:51 +00:00
void
t4_sge_sysctls(struct adapter *sc, struct sysctl_ctx_list *ctx,
struct sysctl_oid_list *children)
{
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
struct sge_params *sp = &sc->params.sge;
Display SGE tunables in the sysctl tree. dev.t5nex.0.fl_pktshift: payload DMA offset in rx buffer (bytes) dev.t5nex.0.fl_pad: payload pad boundary (bytes) dev.t5nex.0.spg_len: status page size (bytes) dev.t5nex.0.cong_drop: congestion drop setting Discussed with: scottl
2013-07-31 05:12:51 +00:00
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "buffer_sizes",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_STRING | CTLFLAG_RD | CTLFLAG_MPSAFE, sc, 0,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
sysctl_bufsizes, "A", "freelist buffer sizes");
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
Display SGE tunables in the sysctl tree. dev.t5nex.0.fl_pktshift: payload DMA offset in rx buffer (bytes) dev.t5nex.0.fl_pad: payload pad boundary (bytes) dev.t5nex.0.spg_len: status page size (bytes) dev.t5nex.0.cong_drop: congestion drop setting Discussed with: scottl
2013-07-31 05:12:51 +00:00
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "fl_pktshift", CTLFLAG_RD,
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
NULL, sp->fl_pktshift, "payload DMA offset in rx buffer (bytes)");
Display SGE tunables in the sysctl tree. dev.t5nex.0.fl_pktshift: payload DMA offset in rx buffer (bytes) dev.t5nex.0.fl_pad: payload pad boundary (bytes) dev.t5nex.0.spg_len: status page size (bytes) dev.t5nex.0.cong_drop: congestion drop setting Discussed with: scottl
2013-07-31 05:12:51 +00:00
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "fl_pad", CTLFLAG_RD,
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
NULL, sp->pad_boundary, "payload pad boundary (bytes)");
Display SGE tunables in the sysctl tree. dev.t5nex.0.fl_pktshift: payload DMA offset in rx buffer (bytes) dev.t5nex.0.fl_pad: payload pad boundary (bytes) dev.t5nex.0.spg_len: status page size (bytes) dev.t5nex.0.cong_drop: congestion drop setting Discussed with: scottl
2013-07-31 05:12:51 +00:00
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "spg_len", CTLFLAG_RD,
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
NULL, sp->spg_len, "status page size (bytes)");
Display SGE tunables in the sysctl tree. dev.t5nex.0.fl_pktshift: payload DMA offset in rx buffer (bytes) dev.t5nex.0.fl_pad: payload pad boundary (bytes) dev.t5nex.0.spg_len: status page size (bytes) dev.t5nex.0.cong_drop: congestion drop setting Discussed with: scottl
2013-07-31 05:12:51 +00:00
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "cong_drop", CTLFLAG_RD,
NULL, cong_drop, "congestion drop setting");
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "fl_pack", CTLFLAG_RD,
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
NULL, sp->pack_boundary, "payload pack boundary (bytes)");
Display SGE tunables in the sysctl tree. dev.t5nex.0.fl_pktshift: payload DMA offset in rx buffer (bytes) dev.t5nex.0.fl_pad: payload pad boundary (bytes) dev.t5nex.0.spg_len: status page size (bytes) dev.t5nex.0.cong_drop: congestion drop setting Discussed with: scottl
2013-07-31 05:12:51 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
int
t4_destroy_dma_tag(struct adapter *sc)
{
if (sc->dmat)
bus_dma_tag_destroy(sc->dmat);
return (0);
}
/*
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
* Allocate and initialize the firmware event queue, control queues, and special
* purpose rx queues owned by the adapter.
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
*
* Returns errno on failure. Resources allocated up to that point may still be
* allocated. Caller is responsible for cleanup in case this function fails.
*/
int
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
t4_setup_adapter_queues(struct adapter *sc)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
struct sysctl_oid *oid;
struct sysctl_oid_list *children;
int rc, i;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
ADAPTER_LOCK_ASSERT_NOTOWNED(sc);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
sysctl_ctx_init(&sc->ctx);
sc->flags |= ADAP_SYSCTL_CTX;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
/*
* Firmware event queue
*/
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
rc = alloc_fwq(sc);
There is no need to report the same error twice. MFC after: 3 days
2012-10-10 16:54:14 +00:00
if (rc != 0)
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
return (rc);
/*
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
* That's all for the VF driver.
*/
if (sc->flags & IS_VF)
return (rc);
oid = device_get_sysctl_tree(sc->dev);
children = SYSCTL_CHILDREN(oid);
/*
* XXX: General purpose rx queues, one per port.
*/
/*
* Control queues, one per port.
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
*/
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
oid = SYSCTL_ADD_NODE(&sc->ctx, children, OID_AUTO, "ctrlq",
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "control queues");
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
for_each_port(sc, i) {
struct sge_wrq *ctrlq = &sc->sge.ctrlq[i];
rc = alloc_ctrlq(sc, ctrlq, i, oid);
if (rc != 0)
return (rc);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
return (rc);
}
/*
* Idempotent
*/
int
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
t4_teardown_adapter_queues(struct adapter *sc)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
int i;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
ADAPTER_LOCK_ASSERT_NOTOWNED(sc);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
/* Do this before freeing the queue */
if (sc->flags & ADAP_SYSCTL_CTX) {
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
sysctl_ctx_free(&sc->ctx);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
sc->flags &= ~ADAP_SYSCTL_CTX;
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
}
cxgbev(4): Updates to the VF driver to cope with recent ifmedia and ctrlq changes in the base driver. MFC after: 1 week Sponsored by: Chelsio Communications
2018-08-23 00:58:10 +00:00
if (!(sc->flags & IS_VF)) {
for_each_port(sc, i)
free_wrq(sc, &sc->sge.ctrlq[i]);
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
free_fwq(sc);
return (0);
}
cxgbe(4): Use large clusters for TOE rx queues when TOE+TLS is enabled. Rx is more efficient within the chip when the receive buffer size matches the TLS PDU size. MFC after: 3 days Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D26127
2020-08-23 04:16:20 +00:00
/* Maximum payload that could arrive with a single iq descriptor. */
The size of the buffers in an Ethernet freelist has to be higher than the interface's MTU. Initialize such freelists with correct values. This wasn't a problem for common MTUs (1500 and 9000) as the buffers (2048 and 9216 in size) happened to have enough spare room. I ran into it when playing around with unusual MTUs. MFC after: 2 weeks
2012-08-15 01:03:13 +00:00
static inline int
cxgbe(4): Use large clusters for TOE rx queues when TOE+TLS is enabled. Rx is more efficient within the chip when the receive buffer size matches the TLS PDU size. MFC after: 3 days Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D26127
2020-08-23 04:16:20 +00:00
max_rx_payload(struct adapter *sc, struct ifnet *ifp, const bool ofld)
The size of the buffers in an Ethernet freelist has to be higher than the interface's MTU. Initialize such freelists with correct values. This wasn't a problem for common MTUs (1500 and 9000) as the buffers (2048 and 9216 in size) happened to have enough spare room. I ran into it when playing around with unusual MTUs. MFC after: 2 weeks
2012-08-15 01:03:13 +00:00
{
cxgbe(4): Use large clusters for TOE rx queues when TOE+TLS is enabled. Rx is more efficient within the chip when the receive buffer size matches the TLS PDU size. MFC after: 3 days Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D26127
2020-08-23 04:16:20 +00:00
int maxp;
The size of the buffers in an Ethernet freelist has to be higher than the interface's MTU. Initialize such freelists with correct values. This wasn't a problem for common MTUs (1500 and 9000) as the buffers (2048 and 9216 in size) happened to have enough spare room. I ran into it when playing around with unusual MTUs. MFC after: 2 weeks
2012-08-15 01:03:13 +00:00
cxgbe(4): Use the same buffer size for TOE rx queues as the NIC rx queues. This is a minor simplification. MFC after: 1 week Sponsored by: Chelsio Communications
2019-08-23 22:22:34 +00:00
/* large enough even when hw VLAN extraction is disabled */
cxgbe(4): Use large clusters for TOE rx queues when TOE+TLS is enabled. Rx is more efficient within the chip when the receive buffer size matches the TLS PDU size. MFC after: 3 days Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D26127
2020-08-23 04:16:20 +00:00
maxp = sc->params.sge.fl_pktshift + ETHER_HDR_LEN +
ETHER_VLAN_ENCAP_LEN + ifp->if_mtu;
if (ofld && sc->tt.tls && sc->cryptocaps & FW_CAPS_CONFIG_TLSKEYS &&
maxp < sc->params.tp.max_rx_pdu)
maxp = sc->params.tp.max_rx_pdu;
return (maxp);
- Make note of interface MTU change if the rx queues exist, and not just when the interface is up. - Add a tunable to control the TOE's rx coalesce feature (enabled by default as it always has been). Consider the interface MTU or the coalesce size when deciding which cluster zone to use to fill the offload rx queue's free list. The tunable is: dev.{t4nex,t5nex}.<N>.toe.rx_coalesce MFC after: 1 day
2013-07-04 21:19:01 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
t4_setup_vi_queues(struct vi_info *vi)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
int rc = 0, i, intr_idx, iqidx;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct sge_rxq *rxq;
struct sge_txq *txq;
- Updated TOE support in the kernel. - Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs. These are available as t3_tom and t4_tom modules that augment cxgb(4) and cxgbe(4) respectively. The cxgb/cxgbe drivers continue to work as usual with or without these extra features. - iWARP driver for Terminator 3 ASIC (kernel verbs). T4 iWARP in the works and will follow soon. Build-tested with make universe. 30s overview ============ What interfaces support TCP offload? Look for TOE4 and/or TOE6 in the capabilities of an interface: # ifconfig -m | grep TOE Enable/disable TCP offload on an interface (just like any other ifnet capability): # ifconfig cxgbe0 toe # ifconfig cxgbe0 -toe Which connections are offloaded? Look for toe4 and/or toe6 in the output of netstat and sockstat: # netstat -np tcp | grep toe # sockstat -46c | grep toe Reviewed by: bz, gnn Sponsored by: Chelsio communications. MFC after: ~3 months (after 9.1, and after ensuring MFC is feasible)
2012-06-19 07:34:13 +00:00
#ifdef TCP_OFFLOAD
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sge_ofld_rxq *ofld_rxq;
cxgbe(4): Allocate offload Tx queues when a card has resources provisioned for NIC_ETHOFLD and the kernel has option RATELIMIT. It is possible to use the chip's offload queues for normal NIC Tx and not just TOE Tx. The difference is that these queues support out of order processing of work requests and have a per-"flowid" mechanism for tracking credits between the driver and hardware. This allows Tx for any number of flows bound to different rate limits to be submitted to a single Tx queue and the work requests for slow flows won't cause HOL blocking for the rest. Sponsored by: Chelsio Communications
2018-05-17 01:42:18 +00:00
#endif
#if defined(TCP_OFFLOAD) || defined(RATELIMIT)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sge_wrq *ofld_txq;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
#endif
#ifdef DEV_NETMAP
cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main cxgbe/cxl interfaces and tunables related to them are not affected by any of this and will continue to operate as usual. The driver used to create an additional 'n' interface for every cxgbe/cxl interface if "device netmap" was in the kernel. The 'n' interface shared the wire with the main interface but was otherwise autonomous (with its own MAC address, etc.). It did not have normal tx/rx but had a specialized netmap-only data path. r291665 added another set of virtual interfaces (the 'v' interfaces) to the driver. These had normal tx/rx but no netmap support. This revision consolidates the features of both the interfaces into the 'v' interface which now has a normal data path, TOE support, and native netmap support. The 'v' interfaces need to be created explicitly with the hw.cxgbe.num_vis tunable. This means "device netmap" will not result in the automatic creation of any virtual interfaces. The following tunables can be used to override the default number of queues allocated for each 'v' interface. nofld* = 0 will disable TOE on the virtual interface and nnm* = 0 to will disable native netmap support. # number of normal NIC queues hw.cxgbe.ntxq_vi hw.cxgbe.nrxq_vi # number of TOE queues hw.cxgbe.nofldtxq_vi hw.cxgbe.nofldrxq_vi # number of netmap queues hw.cxgbe.nnmtxq_vi hw.cxgbe.nnmrxq_vi hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed. --- tl;dr version --- The workflow for netmap on cxgbe starting with FreeBSD 11 is: 1) "device netmap" in the kernel config. 2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll end up with multiple autonomous netmap-capable interfaces for every port. 3) "dmesg | grep vcxl | grep netmap" to verify that the interface has netmap queues. 4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... . One major improvement is that the netmap interface has a normal data path as expected. 5) Just ignore the cxl interfaces if you want to use netmap only. No need to bring them up. The vcxl interfaces are completely independent and everything should just work. --------------------- Approved by: re@ (gjb@) Relnotes: Yes Sponsored by: Chelsio Communications
2016-06-23 02:53:00 +00:00
int saved_idx;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
struct sge_nm_rxq *nm_rxq;
struct sge_nm_txq *nm_txq;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
#endif
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
char name[16];
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
struct port_info *pi = vi->pi;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct adapter *sc = pi->adapter;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
struct ifnet *ifp = vi->ifp;
struct sysctl_oid *oid = device_get_sysctl_tree(vi->dev);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sysctl_oid_list *children = SYSCTL_CHILDREN(oid);
cxgbe(4): Use large clusters for TOE rx queues when TOE+TLS is enabled. Rx is more efficient within the chip when the receive buffer size matches the TLS PDU size. MFC after: 3 days Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D26127
2020-08-23 04:16:20 +00:00
int maxp;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
/* Interrupt vector to start from (when using multiple vectors) */
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
intr_idx = vi->first_intr;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
#ifdef DEV_NETMAP
cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main cxgbe/cxl interfaces and tunables related to them are not affected by any of this and will continue to operate as usual. The driver used to create an additional 'n' interface for every cxgbe/cxl interface if "device netmap" was in the kernel. The 'n' interface shared the wire with the main interface but was otherwise autonomous (with its own MAC address, etc.). It did not have normal tx/rx but had a specialized netmap-only data path. r291665 added another set of virtual interfaces (the 'v' interfaces) to the driver. These had normal tx/rx but no netmap support. This revision consolidates the features of both the interfaces into the 'v' interface which now has a normal data path, TOE support, and native netmap support. The 'v' interfaces need to be created explicitly with the hw.cxgbe.num_vis tunable. This means "device netmap" will not result in the automatic creation of any virtual interfaces. The following tunables can be used to override the default number of queues allocated for each 'v' interface. nofld* = 0 will disable TOE on the virtual interface and nnm* = 0 to will disable native netmap support. # number of normal NIC queues hw.cxgbe.ntxq_vi hw.cxgbe.nrxq_vi # number of TOE queues hw.cxgbe.nofldtxq_vi hw.cxgbe.nofldrxq_vi # number of netmap queues hw.cxgbe.nnmtxq_vi hw.cxgbe.nnmrxq_vi hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed. --- tl;dr version --- The workflow for netmap on cxgbe starting with FreeBSD 11 is: 1) "device netmap" in the kernel config. 2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll end up with multiple autonomous netmap-capable interfaces for every port. 3) "dmesg | grep vcxl | grep netmap" to verify that the interface has netmap queues. 4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... . One major improvement is that the netmap interface has a normal data path as expected. 5) Just ignore the cxl interfaces if you want to use netmap only. No need to bring them up. The vcxl interfaces are completely independent and everything should just work. --------------------- Approved by: re@ (gjb@) Relnotes: Yes Sponsored by: Chelsio Communications
2016-06-23 02:53:00 +00:00
saved_idx = intr_idx;
if (ifp->if_capabilities & IFCAP_NETMAP) {
/* netmap is supported with direct interrupts only. */
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
MPASS(!forwarding_intr_to_fwq(sc));
cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main cxgbe/cxl interfaces and tunables related to them are not affected by any of this and will continue to operate as usual. The driver used to create an additional 'n' interface for every cxgbe/cxl interface if "device netmap" was in the kernel. The 'n' interface shared the wire with the main interface but was otherwise autonomous (with its own MAC address, etc.). It did not have normal tx/rx but had a specialized netmap-only data path. r291665 added another set of virtual interfaces (the 'v' interfaces) to the driver. These had normal tx/rx but no netmap support. This revision consolidates the features of both the interfaces into the 'v' interface which now has a normal data path, TOE support, and native netmap support. The 'v' interfaces need to be created explicitly with the hw.cxgbe.num_vis tunable. This means "device netmap" will not result in the automatic creation of any virtual interfaces. The following tunables can be used to override the default number of queues allocated for each 'v' interface. nofld* = 0 will disable TOE on the virtual interface and nnm* = 0 to will disable native netmap support. # number of normal NIC queues hw.cxgbe.ntxq_vi hw.cxgbe.nrxq_vi # number of TOE queues hw.cxgbe.nofldtxq_vi hw.cxgbe.nofldrxq_vi # number of netmap queues hw.cxgbe.nnmtxq_vi hw.cxgbe.nnmrxq_vi hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed. --- tl;dr version --- The workflow for netmap on cxgbe starting with FreeBSD 11 is: 1) "device netmap" in the kernel config. 2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll end up with multiple autonomous netmap-capable interfaces for every port. 3) "dmesg | grep vcxl | grep netmap" to verify that the interface has netmap queues. 4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... . One major improvement is that the netmap interface has a normal data path as expected. 5) Just ignore the cxl interfaces if you want to use netmap only. No need to bring them up. The vcxl interfaces are completely independent and everything should just work. --------------------- Approved by: re@ (gjb@) Relnotes: Yes Sponsored by: Chelsio Communications
2016-06-23 02:53:00 +00:00
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
/*
* We don't have buffers to back the netmap rx queues
* right now so we create the queues in a way that
* doesn't set off any congestion signal in the chip.
*/
cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main cxgbe/cxl interfaces and tunables related to them are not affected by any of this and will continue to operate as usual. The driver used to create an additional 'n' interface for every cxgbe/cxl interface if "device netmap" was in the kernel. The 'n' interface shared the wire with the main interface but was otherwise autonomous (with its own MAC address, etc.). It did not have normal tx/rx but had a specialized netmap-only data path. r291665 added another set of virtual interfaces (the 'v' interfaces) to the driver. These had normal tx/rx but no netmap support. This revision consolidates the features of both the interfaces into the 'v' interface which now has a normal data path, TOE support, and native netmap support. The 'v' interfaces need to be created explicitly with the hw.cxgbe.num_vis tunable. This means "device netmap" will not result in the automatic creation of any virtual interfaces. The following tunables can be used to override the default number of queues allocated for each 'v' interface. nofld* = 0 will disable TOE on the virtual interface and nnm* = 0 to will disable native netmap support. # number of normal NIC queues hw.cxgbe.ntxq_vi hw.cxgbe.nrxq_vi # number of TOE queues hw.cxgbe.nofldtxq_vi hw.cxgbe.nofldrxq_vi # number of netmap queues hw.cxgbe.nnmtxq_vi hw.cxgbe.nnmrxq_vi hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed. --- tl;dr version --- The workflow for netmap on cxgbe starting with FreeBSD 11 is: 1) "device netmap" in the kernel config. 2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll end up with multiple autonomous netmap-capable interfaces for every port. 3) "dmesg | grep vcxl | grep netmap" to verify that the interface has netmap queues. 4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... . One major improvement is that the netmap interface has a normal data path as expected. 5) Just ignore the cxl interfaces if you want to use netmap only. No need to bring them up. The vcxl interfaces are completely independent and everything should just work. --------------------- Approved by: re@ (gjb@) Relnotes: Yes Sponsored by: Chelsio Communications
2016-06-23 02:53:00 +00:00
oid = SYSCTL_ADD_NODE(&vi->ctx, children, OID_AUTO, "nm_rxq",
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "rx queues");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_nm_rxq(vi, i, nm_rxq) {
rc = alloc_nm_rxq(vi, nm_rxq, intr_idx, i, oid);
if (rc != 0)
goto done;
intr_idx++;
}
cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main cxgbe/cxl interfaces and tunables related to them are not affected by any of this and will continue to operate as usual. The driver used to create an additional 'n' interface for every cxgbe/cxl interface if "device netmap" was in the kernel. The 'n' interface shared the wire with the main interface but was otherwise autonomous (with its own MAC address, etc.). It did not have normal tx/rx but had a specialized netmap-only data path. r291665 added another set of virtual interfaces (the 'v' interfaces) to the driver. These had normal tx/rx but no netmap support. This revision consolidates the features of both the interfaces into the 'v' interface which now has a normal data path, TOE support, and native netmap support. The 'v' interfaces need to be created explicitly with the hw.cxgbe.num_vis tunable. This means "device netmap" will not result in the automatic creation of any virtual interfaces. The following tunables can be used to override the default number of queues allocated for each 'v' interface. nofld* = 0 will disable TOE on the virtual interface and nnm* = 0 to will disable native netmap support. # number of normal NIC queues hw.cxgbe.ntxq_vi hw.cxgbe.nrxq_vi # number of TOE queues hw.cxgbe.nofldtxq_vi hw.cxgbe.nofldrxq_vi # number of netmap queues hw.cxgbe.nnmtxq_vi hw.cxgbe.nnmrxq_vi hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed. --- tl;dr version --- The workflow for netmap on cxgbe starting with FreeBSD 11 is: 1) "device netmap" in the kernel config. 2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll end up with multiple autonomous netmap-capable interfaces for every port. 3) "dmesg | grep vcxl | grep netmap" to verify that the interface has netmap queues. 4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... . One major improvement is that the netmap interface has a normal data path as expected. 5) Just ignore the cxl interfaces if you want to use netmap only. No need to bring them up. The vcxl interfaces are completely independent and everything should just work. --------------------- Approved by: re@ (gjb@) Relnotes: Yes Sponsored by: Chelsio Communications
2016-06-23 02:53:00 +00:00
oid = SYSCTL_ADD_NODE(&vi->ctx, children, OID_AUTO, "nm_txq",
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "tx queues");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_nm_txq(vi, i, nm_txq) {
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
iqidx = vi->first_nm_rxq + (i % vi->nnmrxq);
rc = alloc_nm_txq(vi, nm_txq, iqidx, i, oid);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
if (rc != 0)
goto done;
}
}
cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main cxgbe/cxl interfaces and tunables related to them are not affected by any of this and will continue to operate as usual. The driver used to create an additional 'n' interface for every cxgbe/cxl interface if "device netmap" was in the kernel. The 'n' interface shared the wire with the main interface but was otherwise autonomous (with its own MAC address, etc.). It did not have normal tx/rx but had a specialized netmap-only data path. r291665 added another set of virtual interfaces (the 'v' interfaces) to the driver. These had normal tx/rx but no netmap support. This revision consolidates the features of both the interfaces into the 'v' interface which now has a normal data path, TOE support, and native netmap support. The 'v' interfaces need to be created explicitly with the hw.cxgbe.num_vis tunable. This means "device netmap" will not result in the automatic creation of any virtual interfaces. The following tunables can be used to override the default number of queues allocated for each 'v' interface. nofld* = 0 will disable TOE on the virtual interface and nnm* = 0 to will disable native netmap support. # number of normal NIC queues hw.cxgbe.ntxq_vi hw.cxgbe.nrxq_vi # number of TOE queues hw.cxgbe.nofldtxq_vi hw.cxgbe.nofldrxq_vi # number of netmap queues hw.cxgbe.nnmtxq_vi hw.cxgbe.nnmrxq_vi hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed. --- tl;dr version --- The workflow for netmap on cxgbe starting with FreeBSD 11 is: 1) "device netmap" in the kernel config. 2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll end up with multiple autonomous netmap-capable interfaces for every port. 3) "dmesg | grep vcxl | grep netmap" to verify that the interface has netmap queues. 4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... . One major improvement is that the netmap interface has a normal data path as expected. 5) Just ignore the cxl interfaces if you want to use netmap only. No need to bring them up. The vcxl interfaces are completely independent and everything should just work. --------------------- Approved by: re@ (gjb@) Relnotes: Yes Sponsored by: Chelsio Communications
2016-06-23 02:53:00 +00:00
/* Normal rx queues and netmap rx queues share the same interrupts. */
intr_idx = saved_idx;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
#endif
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
/*
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
* Allocate rx queues first because a default iqid is required when
* creating a tx queue.
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
*/
cxgbe(4): Use large clusters for TOE rx queues when TOE+TLS is enabled. Rx is more efficient within the chip when the receive buffer size matches the TLS PDU size. MFC after: 3 days Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D26127
2020-08-23 04:16:20 +00:00
maxp = max_rx_payload(sc, ifp, false);
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
oid = SYSCTL_ADD_NODE(&vi->ctx, children, OID_AUTO, "rxq",
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "rx queues");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_rxq(vi, i, rxq) {
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
init_iq(&rxq->iq, sc, vi->tmr_idx, vi->pktc_idx, vi->qsize_rxq);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
snprintf(name, sizeof(name), "%s rxq%d-fl",
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
device_get_nameunit(vi->dev), i);
init_fl(sc, &rxq->fl, vi->qsize_rxq / 8, maxp, name);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
rc = alloc_rxq(vi, rxq,
forwarding_intr_to_fwq(sc) ? -1 : intr_idx, i, oid);
if (rc != 0)
goto done;
intr_idx++;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main cxgbe/cxl interfaces and tunables related to them are not affected by any of this and will continue to operate as usual. The driver used to create an additional 'n' interface for every cxgbe/cxl interface if "device netmap" was in the kernel. The 'n' interface shared the wire with the main interface but was otherwise autonomous (with its own MAC address, etc.). It did not have normal tx/rx but had a specialized netmap-only data path. r291665 added another set of virtual interfaces (the 'v' interfaces) to the driver. These had normal tx/rx but no netmap support. This revision consolidates the features of both the interfaces into the 'v' interface which now has a normal data path, TOE support, and native netmap support. The 'v' interfaces need to be created explicitly with the hw.cxgbe.num_vis tunable. This means "device netmap" will not result in the automatic creation of any virtual interfaces. The following tunables can be used to override the default number of queues allocated for each 'v' interface. nofld* = 0 will disable TOE on the virtual interface and nnm* = 0 to will disable native netmap support. # number of normal NIC queues hw.cxgbe.ntxq_vi hw.cxgbe.nrxq_vi # number of TOE queues hw.cxgbe.nofldtxq_vi hw.cxgbe.nofldrxq_vi # number of netmap queues hw.cxgbe.nnmtxq_vi hw.cxgbe.nnmrxq_vi hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed. --- tl;dr version --- The workflow for netmap on cxgbe starting with FreeBSD 11 is: 1) "device netmap" in the kernel config. 2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll end up with multiple autonomous netmap-capable interfaces for every port. 3) "dmesg | grep vcxl | grep netmap" to verify that the interface has netmap queues. 4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... . One major improvement is that the netmap interface has a normal data path as expected. 5) Just ignore the cxl interfaces if you want to use netmap only. No need to bring them up. The vcxl interfaces are completely independent and everything should just work. --------------------- Approved by: re@ (gjb@) Relnotes: Yes Sponsored by: Chelsio Communications
2016-06-23 02:53:00 +00:00
#ifdef DEV_NETMAP
if (ifp->if_capabilities & IFCAP_NETMAP)
intr_idx = saved_idx + max(vi->nrxq, vi->nnmrxq);
#endif
- Updated TOE support in the kernel. - Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs. These are available as t3_tom and t4_tom modules that augment cxgb(4) and cxgbe(4) respectively. The cxgb/cxgbe drivers continue to work as usual with or without these extra features. - iWARP driver for Terminator 3 ASIC (kernel verbs). T4 iWARP in the works and will follow soon. Build-tested with make universe. 30s overview ============ What interfaces support TCP offload? Look for TOE4 and/or TOE6 in the capabilities of an interface: # ifconfig -m | grep TOE Enable/disable TCP offload on an interface (just like any other ifnet capability): # ifconfig cxgbe0 toe # ifconfig cxgbe0 -toe Which connections are offloaded? Look for toe4 and/or toe6 in the output of netstat and sockstat: # netstat -np tcp | grep toe # sockstat -46c | grep toe Reviewed by: bz, gnn Sponsored by: Chelsio communications. MFC after: ~3 months (after 9.1, and after ensuring MFC is feasible)
2012-06-19 07:34:13 +00:00
#ifdef TCP_OFFLOAD
cxgbe(4): Use large clusters for TOE rx queues when TOE+TLS is enabled. Rx is more efficient within the chip when the receive buffer size matches the TLS PDU size. MFC after: 3 days Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D26127
2020-08-23 04:16:20 +00:00
maxp = max_rx_payload(sc, ifp, true);
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
oid = SYSCTL_ADD_NODE(&vi->ctx, children, OID_AUTO, "ofld_rxq",
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "rx queues for offloaded TCP connections");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_ofld_rxq(vi, i, ofld_rxq) {
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Provide knobs to set the holdoff parameters of TOE rx queues separately from NIC rx queues instead of using the same parameters for both types of queues. MFC after: 2 weeks Sponsored by: Chelsio Communications
2017-10-05 07:18:16 +00:00
init_iq(&ofld_rxq->iq, sc, vi->ofld_tmr_idx, vi->ofld_pktc_idx,
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
vi->qsize_rxq);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
snprintf(name, sizeof(name), "%s ofld_rxq%d-fl",
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
device_get_nameunit(vi->dev), i);
init_fl(sc, &ofld_rxq->fl, vi->qsize_rxq / 8, maxp, name);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
rc = alloc_ofld_rxq(vi, ofld_rxq,
forwarding_intr_to_fwq(sc) ? -1 : intr_idx, i, oid);
if (rc != 0)
goto done;
intr_idx++;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
#endif
/*
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
* Now the tx queues.
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
*/
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
oid = SYSCTL_ADD_NODE(&vi->ctx, children, OID_AUTO, "txq",
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "tx queues");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_txq(vi, i, txq) {
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
iqidx = vi->first_rxq + (i % vi->nrxq);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
snprintf(name, sizeof(name), "%s txq%d",
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
device_get_nameunit(vi->dev), i);
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
init_eq(sc, &txq->eq, EQ_ETH, vi->qsize_txq, pi->tx_chan,
sc->sge.rxq[iqidx].iq.cntxt_id, name);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
rc = alloc_txq(vi, txq, i, oid);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
if (rc != 0)
goto done;
}
cxgbe(4): Allocate offload Tx queues when a card has resources provisioned for NIC_ETHOFLD and the kernel has option RATELIMIT. It is possible to use the chip's offload queues for normal NIC Tx and not just TOE Tx. The difference is that these queues support out of order processing of work requests and have a per-"flowid" mechanism for tracking credits between the driver and hardware. This allows Tx for any number of flows bound to different rate limits to be submitted to a single Tx queue and the work requests for slow flows won't cause HOL blocking for the rest. Sponsored by: Chelsio Communications
2018-05-17 01:42:18 +00:00
#if defined(TCP_OFFLOAD) || defined(RATELIMIT)
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
oid = SYSCTL_ADD_NODE(&vi->ctx, children, OID_AUTO, "ofld_txq",
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "tx queues for TOE/ETHOFLD");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_ofld_txq(vi, i, ofld_txq) {
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
struct sysctl_oid *oid2;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
snprintf(name, sizeof(name), "%s ofld_txq%d",
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
device_get_nameunit(vi->dev), i);
cxgbe(4): Fix a divide-by-zero that occurs when hw.cxgbe.toecaps_allowed is set to 0 with a kernel that has both TCP_OFFLOAD and RATELIMIT. Approved by: re@ (gjb@) Sponsored by: Chelsio Communications
2018-10-13 00:13:24 +00:00
if (vi->nofldrxq > 0) {
iqidx = vi->first_ofld_rxq + (i % vi->nofldrxq);
init_eq(sc, &ofld_txq->eq, EQ_OFLD, vi->qsize_txq,
pi->tx_chan, sc->sge.ofld_rxq[iqidx].iq.cntxt_id,
name);
} else {
iqidx = vi->first_rxq + (i % vi->nrxq);
init_eq(sc, &ofld_txq->eq, EQ_OFLD, vi->qsize_txq,
pi->tx_chan, sc->sge.rxq[iqidx].iq.cntxt_id, name);
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
snprintf(name, sizeof(name), "%d", i);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
oid2 = SYSCTL_ADD_NODE(&vi->ctx, SYSCTL_CHILDREN(oid), OID_AUTO,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
name, CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "offload tx queue");
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
rc = alloc_wrq(sc, vi, ofld_txq, oid2);
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
if (rc != 0)
goto done;
}
#endif
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
done:
if (rc)
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
t4_teardown_vi_queues(vi);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
return (rc);
}
/*
* Idempotent
*/
int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
t4_teardown_vi_queues(struct vi_info *vi)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
int i;
struct sge_rxq *rxq;
struct sge_txq *txq;
cxgbe(4): Allocate offload Tx queues when a card has resources provisioned for NIC_ETHOFLD and the kernel has option RATELIMIT. It is possible to use the chip's offload queues for normal NIC Tx and not just TOE Tx. The difference is that these queues support out of order processing of work requests and have a per-"flowid" mechanism for tracking credits between the driver and hardware. This allows Tx for any number of flows bound to different rate limits to be submitted to a single Tx queue and the work requests for slow flows won't cause HOL blocking for the rest. Sponsored by: Chelsio Communications
2018-05-17 01:42:18 +00:00
#if defined(TCP_OFFLOAD) || defined(RATELIMIT)
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
struct port_info *pi = vi->pi;
struct adapter *sc = pi->adapter;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sge_wrq *ofld_txq;
#endif
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
#ifdef TCP_OFFLOAD
struct sge_ofld_rxq *ofld_rxq;
#endif
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
#ifdef DEV_NETMAP
struct sge_nm_rxq *nm_rxq;
struct sge_nm_txq *nm_txq;
#endif
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
/* Do this before freeing the queues */
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
if (vi->flags & VI_SYSCTL_CTX) {
sysctl_ctx_free(&vi->ctx);
vi->flags &= ~VI_SYSCTL_CTX;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
#ifdef DEV_NETMAP
cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main cxgbe/cxl interfaces and tunables related to them are not affected by any of this and will continue to operate as usual. The driver used to create an additional 'n' interface for every cxgbe/cxl interface if "device netmap" was in the kernel. The 'n' interface shared the wire with the main interface but was otherwise autonomous (with its own MAC address, etc.). It did not have normal tx/rx but had a specialized netmap-only data path. r291665 added another set of virtual interfaces (the 'v' interfaces) to the driver. These had normal tx/rx but no netmap support. This revision consolidates the features of both the interfaces into the 'v' interface which now has a normal data path, TOE support, and native netmap support. The 'v' interfaces need to be created explicitly with the hw.cxgbe.num_vis tunable. This means "device netmap" will not result in the automatic creation of any virtual interfaces. The following tunables can be used to override the default number of queues allocated for each 'v' interface. nofld* = 0 will disable TOE on the virtual interface and nnm* = 0 to will disable native netmap support. # number of normal NIC queues hw.cxgbe.ntxq_vi hw.cxgbe.nrxq_vi # number of TOE queues hw.cxgbe.nofldtxq_vi hw.cxgbe.nofldrxq_vi # number of netmap queues hw.cxgbe.nnmtxq_vi hw.cxgbe.nnmrxq_vi hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed. --- tl;dr version --- The workflow for netmap on cxgbe starting with FreeBSD 11 is: 1) "device netmap" in the kernel config. 2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll end up with multiple autonomous netmap-capable interfaces for every port. 3) "dmesg | grep vcxl | grep netmap" to verify that the interface has netmap queues. 4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... . One major improvement is that the netmap interface has a normal data path as expected. 5) Just ignore the cxl interfaces if you want to use netmap only. No need to bring them up. The vcxl interfaces are completely independent and everything should just work. --------------------- Approved by: re@ (gjb@) Relnotes: Yes Sponsored by: Chelsio Communications
2016-06-23 02:53:00 +00:00
if (vi->ifp->if_capabilities & IFCAP_NETMAP) {
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_nm_txq(vi, i, nm_txq) {
free_nm_txq(vi, nm_txq);
}
for_each_nm_rxq(vi, i, nm_rxq) {
free_nm_rxq(vi, nm_rxq);
}
}
#endif
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
/*
* Take down all the tx queues first, as they reference the rx queues
* (for egress updates, etc.).
*/
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_txq(vi, i, txq) {
free_txq(vi, txq);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): Allocate offload Tx queues when a card has resources provisioned for NIC_ETHOFLD and the kernel has option RATELIMIT. It is possible to use the chip's offload queues for normal NIC Tx and not just TOE Tx. The difference is that these queues support out of order processing of work requests and have a per-"flowid" mechanism for tracking credits between the driver and hardware. This allows Tx for any number of flows bound to different rate limits to be submitted to a single Tx queue and the work requests for slow flows won't cause HOL blocking for the rest. Sponsored by: Chelsio Communications
2018-05-17 01:42:18 +00:00
#if defined(TCP_OFFLOAD) || defined(RATELIMIT)
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_ofld_txq(vi, i, ofld_txq) {
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
free_wrq(sc, ofld_txq);
}
#endif
/*
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
* Then take down the rx queues.
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
*/
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_rxq(vi, i, rxq) {
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
free_rxq(vi, rxq);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
- Updated TOE support in the kernel. - Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs. These are available as t3_tom and t4_tom modules that augment cxgb(4) and cxgbe(4) respectively. The cxgb/cxgbe drivers continue to work as usual with or without these extra features. - iWARP driver for Terminator 3 ASIC (kernel verbs). T4 iWARP in the works and will follow soon. Build-tested with make universe. 30s overview ============ What interfaces support TCP offload? Look for TOE4 and/or TOE6 in the capabilities of an interface: # ifconfig -m | grep TOE Enable/disable TCP offload on an interface (just like any other ifnet capability): # ifconfig cxgbe0 toe # ifconfig cxgbe0 -toe Which connections are offloaded? Look for toe4 and/or toe6 in the output of netstat and sockstat: # netstat -np tcp | grep toe # sockstat -46c | grep toe Reviewed by: bz, gnn Sponsored by: Chelsio communications. MFC after: ~3 months (after 9.1, and after ensuring MFC is feasible)
2012-06-19 07:34:13 +00:00
#ifdef TCP_OFFLOAD
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_ofld_rxq(vi, i, ofld_rxq) {
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
free_ofld_rxq(vi, ofld_rxq);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
#endif
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
return (0);
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
/*
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
* Interrupt handler when the driver is using only 1 interrupt. This is a very
* unusual scenario.
*
* a) Deals with errors, if any.
* b) Services firmware event queue, which is taking interrupts for all other
* queues.
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
*/
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
void
t4_intr_all(void *arg)
{
struct adapter *sc = arg;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sge_iq *fwq = &sc->sge.fwq;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
MPASS(sc->intr_count == 1);
cxgbe(4): Move some INTx specific code to a more appropriate place.
2019-01-12 04:44:25 +00:00
if (sc->intr_type == INTR_INTX)
t4_write_reg(sc, MYPF_REG(A_PCIE_PF_CLI), 0);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
t4_intr_err(arg);
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
t4_intr_evt(fwq);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
/*
* Interrupt handler for errors (installed directly when multiple interrupts are
* being used, or called by t4_intr_all).
*/
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
void
t4_intr_err(void *arg)
{
struct adapter *sc = arg;
cxgbe(4): Count and clear interrupts generated at the software's request. An interrupt can be requested by setting the F_SWINT bit in PL_PF_CTL. MFC after: 1 week Sponsored by: Chelsio Communications
2019-03-28 21:22:28 +00:00
uint32_t v;
cxgbe(4): Improved error reporting and diagnostics. "slow" interrupt handler: - Expand the list of INT_CAUSE registers known to the driver. - Add decode information for many more bits but decouple it from the rest of intr_info so that it is entirely optional. - Call t4_fatal_err exactly once, and from the top level PL intr handler. t4_fatal_err: - Use t4_shutdown_adapter from the common code to stop the adapter. - Stop servicing slow interrupts after the first fatal one. Driver/firmware interaction: - CH_DUMP_MBOX: note whether the mailbox being dumped is a command or a reply or something else. - Log the raw value of pcie_fw for some errors. - Use correct log levels (debug vs. error). Sponsored by: Chelsio Communications
2019-02-01 20:42:49 +00:00
const bool verbose = (sc->debug_flags & DF_VERBOSE_SLOWINTR) != 0;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Improved error reporting and diagnostics. "slow" interrupt handler: - Expand the list of INT_CAUSE registers known to the driver. - Add decode information for many more bits but decouple it from the rest of intr_info so that it is entirely optional. - Call t4_fatal_err exactly once, and from the top level PL intr handler. t4_fatal_err: - Use t4_shutdown_adapter from the common code to stop the adapter. - Stop servicing slow interrupts after the first fatal one. Driver/firmware interaction: - CH_DUMP_MBOX: note whether the mailbox being dumped is a command or a reply or something else. - Log the raw value of pcie_fw for some errors. - Use correct log levels (debug vs. error). Sponsored by: Chelsio Communications
2019-02-01 20:42:49 +00:00
if (sc->flags & ADAP_ERR)
return;
cxgbe(4): Count and clear interrupts generated at the software's request. An interrupt can be requested by setting the F_SWINT bit in PL_PF_CTL. MFC after: 1 week Sponsored by: Chelsio Communications
2019-03-28 21:22:28 +00:00
v = t4_read_reg(sc, MYPF_REG(A_PL_PF_INT_CAUSE));
if (v & F_PFSW) {
sc->swintr++;
t4_write_reg(sc, MYPF_REG(A_PL_PF_INT_CAUSE), v);
}
cxgbe(4): Improved error reporting and diagnostics. "slow" interrupt handler: - Expand the list of INT_CAUSE registers known to the driver. - Add decode information for many more bits but decouple it from the rest of intr_info so that it is entirely optional. - Call t4_fatal_err exactly once, and from the top level PL intr handler. t4_fatal_err: - Use t4_shutdown_adapter from the common code to stop the adapter. - Stop servicing slow interrupts after the first fatal one. Driver/firmware interaction: - CH_DUMP_MBOX: note whether the mailbox being dumped is a command or a reply or something else. - Log the raw value of pcie_fw for some errors. - Use correct log levels (debug vs. error). Sponsored by: Chelsio Communications
2019-02-01 20:42:49 +00:00
t4_slow_intr_handler(sc, verbose);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
/*
* Interrupt handler for iq-only queues. The firmware event queue is the only
* such queue right now.
*/
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
void
t4_intr_evt(void *arg)
Fix a couple of bad races that can occur when a cxgbe interface is taken down. The ingress queue lock was unused and has been removed as part of these changes. - An in-flight egress update from the SGE must be handled before the queue that requested it is destroyed. Wait for the update to arrive. - Interrupt handlers must stop processing rx events for a queue before the queue is destroyed. Events that have not yet been processed should be ignored once the queue disappears. MFC after: 1 week
2011-04-15 03:09:27 +00:00
{
struct sge_iq *iq = arg;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (atomic_cmpset_int(&iq->state, IQS_IDLE, IQS_BUSY)) {
service_iq(iq, 0);
cxgbe(4): Be explicit about ignoring the return value of cmpset in some cases. Reported by: Coverity (CIDs 1009398, 1009400, 1009401, 1357325, 1394783). All false positives. Sponsored by: Chelsio Communications
2018-08-21 23:33:38 +00:00
(void) atomic_cmpset_int(&iq->state, IQS_BUSY, IQS_IDLE);
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
}
Fix a couple of bad races that can occur when a cxgbe interface is taken down. The ingress queue lock was unused and has been removed as part of these changes. - An in-flight egress update from the SGE must be handled before the queue that requested it is destroyed. Wait for the update to arrive. - Interrupt handlers must stop processing rx events for a queue before the queue is destroyed. Events that have not yet been processed should be ignored once the queue disappears. MFC after: 1 week
2011-04-15 03:09:27 +00:00
}
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
/*
* Interrupt handler for iq+fl queues.
*/
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
void
t4_intr(void *arg)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
struct sge_iq *iq = arg;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (atomic_cmpset_int(&iq->state, IQS_IDLE, IQS_BUSY)) {
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
service_iq_fl(iq, 0);
cxgbe(4): Be explicit about ignoring the return value of cmpset in some cases. Reported by: Coverity (CIDs 1009398, 1009400, 1009401, 1357325, 1394783). All false positives. Sponsored by: Chelsio Communications
2018-08-21 23:33:38 +00:00
(void) atomic_cmpset_int(&iq->state, IQS_BUSY, IQS_IDLE);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
}
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
#ifdef DEV_NETMAP
/*
* Interrupt handler for netmap rx queues.
*/
cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main cxgbe/cxl interfaces and tunables related to them are not affected by any of this and will continue to operate as usual. The driver used to create an additional 'n' interface for every cxgbe/cxl interface if "device netmap" was in the kernel. The 'n' interface shared the wire with the main interface but was otherwise autonomous (with its own MAC address, etc.). It did not have normal tx/rx but had a specialized netmap-only data path. r291665 added another set of virtual interfaces (the 'v' interfaces) to the driver. These had normal tx/rx but no netmap support. This revision consolidates the features of both the interfaces into the 'v' interface which now has a normal data path, TOE support, and native netmap support. The 'v' interfaces need to be created explicitly with the hw.cxgbe.num_vis tunable. This means "device netmap" will not result in the automatic creation of any virtual interfaces. The following tunables can be used to override the default number of queues allocated for each 'v' interface. nofld* = 0 will disable TOE on the virtual interface and nnm* = 0 to will disable native netmap support. # number of normal NIC queues hw.cxgbe.ntxq_vi hw.cxgbe.nrxq_vi # number of TOE queues hw.cxgbe.nofldtxq_vi hw.cxgbe.nofldrxq_vi # number of netmap queues hw.cxgbe.nnmtxq_vi hw.cxgbe.nnmrxq_vi hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed. --- tl;dr version --- The workflow for netmap on cxgbe starting with FreeBSD 11 is: 1) "device netmap" in the kernel config. 2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll end up with multiple autonomous netmap-capable interfaces for every port. 3) "dmesg | grep vcxl | grep netmap" to verify that the interface has netmap queues. 4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... . One major improvement is that the netmap interface has a normal data path as expected. 5) Just ignore the cxl interfaces if you want to use netmap only. No need to bring them up. The vcxl interfaces are completely independent and everything should just work. --------------------- Approved by: re@ (gjb@) Relnotes: Yes Sponsored by: Chelsio Communications
2016-06-23 02:53:00 +00:00
void
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
t4_nm_intr(void *arg)
cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main cxgbe/cxl interfaces and tunables related to them are not affected by any of this and will continue to operate as usual. The driver used to create an additional 'n' interface for every cxgbe/cxl interface if "device netmap" was in the kernel. The 'n' interface shared the wire with the main interface but was otherwise autonomous (with its own MAC address, etc.). It did not have normal tx/rx but had a specialized netmap-only data path. r291665 added another set of virtual interfaces (the 'v' interfaces) to the driver. These had normal tx/rx but no netmap support. This revision consolidates the features of both the interfaces into the 'v' interface which now has a normal data path, TOE support, and native netmap support. The 'v' interfaces need to be created explicitly with the hw.cxgbe.num_vis tunable. This means "device netmap" will not result in the automatic creation of any virtual interfaces. The following tunables can be used to override the default number of queues allocated for each 'v' interface. nofld* = 0 will disable TOE on the virtual interface and nnm* = 0 to will disable native netmap support. # number of normal NIC queues hw.cxgbe.ntxq_vi hw.cxgbe.nrxq_vi # number of TOE queues hw.cxgbe.nofldtxq_vi hw.cxgbe.nofldrxq_vi # number of netmap queues hw.cxgbe.nnmtxq_vi hw.cxgbe.nnmrxq_vi hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed. --- tl;dr version --- The workflow for netmap on cxgbe starting with FreeBSD 11 is: 1) "device netmap" in the kernel config. 2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll end up with multiple autonomous netmap-capable interfaces for every port. 3) "dmesg | grep vcxl | grep netmap" to verify that the interface has netmap queues. 4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... . One major improvement is that the netmap interface has a normal data path as expected. 5) Just ignore the cxl interfaces if you want to use netmap only. No need to bring them up. The vcxl interfaces are completely independent and everything should just work. --------------------- Approved by: re@ (gjb@) Relnotes: Yes Sponsored by: Chelsio Communications
2016-06-23 02:53:00 +00:00
{
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
struct sge_nm_rxq *nm_rxq = arg;
cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main cxgbe/cxl interfaces and tunables related to them are not affected by any of this and will continue to operate as usual. The driver used to create an additional 'n' interface for every cxgbe/cxl interface if "device netmap" was in the kernel. The 'n' interface shared the wire with the main interface but was otherwise autonomous (with its own MAC address, etc.). It did not have normal tx/rx but had a specialized netmap-only data path. r291665 added another set of virtual interfaces (the 'v' interfaces) to the driver. These had normal tx/rx but no netmap support. This revision consolidates the features of both the interfaces into the 'v' interface which now has a normal data path, TOE support, and native netmap support. The 'v' interfaces need to be created explicitly with the hw.cxgbe.num_vis tunable. This means "device netmap" will not result in the automatic creation of any virtual interfaces. The following tunables can be used to override the default number of queues allocated for each 'v' interface. nofld* = 0 will disable TOE on the virtual interface and nnm* = 0 to will disable native netmap support. # number of normal NIC queues hw.cxgbe.ntxq_vi hw.cxgbe.nrxq_vi # number of TOE queues hw.cxgbe.nofldtxq_vi hw.cxgbe.nofldrxq_vi # number of netmap queues hw.cxgbe.nnmtxq_vi hw.cxgbe.nnmrxq_vi hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed. --- tl;dr version --- The workflow for netmap on cxgbe starting with FreeBSD 11 is: 1) "device netmap" in the kernel config. 2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll end up with multiple autonomous netmap-capable interfaces for every port. 3) "dmesg | grep vcxl | grep netmap" to verify that the interface has netmap queues. 4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... . One major improvement is that the netmap interface has a normal data path as expected. 5) Just ignore the cxl interfaces if you want to use netmap only. No need to bring them up. The vcxl interfaces are completely independent and everything should just work. --------------------- Approved by: re@ (gjb@) Relnotes: Yes Sponsored by: Chelsio Communications
2016-06-23 02:53:00 +00:00
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
if (atomic_cmpset_int(&nm_rxq->nm_state, NM_ON, NM_BUSY)) {
service_nm_rxq(nm_rxq);
cxgbe(4): Be explicit about ignoring the return value of cmpset in some cases. Reported by: Coverity (CIDs 1009398, 1009400, 1009401, 1357325, 1394783). All false positives. Sponsored by: Chelsio Communications
2018-08-21 23:33:38 +00:00
(void) atomic_cmpset_int(&nm_rxq->nm_state, NM_BUSY, NM_ON);
cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main cxgbe/cxl interfaces and tunables related to them are not affected by any of this and will continue to operate as usual. The driver used to create an additional 'n' interface for every cxgbe/cxl interface if "device netmap" was in the kernel. The 'n' interface shared the wire with the main interface but was otherwise autonomous (with its own MAC address, etc.). It did not have normal tx/rx but had a specialized netmap-only data path. r291665 added another set of virtual interfaces (the 'v' interfaces) to the driver. These had normal tx/rx but no netmap support. This revision consolidates the features of both the interfaces into the 'v' interface which now has a normal data path, TOE support, and native netmap support. The 'v' interfaces need to be created explicitly with the hw.cxgbe.num_vis tunable. This means "device netmap" will not result in the automatic creation of any virtual interfaces. The following tunables can be used to override the default number of queues allocated for each 'v' interface. nofld* = 0 will disable TOE on the virtual interface and nnm* = 0 to will disable native netmap support. # number of normal NIC queues hw.cxgbe.ntxq_vi hw.cxgbe.nrxq_vi # number of TOE queues hw.cxgbe.nofldtxq_vi hw.cxgbe.nofldrxq_vi # number of netmap queues hw.cxgbe.nnmtxq_vi hw.cxgbe.nnmrxq_vi hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed. --- tl;dr version --- The workflow for netmap on cxgbe starting with FreeBSD 11 is: 1) "device netmap" in the kernel config. 2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll end up with multiple autonomous netmap-capable interfaces for every port. 3) "dmesg | grep vcxl | grep netmap" to verify that the interface has netmap queues. 4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... . One major improvement is that the netmap interface has a normal data path as expected. 5) Just ignore the cxl interfaces if you want to use netmap only. No need to bring them up. The vcxl interfaces are completely independent and everything should just work. --------------------- Approved by: re@ (gjb@) Relnotes: Yes Sponsored by: Chelsio Communications
2016-06-23 02:53:00 +00:00
}
}
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
/*
* Interrupt handler for vectors shared between NIC and netmap rx queues.
*/
void
t4_vi_intr(void *arg)
cxgbe: Add tunables to control the number of LRO entries and the number of rx mbufs that should be presorted before LRO. There is no change in default behavior. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-17 09:00:20 +00:00
{
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
struct irq *irq = arg;
cxgbe: Add tunables to control the number of LRO entries and the number of rx mbufs that should be presorted before LRO. There is no change in default behavior. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-17 09:00:20 +00:00
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
MPASS(irq->nm_rxq != NULL);
t4_nm_intr(irq->nm_rxq);
MPASS(irq->rxq != NULL);
t4_intr(irq->rxq);
cxgbe: Add tunables to control the number of LRO entries and the number of rx mbufs that should be presorted before LRO. There is no change in default behavior. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-17 09:00:20 +00:00
}
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
#endif
cxgbe: Add tunables to control the number of LRO entries and the number of rx mbufs that should be presorted before LRO. There is no change in default behavior. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-17 09:00:20 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
/*
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
* Deals with interrupts on an iq-only (no freelist) queue.
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
*/
static int
service_iq(struct sge_iq *iq, int budget)
{
struct sge_iq *q;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct adapter *sc = iq->adapter;
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
struct iq_desc *d = &iq->desc[iq->cidx];
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
int ndescs = 0, limit;
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
int rsp_type;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
uint32_t lq;
STAILQ_HEAD(, sge_iq) iql = STAILQ_HEAD_INITIALIZER(iql);
KASSERT(iq->state == IQS_BUSY, ("%s: iq %p not BUSY", __func__, iq));
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
KASSERT((iq->flags & IQ_HAS_FL) == 0,
("%s: called for iq %p with fl (iq->flags 0x%x)", __func__, iq,
iq->flags));
MPASS((iq->flags & IQ_ADJ_CREDIT) == 0);
MPASS((iq->flags & IQ_LRO_ENABLED) == 0);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
limit = budget ? budget : iq->qsize / 16;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
/*
* We always come back and check the descriptor ring for new indirect
* interrupts and other responses after running a single handler.
*/
for (;;) {
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
while ((d->rsp.u.type_gen & F_RSPD_GEN) == iq->gen) {
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
rmb();
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
rsp_type = G_RSPD_TYPE(d->rsp.u.type_gen);
lq = be32toh(d->rsp.pldbuflen_qid);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
switch (rsp_type) {
case X_RSPD_TYPE_FLBUF:
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
panic("%s: data for an iq (%p) with no freelist",
__func__, iq);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
/* NOTREACHED */
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
case X_RSPD_TYPE_CPL:
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
KASSERT(d->rss.opcode < NUM_CPL_CMDS,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
("%s: bad opcode %02x.", __func__,
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
d->rss.opcode));
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
t4_cpl_handler[d->rss.opcode](iq, &d->rss, NULL);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
break;
case X_RSPD_TYPE_INTR:
Add hooks in base cxgbe(4) for the iWARP upper-layer driver. Update a couple of assertions in the TOE driver as well.
2013-08-28 20:45:45 +00:00
/*
* There are 1K interrupt-capable queues (qids 0
* through 1023). A response type indicating a
* forwarded interrupt with a qid >= 1K is an
* iWARP async notification.
*/
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
if (__predict_true(lq >= 1024)) {
t4_an_handler(iq, &d->rsp);
break;
}
Add hooks in base cxgbe(4) for the iWARP upper-layer driver. Update a couple of assertions in the TOE driver as well.
2013-08-28 20:45:45 +00:00
Track the base absolute ID of ingress and egress queues. Use this to map an absolute queue ID to a logical queue ID in interrupt handlers. For the regular cxgbe/cxl drivers this should be a no-op as the base absolute ID should be zero. VF devices have a non-zero base absolute ID and require this change. While here, export the absolute ID of egress queues via a sysctl. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7446
2016-08-09 17:49:42 +00:00
q = sc->sge.iqmap[lq - sc->sge.iq_start -
sc->sge.iq_base];
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (atomic_cmpset_int(&q->state, IQS_IDLE,
IQS_BUSY)) {
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
if (service_iq_fl(q, q->qsize / 16) == 0) {
cxgbe(4): Be explicit about ignoring the return value of cmpset in some cases. Reported by: Coverity (CIDs 1009398, 1009400, 1009401, 1357325, 1394783). All false positives. Sponsored by: Chelsio Communications
2018-08-21 23:33:38 +00:00
(void) atomic_cmpset_int(&q->state,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
IQS_BUSY, IQS_IDLE);
} else {
STAILQ_INSERT_TAIL(&iql, q,
link);
}
}
break;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
default:
Add hooks in base cxgbe(4) for the iWARP upper-layer driver. Update a couple of assertions in the TOE driver as well.
2013-08-28 20:45:45 +00:00
KASSERT(0,
("%s: illegal response type %d on iq %p",
__func__, rsp_type, iq));
log(LOG_ERR,
"%s: illegal response type %d on iq %p",
device_get_nameunit(sc->dev), rsp_type, iq);
- Updated TOE support in the kernel. - Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs. These are available as t3_tom and t4_tom modules that augment cxgb(4) and cxgbe(4) respectively. The cxgb/cxgbe drivers continue to work as usual with or without these extra features. - iWARP driver for Terminator 3 ASIC (kernel verbs). T4 iWARP in the works and will follow soon. Build-tested with make universe. 30s overview ============ What interfaces support TCP offload? Look for TOE4 and/or TOE6 in the capabilities of an interface: # ifconfig -m | grep TOE Enable/disable TCP offload on an interface (just like any other ifnet capability): # ifconfig cxgbe0 toe # ifconfig cxgbe0 -toe Which connections are offloaded? Look for toe4 and/or toe6 in the output of netstat and sockstat: # netstat -np tcp | grep toe # sockstat -46c | grep toe Reviewed by: bz, gnn Sponsored by: Chelsio communications. MFC after: ~3 months (after 9.1, and after ensuring MFC is feasible)
2012-06-19 07:34:13 +00:00
break;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
d++;
if (__predict_false(++iq->cidx == iq->sidx)) {
iq->cidx = 0;
iq->gen ^= F_RSPD_GEN;
d = &iq->desc[0];
}
if (__predict_false(++ndescs == limit)) {
Store the offset of the KDOORBELL and GTS registers in the softc. VF devices use a different register layout than PF devices. Storing the offset in a value in the softc allows code to be shared between the PF and VF drivers. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7389
2016-08-01 22:39:51 +00:00
t4_write_reg(sc, sc->sge_gts_reg,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
V_CIDXINC(ndescs) |
V_INGRESSQID(iq->cntxt_id) |
V_SEINTARM(V_QINTR_TIMER_IDX(X_TIMERREG_UPDATE_CIDX)));
ndescs = 0;
cxgbe(4): Properly account for the freelist buffers used when returning early from service_iq due to a budget restriction. This fixes a potential rx hang when using INTx. MFC after: 3 days
2014-06-05 00:38:32 +00:00
if (budget) {
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
return (EINPROGRESS);
cxgbe(4): Properly account for the freelist buffers used when returning early from service_iq due to a budget restriction. This fixes a potential rx hang when using INTx. MFC after: 3 days
2014-06-05 00:38:32 +00:00
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (STAILQ_EMPTY(&iql))
break;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
/*
* Process the head only, and send it to the back of the list if
* it's still not done.
*/
q = STAILQ_FIRST(&iql);
STAILQ_REMOVE_HEAD(&iql, link);
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
if (service_iq_fl(q, q->qsize / 8) == 0)
cxgbe(4): Be explicit about ignoring the return value of cmpset in some cases. Reported by: Coverity (CIDs 1009398, 1009400, 1009401, 1357325, 1394783). All false positives. Sponsored by: Chelsio Communications
2018-08-21 23:33:38 +00:00
(void) atomic_cmpset_int(&q->state, IQS_BUSY, IQS_IDLE);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
else
STAILQ_INSERT_TAIL(&iql, q, link);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
t4_write_reg(sc, sc->sge_gts_reg, V_CIDXINC(ndescs) |
V_INGRESSQID((u32)iq->cntxt_id) | V_SEINTARM(iq->intr_params));
return (0);
}
static inline int
sort_before_lro(struct lro_ctrl *lro)
{
return (lro->lro_mbuf_max != 0);
}
cxgbe(4): Replace T4_PKT_TIMESTAMP with something slightly less hackish.
2018-08-18 04:23:51 +00:00
static inline uint64_t
last_flit_to_ns(struct adapter *sc, uint64_t lf)
{
uint64_t n = be64toh(lf) & 0xfffffffffffffff; /* 60b, not 64b. */
if (n > UINT64_MAX / 1000000)
return (n / sc->params.vpd.cclk * 1000000);
else
return (n * 1000000 / sc->params.vpd.cclk);
}
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
static inline void
move_to_next_rxbuf(struct sge_fl *fl)
{
fl->rx_offset = 0;
if (__predict_false((++fl->cidx & 7) == 0)) {
uint16_t cidx = fl->cidx >> 3;
if (__predict_false(cidx == fl->sidx))
fl->cidx = cidx = 0;
fl->hw_cidx = cidx;
}
}
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
/*
* Deals with interrupts on an iq+fl queue.
*/
static int
service_iq_fl(struct sge_iq *iq, int budget)
{
struct sge_rxq *rxq = iq_to_rxq(iq);
struct sge_fl *fl;
struct adapter *sc = iq->adapter;
struct iq_desc *d = &iq->desc[iq->cidx];
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
int ndescs, limit;
int rsp_type, starved;
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
uint32_t lq;
uint16_t fl_hw_cidx;
struct mbuf *m0;
#if defined(INET) || defined(INET6)
const struct timeval lro_timeout = {0, sc->lro_timeout};
struct lro_ctrl *lro = &rxq->lro;
#endif
KASSERT(iq->state == IQS_BUSY, ("%s: iq %p not BUSY", __func__, iq));
MPASS(iq->flags & IQ_HAS_FL);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
ndescs = 0;
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
#if defined(INET) || defined(INET6)
if (iq->flags & IQ_ADJ_CREDIT) {
MPASS(sort_before_lro(lro));
iq->flags &= ~IQ_ADJ_CREDIT;
if ((d->rsp.u.type_gen & F_RSPD_GEN) != iq->gen) {
tcp_lro_flush_all(lro);
t4_write_reg(sc, sc->sge_gts_reg, V_CIDXINC(1) |
V_INGRESSQID((u32)iq->cntxt_id) |
V_SEINTARM(iq->intr_params));
return (0);
}
ndescs = 1;
}
#else
MPASS((iq->flags & IQ_ADJ_CREDIT) == 0);
#endif
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
limit = budget ? budget : iq->qsize / 16;
fl = &rxq->fl;
fl_hw_cidx = fl->hw_cidx; /* stable snapshot */
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
while ((d->rsp.u.type_gen & F_RSPD_GEN) == iq->gen) {
rmb();
m0 = NULL;
rsp_type = G_RSPD_TYPE(d->rsp.u.type_gen);
lq = be32toh(d->rsp.pldbuflen_qid);
switch (rsp_type) {
case X_RSPD_TYPE_FLBUF:
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if (lq & F_RSPD_NEWBUF) {
if (fl->rx_offset > 0)
move_to_next_rxbuf(fl);
lq = G_RSPD_LEN(lq);
}
if (IDXDIFF(fl->hw_cidx, fl_hw_cidx, fl->sidx) > 4) {
FL_LOCK(fl);
refill_fl(sc, fl, 64);
FL_UNLOCK(fl);
fl_hw_cidx = fl->hw_cidx;
}
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
cxgbe(4): Treat NIC rx as special and run its handler directly and not via the t4_cpl_handler dispatch table. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:01:35 +00:00
if (d->rss.opcode == CPL_RX_PKT) {
if (__predict_true(eth_rx(sc, rxq, d, lq) == 0))
break;
goto out;
}
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
m0 = get_fl_payload(sc, fl, lq);
if (__predict_false(m0 == NULL))
goto out;
cxgbe(4): Replace T4_PKT_TIMESTAMP with something slightly less hackish.
2018-08-18 04:23:51 +00:00
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
/* fall through */
case X_RSPD_TYPE_CPL:
KASSERT(d->rss.opcode < NUM_CPL_CMDS,
("%s: bad opcode %02x.", __func__, d->rss.opcode));
t4_cpl_handler[d->rss.opcode](iq, &d->rss, m0);
break;
case X_RSPD_TYPE_INTR:
/*
* There are 1K interrupt-capable queues (qids 0
* through 1023). A response type indicating a
* forwarded interrupt with a qid >= 1K is an
* iWARP async notification. That is the only
* acceptable indirect interrupt on this queue.
*/
if (__predict_false(lq < 1024)) {
panic("%s: indirect interrupt on iq_fl %p "
"with qid %u", __func__, iq, lq);
}
t4_an_handler(iq, &d->rsp);
break;
default:
KASSERT(0, ("%s: illegal response type %d on iq %p",
__func__, rsp_type, iq));
log(LOG_ERR, "%s: illegal response type %d on iq %p",
device_get_nameunit(sc->dev), rsp_type, iq);
break;
}
d++;
if (__predict_false(++iq->cidx == iq->sidx)) {
iq->cidx = 0;
iq->gen ^= F_RSPD_GEN;
d = &iq->desc[0];
}
if (__predict_false(++ndescs == limit)) {
t4_write_reg(sc, sc->sge_gts_reg, V_CIDXINC(ndescs) |
V_INGRESSQID(iq->cntxt_id) |
V_SEINTARM(V_QINTR_TIMER_IDX(X_TIMERREG_UPDATE_CIDX)));
#if defined(INET) || defined(INET6)
if (iq->flags & IQ_LRO_ENABLED &&
!sort_before_lro(lro) &&
sc->lro_timeout != 0) {
tcp_lro_flush_inactive(lro, &lro_timeout);
}
#endif
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if (budget)
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
return (EINPROGRESS);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
ndescs = 0;
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
}
}
out:
cxgbe(4): support for IPv6 TSO and LRO. Submitted by: bz (this is a modified version of that patch)
2012-06-29 19:51:06 +00:00
#if defined(INET) || defined(INET6)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (iq->flags & IQ_LRO_ENABLED) {
cxgbe: Add tunables to control the number of LRO entries and the number of rx mbufs that should be presorted before LRO. There is no change in default behavior. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-17 09:00:20 +00:00
if (ndescs > 0 && lro->lro_mbuf_count > 8) {
MPASS(sort_before_lro(lro));
/* hold back one credit and don't flush LRO state */
iq->flags |= IQ_ADJ_CREDIT;
ndescs--;
} else {
tcp_lro_flush_all(lro);
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
#endif
Store the offset of the KDOORBELL and GTS registers in the softc. VF devices use a different register layout than PF devices. Storing the offset in a value in the softc allows code to be shared between the PF and VF drivers. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7389
2016-08-01 22:39:51 +00:00
t4_write_reg(sc, sc->sge_gts_reg, V_CIDXINC(ndescs) |
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
V_INGRESSQID((u32)iq->cntxt_id) | V_SEINTARM(iq->intr_params));
cxgbe(4): Create two variants of service_iq, one for queues with freelists and one for those without. MFH: 3 weeks Sponsored by: Chelsio Communications
2018-08-11 04:55:47 +00:00
FL_LOCK(fl);
starved = refill_fl(sc, fl, 64);
FL_UNLOCK(fl);
if (__predict_false(starved != 0))
add_fl_to_sfl(sc, fl);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
return (0);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
static inline struct cluster_metadata *
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
cl_metadata(struct fl_sdesc *sd)
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
{
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
return ((void *)(sd->cl + sd->moff));
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
}
All mbuf external free functions never fail, so let them be void. Sponsored by: Nginx, Inc.
2014-07-11 13:58:48 +00:00
static void
Shorten list of arguments to mbuf external storage freeing function. All of these arguments are stored in m_ext, so there is no reason to pass them in the argument list. Not all functions need the second argument, some don't even need the first one. The second argument lives in next cache line, so not dereferencing it is a performance gain. This was discovered in sendfile(2), which will be covered by next commits. The second goal of this commit is to bring even more flexibility to m_ext mbufs, allowing to create more fields in m_ext, opaque to the generic mbuf code, and potentially set and dereferenced by subsystems. Reviewed by: gallatin, kbowling Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 20:35:31 +00:00
rxb_free(struct mbuf *m)
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
{
cxgbe(4): Avoid ext_arg2 in rxb_free. ext_arg2 is the only item in the third cacheline in an mbuf and could be cold by the time rxb_free runs. Put the information needed by rxb_free in the same line as the refcount, which is very likely to be hot given that rxb_free runs when the refcount is decremented and reaches 0. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-03 23:50:29 +00:00
struct cluster_metadata *clm = m->m_ext.ext_arg1;
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
cxgbe(4): Avoid ext_arg2 in rxb_free. ext_arg2 is the only item in the third cacheline in an mbuf and could be cold by the time rxb_free runs. Put the information needed by rxb_free in the same line as the refcount, which is very likely to be hot given that rxb_free runs when the refcount is decremented and reaches 0. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-03 23:50:29 +00:00
uma_zfree(clm->zone, clm->cl);
cxgbe(4): Keep track of the clusters that have to be freed by the custom free routine (rxb_free) in the driver. Fail MOD_UNLOAD with EBUSY if any such cluster has been handed up to the kernel but hasn't been freed yet. This prevents a panic later when the cluster finally needs to be freed but rxb_free is gone from the kernel. MFC after: 1 week
2014-07-23 22:29:22 +00:00
counter_u64_add(extfree_rels, 1);
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
/*
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
* The mbuf returned comes from zone_muf and carries the payload in one of these
* ways
* a) complete frame inside the mbuf
* b) m_cljset (for clusters without metadata)
* d) m_extaddref (cluster with metadata)
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
*/
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
static struct mbuf *
cxgbe(4): allow the driver to use rx buffers that do not end on a pack boundary. MFC after: 2 weeks
2014-12-06 01:47:38 +00:00
get_scatter_segment(struct adapter *sc, struct sge_fl *fl, int fr_offset,
int remaining)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
struct mbuf *m;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct fl_sdesc *sd = &fl->sdesc[fl->cidx];
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
struct rx_buf_info *rxb = &sc->sge.rx_buf_info[sd->zidx];
struct cluster_metadata *clm;
cxgbe(4): allow the driver to use rx buffers that do not end on a pack boundary. MFC after: 2 weeks
2014-12-06 01:47:38 +00:00
int len, blen;
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
caddr_t payload;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
if (fl->flags & FL_BUF_PACKING) {
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
u_int l, pad;
blen = rxb->size2 - fl->rx_offset; /* max possible in this buf */
len = min(remaining, blen);
payload = sd->cl + fl->rx_offset;
cxgbe(4): allow the driver to use rx buffers that do not end on a pack boundary. MFC after: 2 weeks
2014-12-06 01:47:38 +00:00
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
l = fr_offset + len;
pad = roundup2(l, fl->buf_boundary) - l;
if (fl->rx_offset + len + pad < rxb->size2)
cxgbe(4): allow the driver to use rx buffers that do not end on a pack boundary. MFC after: 2 weeks
2014-12-06 01:47:38 +00:00
blen = len + pad;
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
MPASS(fl->rx_offset + blen <= rxb->size2);
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
} else {
MPASS(fl->rx_offset == 0); /* not packing */
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
blen = rxb->size1;
len = min(remaining, blen);
payload = sd->cl;
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
}
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if (fr_offset == 0) {
m = m_gethdr(M_NOWAIT, MT_DATA);
if (__predict_false(m == NULL))
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
return (NULL);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
m->m_pkthdr.len = remaining;
} else {
m = m_get(M_NOWAIT, MT_DATA);
if (__predict_false(m == NULL))
return (NULL);
}
m->m_len = len;
cxgbe(4): Replace T4_PKT_TIMESTAMP with something slightly less hackish.
2018-08-18 04:23:51 +00:00
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if (sc->sc_do_rxcopy && len < RX_COPY_THRESHOLD) {
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
/* copy data to mbuf */
bcopy(payload, mtod(m, caddr_t), len);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if (fl->flags & FL_BUF_PACKING) {
fl->rx_offset += blen;
MPASS(fl->rx_offset <= rxb->size2);
if (fl->rx_offset < rxb->size2)
return (m); /* without advancing the cidx */
}
} else if (fl->flags & FL_BUF_PACKING) {
clm = cl_metadata(sd);
cxgbe(4): Initialize the rx buffer's metadata on first-use and not on allocation. refill_fl doesn't touch any part of a freshly allocated cluster after this change. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-03 23:25:12 +00:00
if (sd->nmbuf++ == 0) {
clm->refcount = 1;
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
clm->zone = rxb->zone;
cxgbe(4): Avoid ext_arg2 in rxb_free. ext_arg2 is the only item in the third cacheline in an mbuf and could be cold by the time rxb_free runs. Put the information needed by rxb_free in the same line as the refcount, which is very likely to be hot given that rxb_free runs when the refcount is decremented and reaches 0. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-03 23:50:29 +00:00
clm->cl = sd->cl;
cxgbe(4): Initialize the rx buffer's metadata on first-use and not on allocation. refill_fl doesn't touch any part of a freshly allocated cluster after this change. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-03 23:25:12 +00:00
counter_u64_add(extfree_refs, 1);
}
cxgbe(4): Avoid ext_arg2 in rxb_free. ext_arg2 is the only item in the third cacheline in an mbuf and could be cold by the time rxb_free runs. Put the information needed by rxb_free in the same line as the refcount, which is very likely to be hot given that rxb_free runs when the refcount is decremented and reaches 0. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-03 23:50:29 +00:00
m_extaddref(m, payload, blen, &clm->refcount, rxb_free, clm,
NULL);
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
cxgbe(4): allow the driver to use rx buffers that do not end on a pack boundary. MFC after: 2 weeks
2014-12-06 01:47:38 +00:00
fl->rx_offset += blen;
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
MPASS(fl->rx_offset <= rxb->size2);
if (fl->rx_offset < rxb->size2)
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
return (m); /* without advancing the cidx */
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
} else {
m_cljset(m, sd->cl, rxb->type);
sd->cl = NULL; /* consumed, not a recycle candidate */
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
}
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
move_to_next_rxbuf(fl);
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
return (m);
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
}
static struct mbuf *
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
get_fl_payload(struct adapter *sc, struct sge_fl *fl, const u_int plen)
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
{
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
struct mbuf *m0, *m, **pnext;
cxgbe(4): allow the driver to use rx buffers that do not end on a pack boundary. MFC after: 2 weeks
2014-12-06 01:47:38 +00:00
u_int remaining;
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
if (__predict_false(fl->flags & FL_BUF_RESUME)) {
cxgbe(4): Fix a NULL dereference when the very first call to get_scatter_segment() in get_fl_payload() fails. While here, fix the code to adjust fl_bufs_used when a failure occurs for any other scatter segment. MFC after: 3 days
2014-05-30 22:59:45 +00:00
M_ASSERTPKTHDR(fl->m0);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
MPASS(fl->m0->m_pkthdr.len == plen);
MPASS(fl->remaining < plen);
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
m0 = fl->m0;
pnext = fl->pnext;
cxgbe(4): allow the driver to use rx buffers that do not end on a pack boundary. MFC after: 2 weeks
2014-12-06 01:47:38 +00:00
remaining = fl->remaining;
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
fl->flags &= ~FL_BUF_RESUME;
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
goto get_segment;
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
/*
* Payload starts at rx_offset in the current hw buffer. Its length is
* 'len' and it may span multiple hw buffers.
*/
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
m0 = get_scatter_segment(sc, fl, 0, plen);
cxgbe(4): Fix a NULL dereference when the very first call to get_scatter_segment() in get_fl_payload() fails. While here, fix the code to adjust fl_bufs_used when a failure occurs for any other scatter segment. MFC after: 3 days
2014-05-30 22:59:45 +00:00
if (m0 == NULL)
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
return (NULL);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
remaining = plen - m0->m_len;
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
pnext = &m0->m_next;
cxgbe(4): allow the driver to use rx buffers that do not end on a pack boundary. MFC after: 2 weeks
2014-12-06 01:47:38 +00:00
while (remaining > 0) {
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
get_segment:
MPASS(fl->rx_offset == 0);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
m = get_scatter_segment(sc, fl, plen - remaining, remaining);
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
if (__predict_false(m == NULL)) {
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
fl->m0 = m0;
fl->pnext = pnext;
cxgbe(4): allow the driver to use rx buffers that do not end on a pack boundary. MFC after: 2 weeks
2014-12-06 01:47:38 +00:00
fl->remaining = remaining;
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
fl->flags |= FL_BUF_RESUME;
return (NULL);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
*pnext = m;
pnext = &m->m_next;
cxgbe(4): allow the driver to use rx buffers that do not end on a pack boundary. MFC after: 2 weeks
2014-12-06 01:47:38 +00:00
remaining -= m->m_len;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
*pnext = NULL;
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
cxgbe: get_fl_payload returns a header mbuf when successful. MFC after: 3 days
2015-06-23 05:55:13 +00:00
M_ASSERTPKTHDR(m0);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
return (m0);
}
cxgbe(4): Add pfil(9) hooks to the driver's rx. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:09:02 +00:00
static int
skip_scatter_segment(struct adapter *sc, struct sge_fl *fl, int fr_offset,
int remaining)
{
struct fl_sdesc *sd = &fl->sdesc[fl->cidx];
struct rx_buf_info *rxb = &sc->sge.rx_buf_info[sd->zidx];
int len, blen;
if (fl->flags & FL_BUF_PACKING) {
u_int l, pad;
blen = rxb->size2 - fl->rx_offset; /* max possible in this buf */
len = min(remaining, blen);
l = fr_offset + len;
pad = roundup2(l, fl->buf_boundary) - l;
if (fl->rx_offset + len + pad < rxb->size2)
blen = len + pad;
fl->rx_offset += blen;
MPASS(fl->rx_offset <= rxb->size2);
if (fl->rx_offset < rxb->size2)
return (len); /* without advancing the cidx */
} else {
MPASS(fl->rx_offset == 0); /* not packing */
blen = rxb->size1;
len = min(remaining, blen);
}
move_to_next_rxbuf(fl);
return (len);
}
static inline void
skip_fl_payload(struct adapter *sc, struct sge_fl *fl, int plen)
{
int remaining, fr_offset, len;
fr_offset = 0;
remaining = plen;
while (remaining > 0) {
len = skip_scatter_segment(sc, fl, fr_offset, remaining);
fr_offset += len;
remaining -= len;
}
}
static inline int
get_segment_len(struct adapter *sc, struct sge_fl *fl, int plen)
{
int len;
struct fl_sdesc *sd = &fl->sdesc[fl->cidx];
struct rx_buf_info *rxb = &sc->sge.rx_buf_info[sd->zidx];
if (fl->flags & FL_BUF_PACKING)
len = rxb->size2 - fl->rx_offset;
else
len = rxb->size1;
return (min(plen, len));
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static int
cxgbe(4): Treat NIC rx as special and run its handler directly and not via the t4_cpl_handler dispatch table. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:01:35 +00:00
eth_rx(struct adapter *sc, struct sge_rxq *rxq, const struct iq_desc *d,
u_int plen)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
{
cxgbe(4): Treat NIC rx as special and run its handler directly and not via the t4_cpl_handler dispatch table. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:01:35 +00:00
struct mbuf *m0;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct ifnet *ifp = rxq->ifp;
cxgbe(4): Treat NIC rx as special and run its handler directly and not via the t4_cpl_handler dispatch table. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:01:35 +00:00
struct sge_fl *fl = &rxq->fl;
cxgbe(4): Add pfil(9) hooks to the driver's rx. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:09:02 +00:00
struct vi_info *vi = ifp->if_softc;
cxgbe(4): Treat NIC rx as special and run its handler directly and not via the t4_cpl_handler dispatch table. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:01:35 +00:00
const struct cpl_rx_pkt *cpl;
cxgbe(4): support for IPv6 TSO and LRO. Submitted by: bz (this is a modified version of that patch)
2012-06-29 19:51:06 +00:00
#if defined(INET) || defined(INET6)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct lro_ctrl *lro = &rxq->lro;
#endif
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
uint16_t err_vec, tnl_type, tnlhdr_len;
cxgbe(4): provide the exact RSS hash type instead of a catch-all value to the upper layers.
2015-03-26 18:45:51 +00:00
static const int sw_hashtype[4][2] = {
{M_HASHTYPE_NONE, M_HASHTYPE_NONE},
{M_HASHTYPE_RSS_IPV4, M_HASHTYPE_RSS_IPV6},
{M_HASHTYPE_RSS_TCP_IPV4, M_HASHTYPE_RSS_TCP_IPV6},
{M_HASHTYPE_RSS_UDP_IPV4, M_HASHTYPE_RSS_UDP_IPV6},
};
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
static const int sw_csum_flags[2][2] = {
{
/* IP, inner IP */
CSUM_ENCAP_VXLAN |
CSUM_L3_CALC | CSUM_L3_VALID |
CSUM_L4_CALC | CSUM_L4_VALID |
CSUM_INNER_L3_CALC | CSUM_INNER_L3_VALID |
CSUM_INNER_L4_CALC | CSUM_INNER_L4_VALID,
/* IP, inner IP6 */
CSUM_ENCAP_VXLAN |
CSUM_L3_CALC | CSUM_L3_VALID |
CSUM_L4_CALC | CSUM_L4_VALID |
CSUM_INNER_L4_CALC | CSUM_INNER_L4_VALID,
},
{
/* IP6, inner IP */
CSUM_ENCAP_VXLAN |
CSUM_L4_CALC | CSUM_L4_VALID |
CSUM_INNER_L3_CALC | CSUM_INNER_L3_VALID |
CSUM_INNER_L4_CALC | CSUM_INNER_L4_VALID,
/* IP6, inner IP6 */
CSUM_ENCAP_VXLAN |
CSUM_L4_CALC | CSUM_L4_VALID |
CSUM_INNER_L4_CALC | CSUM_INNER_L4_VALID,
},
};
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Treat NIC rx as special and run its handler directly and not via the t4_cpl_handler dispatch table. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:01:35 +00:00
MPASS(plen > sc->params.sge.fl_pktshift);
cxgbe(4): Add pfil(9) hooks to the driver's rx. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:09:02 +00:00
if (vi->pfil != NULL && PFIL_HOOKED_IN(vi->pfil) &&
__predict_true((fl->flags & FL_BUF_RESUME) == 0)) {
struct fl_sdesc *sd = &fl->sdesc[fl->cidx];
caddr_t frame;
int rc, slen;
slen = get_segment_len(sc, fl, plen) -
sc->params.sge.fl_pktshift;
frame = sd->cl + fl->rx_offset + sc->params.sge.fl_pktshift;
CURVNET_SET_QUIET(ifp->if_vnet);
rc = pfil_run_hooks(vi->pfil, frame, ifp,
slen | PFIL_MEMPTR | PFIL_IN, NULL);
CURVNET_RESTORE();
if (rc == PFIL_DROPPED || rc == PFIL_CONSUMED) {
skip_fl_payload(sc, fl, plen);
return (0);
}
if (rc == PFIL_REALLOCED) {
skip_fl_payload(sc, fl, plen);
m0 = pfil_mem2mbuf(frame);
goto have_mbuf;
}
}
cxgbe(4): Treat NIC rx as special and run its handler directly and not via the t4_cpl_handler dispatch table. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:01:35 +00:00
m0 = get_fl_payload(sc, fl, plen);
if (__predict_false(m0 == NULL))
return (ENOMEM);
Tweaks for rx: - everything related to LRO should be in #ifdef INET blocks - reorder sge_iq's fields so that the most frequently used are all together - pull all rx code into t4_intr_data directly - let go of the ingress queue lock when passing up data - refill the freelist only if it is short of at least 32 buffers
2011-03-05 03:42:03 +00:00
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
m0->m_pkthdr.len -= sc->params.sge.fl_pktshift;
m0->m_len -= sc->params.sge.fl_pktshift;
m0->m_data += sc->params.sge.fl_pktshift;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Add pfil(9) hooks to the driver's rx. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:09:02 +00:00
have_mbuf:
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
m0->m_pkthdr.rcvif = ifp;
cxgbe(4): Treat NIC rx as special and run its handler directly and not via the t4_cpl_handler dispatch table. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:01:35 +00:00
M_HASHTYPE_SET(m0, sw_hashtype[d->rss.hash_type][d->rss.ipv6]);
m0->m_pkthdr.flowid = be32toh(d->rss.hash_val);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Treat NIC rx as special and run its handler directly and not via the t4_cpl_handler dispatch table. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:01:35 +00:00
cpl = (const void *)(&d->rss + 1);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (sc->params.tp.rx_pkt_encap) {
const uint16_t ev = be16toh(cpl->err_vec);
err_vec = G_T6_COMPR_RXERR_VEC(ev);
tnl_type = G_T6_RX_TNL_TYPE(ev);
tnlhdr_len = G_T6_RX_TNLHDR_LEN(ev);
} else {
err_vec = be16toh(cpl->err_vec);
tnl_type = 0;
tnlhdr_len = 0;
}
if (cpl->csum_calc && err_vec == 0) {
int ipv6 = !!(cpl->l2info & htobe32(F_RXF_IP6));
/* checksum(s) calculated and found to be correct. */
MPASS((cpl->l2info & htobe32(F_RXF_IP)) ^
(cpl->l2info & htobe32(F_RXF_IP6)));
m0->m_pkthdr.csum_data = be16toh(cpl->csum);
if (tnl_type == 0) {
if (!ipv6 && ifp->if_capenable & IFCAP_RXCSUM) {
m0->m_pkthdr.csum_flags = CSUM_L3_CALC |
CSUM_L3_VALID | CSUM_L4_CALC |
CSUM_L4_VALID;
} else if (ipv6 && ifp->if_capenable & IFCAP_RXCSUM_IPV6) {
m0->m_pkthdr.csum_flags = CSUM_L4_CALC |
CSUM_L4_VALID;
}
cxgbe(4): support for IPv6 hardware checksumming (rx and tx).
2012-06-29 16:50:52 +00:00
rxq->rxcsum++;
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
} else {
MPASS(tnl_type == RX_PKT_TNL_TYPE_VXLAN);
if (__predict_false(cpl->ip_frag)) {
/*
* csum_data is for the inner frame (which is an
* IP fragment) and is not 0xffff. There is no
* way to pass the inner csum_data to the stack.
* We don't want the stack to use the inner
* csum_data to validate the outer frame or it
* will get rejected. So we fix csum_data here
* and let sw do the checksum of inner IP
* fragments.
*
* XXX: Need 32b for csum_data2 in an rx mbuf.
* Maybe stuff it into rcv_tstmp?
*/
m0->m_pkthdr.csum_data = 0xffff;
if (ipv6) {
m0->m_pkthdr.csum_flags = CSUM_L4_CALC |
CSUM_L4_VALID;
} else {
m0->m_pkthdr.csum_flags = CSUM_L3_CALC |
CSUM_L3_VALID | CSUM_L4_CALC |
CSUM_L4_VALID;
}
} else {
int outer_ipv6;
cxgbe(4): support for IPv6 hardware checksumming (rx and tx).
2012-06-29 16:50:52 +00:00
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
MPASS(m0->m_pkthdr.csum_data == 0xffff);
outer_ipv6 = tnlhdr_len >=
sizeof(struct ether_header) +
sizeof(struct ip6_hdr);
m0->m_pkthdr.csum_flags =
sw_csum_flags[outer_ipv6][ipv6];
}
rxq->vxlan_rxcsum++;
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (cpl->vlan_ex) {
m0->m_pkthdr.ether_vtag = be16toh(cpl->vlan);
m0->m_flags |= M_VLANTAG;
rxq->vlan_extraction++;
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Treat NIC rx as special and run its handler directly and not via the t4_cpl_handler dispatch table. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 01:01:35 +00:00
if (rxq->iq.flags & IQ_RX_TIMESTAMP) {
/*
* Fill up rcv_tstmp but do not set M_TSTMP.
* rcv_tstmp is not in the format that the
* kernel expects and we don't want to mislead
* it. For now this is only for custom code
* that knows how to interpret cxgbe's stamp.
*/
m0->m_pkthdr.rcv_tstmp =
last_flit_to_ns(sc, d->rsp.u.last_flit);
#ifdef notyet
m0->m_flags |= M_TSTMP;
#endif
}
Track TCP connection's NUMA domain in the inpcb Drivers can now pass up numa domain information via the mbuf numa domain field. This information is then used by TCP syncache_socket() to associate that information with the inpcb. The domain information is then fed back into transmitted mbufs in ip{6}_output(). This mechanism is nearly identical to what is done to track RSS hash values in the inp_flowid. Follow on changes will use this information for lacp egress port selection, binding TCP pacers to the appropriate NUMA domain, etc. Reviewed by: markj, kib, slavash, bz, scottl, jtl, tuexen Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20028
2019-04-25 15:37:28 +00:00
#ifdef NUMA
m0->m_pkthdr.numa_domain = ifp->if_numa_domain;
#endif
cxgbe(4): support for IPv6 TSO and LRO. Submitted by: bz (this is a modified version of that patch)
2012-06-29 19:51:06 +00:00
#if defined(INET) || defined(INET6)
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (rxq->iq.flags & IQ_LRO_ENABLED && tnl_type == 0 &&
cxgbe(4): Only checksummed TCP should be considered for LRO. This avoids the per-packet nanouptime in tcp_lro_rx for traffic that's not even TCP. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-03 23:06:42 +00:00
(M_HASHTYPE_GET(m0) == M_HASHTYPE_RSS_TCP_IPV4 ||
M_HASHTYPE_GET(m0) == M_HASHTYPE_RSS_TCP_IPV6)) {
cxgbe: Add tunables to control the number of LRO entries and the number of rx mbufs that should be presorted before LRO. There is no change in default behavior. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-17 09:00:20 +00:00
if (sort_before_lro(lro)) {
tcp_lro_queue_mbuf(lro, m0);
return (0); /* queued for sort, then LRO */
}
if (tcp_lro_rx(lro, m0, 0) == 0)
return (0); /* queued for LRO */
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
#endif
ifp->if_input(ifp, m0);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
return (0);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/*
* Must drain the wrq or make sure that someone else will.
*/
static void
wrq_tx_drain(void *arg, int n)
{
struct sge_wrq *wrq = arg;
struct sge_eq *eq = &wrq->eq;
EQ_LOCK(eq);
if (TAILQ_EMPTY(&wrq->incomplete_wrs) && !STAILQ_EMPTY(&wrq->wr_list))
drain_wrq_wr_list(wrq->adapter, wrq);
EQ_UNLOCK(eq);
}
static void
drain_wrq_wr_list(struct adapter *sc, struct sge_wrq *wrq)
{
struct sge_eq *eq = &wrq->eq;
u_int available, dbdiff; /* # of hardware descriptors */
u_int n;
struct wrqe *wr;
struct fw_eth_tx_pkt_wr *dst; /* any fw WR struct will do */
EQ_LOCK_ASSERT_OWNED(eq);
MPASS(TAILQ_EMPTY(&wrq->incomplete_wrs));
wr = STAILQ_FIRST(&wrq->wr_list);
MPASS(wr != NULL); /* Must be called with something useful to do */
cxgbe(4): Always dispatch all work requests that have been written to the descriptor ring before leaving drain_wrq_wr_list.
2016-04-12 22:11:29 +00:00
MPASS(eq->pidx == eq->dbidx);
dbdiff = 0;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
do {
eq->cidx = read_hw_cidx(eq);
if (eq->pidx == eq->cidx)
available = eq->sidx - 1;
else
available = IDXDIFF(eq->cidx, eq->pidx, eq->sidx) - 1;
MPASS(wr->wrq == wrq);
n = howmany(wr->wr_len, EQ_ESIZE);
if (available < n)
cxgbe(4): Always dispatch all work requests that have been written to the descriptor ring before leaving drain_wrq_wr_list.
2016-04-12 22:11:29 +00:00
break;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
dst = (void *)&eq->desc[eq->pidx];
if (__predict_true(eq->sidx - eq->pidx > n)) {
/* Won't wrap, won't end exactly at the status page. */
bcopy(&wr->wr[0], dst, wr->wr_len);
eq->pidx += n;
} else {
int first_portion = (eq->sidx - eq->pidx) * EQ_ESIZE;
bcopy(&wr->wr[0], dst, first_portion);
if (wr->wr_len > first_portion) {
bcopy(&wr->wr[first_portion], &eq->desc[0],
wr->wr_len - first_portion);
}
eq->pidx = n - (eq->sidx - eq->pidx);
}
cxgbe(4): Fixes to wrq stats. - Increment tx_wrs_copied in the correct place. - Add tx_wrs_sspace to the sysctl MIB. Sponsored by: Chelsio Communications
2016-09-19 17:16:51 +00:00
wrq->tx_wrs_copied++;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (available < eq->sidx / 4 &&
atomic_cmpset_int(&eq->equiq, 0, 1)) {
cxgbe(4): Use automatic cidx updates with ofld and ctrl queues. The bits that explicitly request cidx updates do not work reliably with all possible WRs that can be sent over the queue. The F_FW_WR_EQUIQ requests that still remain may also have to be replaced with explicit credit flush WRs in the future. MFC after: 2 days Sponsored by: Chelsio Communications
2018-10-22 23:06:23 +00:00
/*
* XXX: This is not 100% reliable with some
* types of WRs. But this is a very unusual
* situation for an ofld/ctrl queue anyway.
*/
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
dst->equiq_to_len16 |= htobe32(F_FW_WR_EQUIQ |
F_FW_WR_EQUEQ);
}
dbdiff += n;
if (dbdiff >= 16) {
ring_eq_db(sc, eq, dbdiff);
dbdiff = 0;
}
STAILQ_REMOVE_HEAD(&wrq->wr_list, link);
free_wrqe(wr);
MPASS(wrq->nwr_pending > 0);
wrq->nwr_pending--;
MPASS(wrq->ndesc_needed >= n);
wrq->ndesc_needed -= n;
} while ((wr = STAILQ_FIRST(&wrq->wr_list)) != NULL);
if (dbdiff)
ring_eq_db(sc, eq, dbdiff);
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
/*
* Doesn't fail. Holds on to work requests it can't send right away.
*/
- Updated TOE support in the kernel. - Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs. These are available as t3_tom and t4_tom modules that augment cxgb(4) and cxgbe(4) respectively. The cxgb/cxgbe drivers continue to work as usual with or without these extra features. - iWARP driver for Terminator 3 ASIC (kernel verbs). T4 iWARP in the works and will follow soon. Build-tested with make universe. 30s overview ============ What interfaces support TCP offload? Look for TOE4 and/or TOE6 in the capabilities of an interface: # ifconfig -m | grep TOE Enable/disable TCP offload on an interface (just like any other ifnet capability): # ifconfig cxgbe0 toe # ifconfig cxgbe0 -toe Which connections are offloaded? Look for toe4 and/or toe6 in the output of netstat and sockstat: # netstat -np tcp | grep toe # sockstat -46c | grep toe Reviewed by: bz, gnn Sponsored by: Chelsio communications. MFC after: ~3 months (after 9.1, and after ensuring MFC is feasible)
2012-06-19 07:34:13 +00:00
void
t4_wrq_tx_locked(struct adapter *sc, struct sge_wrq *wrq, struct wrqe *wr)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
{
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
#ifdef INVARIANTS
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sge_eq *eq = &wrq->eq;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
#endif
EQ_LOCK_ASSERT_OWNED(eq);
MPASS(wr != NULL);
MPASS(wr->wr_len > 0 && wr->wr_len <= SGE_MAX_WR_LEN);
MPASS((wr->wr_len & 0x7) == 0);
STAILQ_INSERT_TAIL(&wrq->wr_list, wr, link);
wrq->nwr_pending++;
wrq->ndesc_needed += howmany(wr->wr_len, EQ_ESIZE);
if (!TAILQ_EMPTY(&wrq->incomplete_wrs))
return; /* commit_wrq_wr will drain wr_list as well. */
drain_wrq_wr_list(sc, wrq);
/* Doorbell must have caught up to the pidx. */
MPASS(eq->pidx == eq->dbidx);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
void
t4_update_fl_bufsize(struct ifnet *ifp)
{
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
struct vi_info *vi = ifp->if_softc;
cxgbe(4): Add a pointer to the adapter softc in vi_info. There were quite a few places where port_info was being accessed only to get to the adapter. Reviewed by: jhb@ MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25432
2020-06-25 17:04:22 +00:00
struct adapter *sc = vi->adapter;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
struct sge_rxq *rxq;
- Updated TOE support in the kernel. - Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs. These are available as t3_tom and t4_tom modules that augment cxgb(4) and cxgbe(4) respectively. The cxgb/cxgbe drivers continue to work as usual with or without these extra features. - iWARP driver for Terminator 3 ASIC (kernel verbs). T4 iWARP in the works and will follow soon. Build-tested with make universe. 30s overview ============ What interfaces support TCP offload? Look for TOE4 and/or TOE6 in the capabilities of an interface: # ifconfig -m | grep TOE Enable/disable TCP offload on an interface (just like any other ifnet capability): # ifconfig cxgbe0 toe # ifconfig cxgbe0 -toe Which connections are offloaded? Look for toe4 and/or toe6 in the output of netstat and sockstat: # netstat -np tcp | grep toe # sockstat -46c | grep toe Reviewed by: bz, gnn Sponsored by: Chelsio communications. MFC after: ~3 months (after 9.1, and after ensuring MFC is feasible)
2012-06-19 07:34:13 +00:00
#ifdef TCP_OFFLOAD
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
struct sge_ofld_rxq *ofld_rxq;
- Updated TOE support in the kernel. - Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs. These are available as t3_tom and t4_tom modules that augment cxgb(4) and cxgbe(4) respectively. The cxgb/cxgbe drivers continue to work as usual with or without these extra features. - iWARP driver for Terminator 3 ASIC (kernel verbs). T4 iWARP in the works and will follow soon. Build-tested with make universe. 30s overview ============ What interfaces support TCP offload? Look for TOE4 and/or TOE6 in the capabilities of an interface: # ifconfig -m | grep TOE Enable/disable TCP offload on an interface (just like any other ifnet capability): # ifconfig cxgbe0 toe # ifconfig cxgbe0 -toe Which connections are offloaded? Look for toe4 and/or toe6 in the output of netstat and sockstat: # netstat -np tcp | grep toe # sockstat -46c | grep toe Reviewed by: bz, gnn Sponsored by: Chelsio communications. MFC after: ~3 months (after 9.1, and after ensuring MFC is feasible)
2012-06-19 07:34:13 +00:00
#endif
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
struct sge_fl *fl;
cxgbe(4): Use large clusters for TOE rx queues when TOE+TLS is enabled. Rx is more efficient within the chip when the receive buffer size matches the TLS PDU size. MFC after: 3 days Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D26127
2020-08-23 04:16:20 +00:00
int i, maxp;
Tweaks for rx: - everything related to LRO should be in #ifdef INET blocks - reorder sge_iq's fields so that the most frequently used are all together - pull all rx code into t4_intr_data directly - let go of the ingress queue lock when passing up data - refill the freelist only if it is short of at least 32 buffers
2011-03-05 03:42:03 +00:00
cxgbe(4): Use large clusters for TOE rx queues when TOE+TLS is enabled. Rx is more efficient within the chip when the receive buffer size matches the TLS PDU size. MFC after: 3 days Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D26127
2020-08-23 04:16:20 +00:00
maxp = max_rx_payload(sc, ifp, false);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_rxq(vi, i, rxq) {
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
fl = &rxq->fl;
Tweaks for rx: - everything related to LRO should be in #ifdef INET blocks - reorder sge_iq's fields so that the most frequently used are all together - pull all rx code into t4_intr_data directly - let go of the ingress queue lock when passing up data - refill the freelist only if it is short of at least 32 buffers
2011-03-05 03:42:03 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
FL_LOCK(fl);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
fl->zidx = find_refill_source(sc, maxp,
fl->flags & FL_BUF_PACKING);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
FL_UNLOCK(fl);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
#ifdef TCP_OFFLOAD
cxgbe(4): Use large clusters for TOE rx queues when TOE+TLS is enabled. Rx is more efficient within the chip when the receive buffer size matches the TLS PDU size. MFC after: 3 days Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D26127
2020-08-23 04:16:20 +00:00
maxp = max_rx_payload(sc, ifp, true);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
for_each_ofld_rxq(vi, i, ofld_rxq) {
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
fl = &ofld_rxq->fl;
T4 packet timestamps. Reference code that shows how to get a packet's timestamp out of cxgbe(4). Disabled by default because we don't have a standard way today to pass this information up the stack. The timestamp is 60 bits wide and each increment represents 1 tick of the T4's core clock. As an example, the timestamp granularity is ~4.4ns for this card: # sysctl dev.t4nex.0.core_clock dev.t4nex.0.core_clock: 228125 MFC after: 1 week
2011-05-05 02:38:08 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
FL_LOCK(fl);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
fl->zidx = find_refill_source(sc, maxp,
fl->flags & FL_BUF_PACKING);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
FL_UNLOCK(fl);
}
#endif
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static inline int
mbuf_nsegs(struct mbuf *m)
{
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
M_ASSERTPKTHDR(m);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
KASSERT(m->m_pkthdr.inner_l5hlen > 0,
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
("%s: mbuf %p missing information on # of segments.", __func__, m));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
return (m->m_pkthdr.inner_l5hlen);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static inline void
set_mbuf_nsegs(struct mbuf *m, uint8_t nsegs)
{
M_ASSERTPKTHDR(m);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
m->m_pkthdr.inner_l5hlen = nsegs;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
static inline int
mbuf_cflags(struct mbuf *m)
{
M_ASSERTPKTHDR(m);
return (m->m_pkthdr.PH_loc.eight[4]);
}
static inline void
set_mbuf_cflags(struct mbuf *m, uint8_t flags)
{
M_ASSERTPKTHDR(m);
m->m_pkthdr.PH_loc.eight[4] = flags;
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static inline int
mbuf_len16(struct mbuf *m)
{
int n;
Tweaks for rx: - everything related to LRO should be in #ifdef INET blocks - reorder sge_iq's fields so that the most frequently used are all together - pull all rx code into t4_intr_data directly - let go of the ingress queue lock when passing up data - refill the freelist only if it is short of at least 32 buffers
2011-03-05 03:42:03 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
M_ASSERTPKTHDR(m);
n = m->m_pkthdr.PH_loc.eight[0];
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
if (!(mbuf_cflags(m) & MC_TLS))
MPASS(n > 0 && n <= SGE_MAX_WR_LEN / 16);
Tweaks for rx: - everything related to LRO should be in #ifdef INET blocks - reorder sge_iq's fields so that the most frequently used are all together - pull all rx code into t4_intr_data directly - let go of the ingress queue lock when passing up data - refill the freelist only if it is short of at least 32 buffers
2011-03-05 03:42:03 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (n);
}
Tweaks for rx: - everything related to LRO should be in #ifdef INET blocks - reorder sge_iq's fields so that the most frequently used are all together - pull all rx code into t4_intr_data directly - let go of the ingress queue lock when passing up data - refill the freelist only if it is short of at least 32 buffers
2011-03-05 03:42:03 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static inline void
set_mbuf_len16(struct mbuf *m, uint8_t len16)
{
M_ASSERTPKTHDR(m);
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
if (!(mbuf_cflags(m) & MC_TLS))
MPASS(len16 > 0 && len16 <= SGE_MAX_WR_LEN / 16);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
m->m_pkthdr.PH_loc.eight[0] = len16;
}
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#ifdef RATELIMIT
static inline int
mbuf_eo_nsegs(struct mbuf *m)
{
M_ASSERTPKTHDR(m);
return (m->m_pkthdr.PH_loc.eight[1]);
}
static inline void
set_mbuf_eo_nsegs(struct mbuf *m, uint8_t nsegs)
{
M_ASSERTPKTHDR(m);
m->m_pkthdr.PH_loc.eight[1] = nsegs;
}
static inline int
mbuf_eo_len16(struct mbuf *m)
{
int n;
M_ASSERTPKTHDR(m);
n = m->m_pkthdr.PH_loc.eight[2];
MPASS(n > 0 && n <= SGE_MAX_WR_LEN / 16);
return (n);
}
static inline void
set_mbuf_eo_len16(struct mbuf *m, uint8_t len16)
{
M_ASSERTPKTHDR(m);
m->m_pkthdr.PH_loc.eight[2] = len16;
}
static inline int
mbuf_eo_tsclk_tsoff(struct mbuf *m)
{
M_ASSERTPKTHDR(m);
return (m->m_pkthdr.PH_loc.eight[3]);
}
static inline void
set_mbuf_eo_tsclk_tsoff(struct mbuf *m, uint8_t tsclk_tsoff)
{
M_ASSERTPKTHDR(m);
m->m_pkthdr.PH_loc.eight[3] = tsclk_tsoff;
}
static inline int
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
needs_eo(struct m_snd_tag *mst)
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
{
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
return (mst != NULL && mst->type == IF_SND_TAG_TYPE_RATE_LIMIT);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
}
#endif
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
/*
* Try to allocate an mbuf to contain a raw work request. To make it
* easy to construct the work request, don't allocate a chain but a
* single mbuf.
*/
struct mbuf *
alloc_wr_mbuf(int len, int how)
{
struct mbuf *m;
if (len <= MHLEN)
m = m_gethdr(how, MT_DATA);
else if (len <= MCLBYTES)
m = m_getcl(how, MT_DATA, M_PKTHDR);
else
m = NULL;
if (m == NULL)
return (NULL);
m->m_pkthdr.len = len;
m->m_len = len;
set_mbuf_cflags(m, MC_RAW_WR);
set_mbuf_len16(m, howmany(len, 16));
return (m);
}
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
static inline bool
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
needs_hwcsum(struct mbuf *m)
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
{
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
const uint32_t csum_flags = CSUM_IP | CSUM_IP_UDP | CSUM_IP_TCP |
CSUM_IP_TSO | CSUM_INNER_IP | CSUM_INNER_IP_UDP |
CSUM_INNER_IP_TCP | CSUM_INNER_IP_TSO | CSUM_IP6_UDP |
CSUM_IP6_TCP | CSUM_IP6_TSO | CSUM_INNER_IP6_UDP |
CSUM_INNER_IP6_TCP | CSUM_INNER_IP6_TSO;
Tweaks for rx: - everything related to LRO should be in #ifdef INET blocks - reorder sge_iq's fields so that the most frequently used are all together - pull all rx code into t4_intr_data directly - let go of the ingress queue lock when passing up data - refill the freelist only if it is short of at least 32 buffers
2011-03-05 03:42:03 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
M_ASSERTPKTHDR(m);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
return (m->m_pkthdr.csum_flags & csum_flags);
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
}
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
static inline bool
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
needs_tso(struct mbuf *m)
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
{
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
const uint32_t csum_flags = CSUM_IP_TSO | CSUM_IP6_TSO |
CSUM_INNER_IP_TSO | CSUM_INNER_IP6_TSO;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
M_ASSERTPKTHDR(m);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
return (m->m_pkthdr.csum_flags & csum_flags);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
static inline bool
needs_vxlan_csum(struct mbuf *m)
{
M_ASSERTPKTHDR(m);
return (m->m_pkthdr.csum_flags & CSUM_ENCAP_VXLAN);
}
static inline bool
needs_vxlan_tso(struct mbuf *m)
{
const uint32_t csum_flags = CSUM_ENCAP_VXLAN | CSUM_INNER_IP_TSO |
CSUM_INNER_IP6_TSO;
M_ASSERTPKTHDR(m);
return ((m->m_pkthdr.csum_flags & csum_flags) != 0 &&
(m->m_pkthdr.csum_flags & csum_flags) != CSUM_ENCAP_VXLAN);
}
static inline bool
needs_inner_tcp_csum(struct mbuf *m)
{
const uint32_t csum_flags = CSUM_INNER_IP_TSO | CSUM_INNER_IP6_TSO;
M_ASSERTPKTHDR(m);
return (m->m_pkthdr.csum_flags & csum_flags);
}
static inline bool
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
needs_l3_csum(struct mbuf *m)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
const uint32_t csum_flags = CSUM_IP | CSUM_IP_TSO | CSUM_INNER_IP |
CSUM_INNER_IP_TSO;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
M_ASSERTPKTHDR(m);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
return (m->m_pkthdr.csum_flags & csum_flags);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
Calculate how many descriptors can be reclaimed before calling reclaim_tx_descs
2011-03-05 03:54:37 +00:00
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
static inline bool
needs_outer_tcp_csum(struct mbuf *m)
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
{
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
const uint32_t csum_flags = CSUM_IP_TCP | CSUM_IP_TSO | CSUM_IP6_TCP |
CSUM_IP6_TSO;
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
M_ASSERTPKTHDR(m);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
return (m->m_pkthdr.csum_flags & csum_flags);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
}
cxgbe(4): Suppress a warning about code that is used only with options RATELIMIT. Reported by: mmacy@
2018-05-25 18:57:41 +00:00
#ifdef RATELIMIT
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
static inline bool
needs_outer_l4_csum(struct mbuf *m)
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
{
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
const uint32_t csum_flags = CSUM_IP_UDP | CSUM_IP_TCP | CSUM_IP_TSO |
CSUM_IP6_UDP | CSUM_IP6_TCP | CSUM_IP6_TSO;
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
M_ASSERTPKTHDR(m);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
return (m->m_pkthdr.csum_flags & csum_flags);
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
}
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
static inline bool
needs_outer_udp_csum(struct mbuf *m)
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
{
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
const uint32_t csum_flags = CSUM_IP_UDP | CSUM_IP6_UDP;
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
M_ASSERTPKTHDR(m);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
return (m->m_pkthdr.csum_flags & csum_flags);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
}
cxgbe(4): Suppress a warning about code that is used only with options RATELIMIT. Reported by: mmacy@
2018-05-25 18:57:41 +00:00
#endif
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
static inline bool
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
needs_vlan_insertion(struct mbuf *m)
{
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
M_ASSERTPKTHDR(m);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): Slightly simpler needs_<foo> functions.
2018-05-24 07:38:46 +00:00
return (m->m_flags & M_VLANTAG);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static void *
m_advance(struct mbuf **pm, int *poffset, int len)
{
struct mbuf *m = *pm;
int offset = *poffset;
uintptr_t p = 0;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
MPASS(len > 0);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Don't break out of the m_advance() loop if len drops to zero. If a packet contains the Ethernet header (14 bytes) in the first mbuf and the payload (IP + UDP + data) in the second mbuf, then the attempt to fetch the l3hdr will return a NULL pointer. The first loop iteration will drop len to zero and exit the loop without setting 'p'. However, the desired data is at the start of the second mbuf, so the correct behavior is to loop around and let the conditional set 'p' to m_data of the next mbuf (and leave offset as 0). Reviewed by: np Sponsored by: Chelsio Communications
2016-09-07 18:08:43 +00:00
for (;;) {
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (offset + len < m->m_len) {
offset += len;
p = mtod(m, uintptr_t) + offset;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
break;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
len -= m->m_len - offset;
m = m->m_next;
offset = 0;
MPASS(m != NULL);
}
*poffset = offset;
*pm = m;
return ((void *)p);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
static inline int
count_mbuf_ext_pgs(struct mbuf *m, int skip, vm_paddr_t *nextaddr)
{
vm_paddr_t paddr;
int i, len, off, pglen, pgoff, seglen, segoff;
int nsegs = 0;
Mechanically rename MBUF_EXT_PGS_ASSERT() to M_ASSERTEXTPG() to match classical M_ASSERTPKTHDR. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:27:41 +00:00
M_ASSERTEXTPG(m);
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
off = mtod(m, vm_offset_t);
len = m->m_len;
off += skip;
len -= skip;
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
if (m->m_epg_hdrlen != 0) {
if (off >= m->m_epg_hdrlen) {
off -= m->m_epg_hdrlen;
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
} else {
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
seglen = m->m_epg_hdrlen - off;
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
segoff = off;
seglen = min(seglen, len);
off = 0;
len -= seglen;
paddr = pmap_kextract(
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
(vm_offset_t)&m->m_epg_hdr[segoff]);
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
if (*nextaddr != paddr)
nsegs++;
*nextaddr = paddr + seglen;
}
}
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
pgoff = m->m_epg_1st_off;
for (i = 0; i < m->m_epg_npgs && len > 0; i++) {
Step 2.3: Rename mbuf_ext_pg_len() to m_epg_pagelen() that uses mbuf argument. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 23:52:35 +00:00
pglen = m_epg_pagelen(m, i, pgoff);
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
if (off >= pglen) {
off -= pglen;
pgoff = 0;
continue;
}
seglen = pglen - off;
segoff = pgoff + off;
off = 0;
seglen = min(seglen, len);
len -= seglen;
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
paddr = m->m_epg_pa[i] + segoff;
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
if (*nextaddr != paddr)
nsegs++;
*nextaddr = paddr + seglen;
pgoff = 0;
};
if (len != 0) {
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
seglen = min(len, m->m_epg_trllen - off);
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
len -= seglen;
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
paddr = pmap_kextract((vm_offset_t)&m->m_epg_trail[off]);
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
if (*nextaddr != paddr)
nsegs++;
*nextaddr = paddr + seglen;
}
return (nsegs);
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/*
* Can deal with empty mbufs in the chain that have m_len = 0, but the chain
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
* must have at least one mbuf that's not empty. It is possible for this
* routine to return 0 if skip accounts for all the contents of the mbuf chain.
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
*/
static inline int
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
count_mbuf_nsegs(struct mbuf *m, int skip, uint8_t *cflags)
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
{
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
vm_paddr_t nextaddr, paddr;
cxgbe(4): Fix bug in the calculation of the number of physically contiguous regions in an mbuf chain. If the payload of an mbuf ends at a page boundary count_mbuf_nsegs would incorrectly consider the next mbuf's payload physically contiguous based solely on a KVA comparison. MFC after: 1 week Sponsored by: Chelsio Communications
2016-10-24 19:09:56 +00:00
vm_offset_t va;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
int len, nsegs;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
M_ASSERTPKTHDR(m);
MPASS(m->m_pkthdr.len > 0);
MPASS(m->m_pkthdr.len >= skip);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
nsegs = 0;
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
nextaddr = 0;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
for (; m; m = m->m_next) {
len = m->m_len;
if (__predict_false(len == 0))
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
continue;
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
if (skip >= len) {
skip -= len;
continue;
}
Step 4.1: mechanically rename M_NOMAP to M_EXTPG Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:21:11 +00:00
if ((m->m_flags & M_EXTPG) != 0) {
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
*cflags |= MC_NOMAP;
nsegs += count_mbuf_ext_pgs(m, skip, &nextaddr);
skip = 0;
continue;
}
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
va = mtod(m, vm_offset_t) + skip;
len -= skip;
skip = 0;
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
paddr = pmap_kextract(va);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
nsegs += sglist_count((void *)(uintptr_t)va, len);
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
if (paddr == nextaddr)
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
nsegs--;
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
nextaddr = pmap_kextract(va + len - 1) + 1;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (nsegs);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
/*
* The maximum number of segments that can fit in a WR.
*/
static int
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
max_nsegs_allowed(struct mbuf *m, bool vm_wr)
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
{
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
if (vm_wr) {
if (needs_tso(m))
return (TX_SGL_SEGS_VM_TSO);
return (TX_SGL_SEGS_VM);
}
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (needs_tso(m)) {
if (needs_vxlan_tso(m))
return (TX_SGL_SEGS_VXLAN_TSO);
else
return (TX_SGL_SEGS_TSO);
}
return (TX_SGL_SEGS);
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/*
* Analyze the mbuf to determine its tx needs. The mbuf passed in may change:
* a) caller can assume it's been freed if this function returns with an error.
* b) it may get defragged up if the gather list is too long for the hardware.
*/
int
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
parse_pkt(struct mbuf **mp, bool vm_wr)
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
{
struct mbuf *m0 = *mp, *m;
int rc, nsegs, defragged = 0, offset;
struct ether_header *eh;
void *l3hdr;
#if defined(INET) || defined(INET6)
struct tcphdr *tcp;
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
#endif
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
#if defined(KERN_TLS) || defined(RATELIMIT)
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
struct m_snd_tag *mst;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
#endif
uint16_t eh_type;
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
uint8_t cflags;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
cflags = 0;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
M_ASSERTPKTHDR(m0);
if (__predict_false(m0->m_pkthdr.len < ETHER_HDR_LEN)) {
rc = EINVAL;
fail:
m_freem(m0);
*mp = NULL;
return (rc);
}
restart:
/*
* First count the number of gather list segments in the payload.
* Defrag the mbuf if nsegs exceeds the hardware limit.
*/
M_ASSERTPKTHDR(m0);
MPASS(m0->m_pkthdr.len > 0);
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
nsegs = count_mbuf_nsegs(m0, 0, &cflags);
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
#if defined(KERN_TLS) || defined(RATELIMIT)
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
if (m0->m_pkthdr.csum_flags & CSUM_SND_TAG)
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
mst = m0->m_pkthdr.snd_tag;
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
else
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
mst = NULL;
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
#endif
#ifdef KERN_TLS
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
if (mst != NULL && mst->type == IF_SND_TAG_TYPE_TLS) {
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
int len16;
cflags |= MC_TLS;
set_mbuf_cflags(m0, cflags);
rc = t6_ktls_parse_pkt(m0, &nsegs, &len16);
if (rc != 0)
goto fail;
set_mbuf_nsegs(m0, nsegs);
set_mbuf_len16(m0, len16);
return (0);
}
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
#endif
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
if (nsegs > max_nsegs_allowed(m0, vm_wr)) {
cxgbe(4): add counters for mbuf pullups and defrags. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 03:06:36 +00:00
if (defragged++ > 0) {
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
rc = EFBIG;
goto fail;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): add counters for mbuf pullups and defrags. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 03:06:36 +00:00
counter_u64_add(defrags, 1);
if ((m = m_defrag(m0, M_NOWAIT)) == NULL) {
rc = ENOMEM;
goto fail;
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
*mp = m0 = m; /* update caller's copy after defrag */
goto restart;
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
if (__predict_false(nsegs > 2 && m0->m_pkthdr.len <= MHLEN &&
!(cflags & MC_NOMAP))) {
cxgbe(4): add counters for mbuf pullups and defrags. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 03:06:36 +00:00
counter_u64_add(pullups, 1);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
m0 = m_pullup(m0, m0->m_pkthdr.len);
if (m0 == NULL) {
/* Should have left well enough alone. */
rc = EFBIG;
goto fail;
}
*mp = m0; /* update caller's copy after pullup */
goto restart;
}
set_mbuf_nsegs(m0, nsegs);
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
set_mbuf_cflags(m0, cflags);
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
calculate_mbuf_len16(m0, vm_wr);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#ifdef RATELIMIT
/*
* Ethofld is limited to TCP and UDP for now, and only when L4 hw
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
* checksumming is enabled. needs_outer_l4_csum happens to check for
* all the right things.
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
*/
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
if (__predict_false(needs_eo(mst) && !needs_outer_l4_csum(m0))) {
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
m_snd_tag_rele(m0->m_pkthdr.snd_tag);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
m0->m_pkthdr.snd_tag = NULL;
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
m0->m_pkthdr.csum_flags &= ~CSUM_SND_TAG;
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
mst = NULL;
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
}
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#endif
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
if (!needs_hwcsum(m0)
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#ifdef RATELIMIT
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
&& !needs_eo(mst)
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#endif
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
)
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (0);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
m = m0;
eh = mtod(m, struct ether_header *);
eh_type = ntohs(eh->ether_type);
if (eh_type == ETHERTYPE_VLAN) {
struct ether_vlan_header *evh = (void *)eh;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
eh_type = ntohs(evh->evl_proto);
m0->m_pkthdr.l2hlen = sizeof(*evh);
} else
m0->m_pkthdr.l2hlen = sizeof(*eh);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
offset = 0;
l3hdr = m_advance(&m, &offset, m0->m_pkthdr.l2hlen);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
switch (eh_type) {
#ifdef INET6
case ETHERTYPE_IPV6:
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
m0->m_pkthdr.l3hlen = sizeof(struct ip6_hdr);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
break;
#endif
#ifdef INET
case ETHERTYPE_IP:
{
struct ip *ip = l3hdr;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (needs_vxlan_csum(m0)) {
/* Driver will do the outer IP hdr checksum. */
ip->ip_sum = 0;
if (needs_vxlan_tso(m0)) {
const uint16_t ipl = ip->ip_len;
ip->ip_len = 0;
ip->ip_sum = ~in_cksum_hdr(ip);
ip->ip_len = ipl;
} else
ip->ip_sum = in_cksum_hdr(ip);
}
m0->m_pkthdr.l3hlen = ip->ip_hl << 2;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
break;
}
#endif
default:
panic("%s: ethertype 0x%04x unknown. if_cxgbe must be compiled"
" with the same INET/INET6 options as the kernel.",
__func__, eh_type);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (needs_vxlan_csum(m0)) {
m0->m_pkthdr.l4hlen = sizeof(struct udphdr);
m0->m_pkthdr.l5hlen = sizeof(struct vxlan_header);
/* Inner headers. */
eh = m_advance(&m, &offset, m0->m_pkthdr.l3hlen +
sizeof(struct udphdr) + sizeof(struct vxlan_header));
eh_type = ntohs(eh->ether_type);
if (eh_type == ETHERTYPE_VLAN) {
struct ether_vlan_header *evh = (void *)eh;
eh_type = ntohs(evh->evl_proto);
m0->m_pkthdr.inner_l2hlen = sizeof(*evh);
} else
m0->m_pkthdr.inner_l2hlen = sizeof(*eh);
l3hdr = m_advance(&m, &offset, m0->m_pkthdr.inner_l2hlen);
switch (eh_type) {
#ifdef INET6
case ETHERTYPE_IPV6:
m0->m_pkthdr.inner_l3hlen = sizeof(struct ip6_hdr);
break;
#endif
#ifdef INET
case ETHERTYPE_IP:
{
struct ip *ip = l3hdr;
m0->m_pkthdr.inner_l3hlen = ip->ip_hl << 2;
break;
}
#endif
default:
panic("%s: VXLAN hw offload requested with unknown "
"ethertype 0x%04x. if_cxgbe must be compiled"
" with the same INET/INET6 options as the kernel.",
__func__, eh_type);
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
#if defined(INET) || defined(INET6)
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (needs_inner_tcp_csum(m0)) {
tcp = m_advance(&m, &offset, m0->m_pkthdr.inner_l3hlen);
m0->m_pkthdr.inner_l4hlen = tcp->th_off * 4;
}
#endif
MPASS((m0->m_pkthdr.csum_flags & CSUM_SND_TAG) == 0);
m0->m_pkthdr.csum_flags &= CSUM_INNER_IP6_UDP |
CSUM_INNER_IP6_TCP | CSUM_INNER_IP6_TSO | CSUM_INNER_IP |
CSUM_INNER_IP_UDP | CSUM_INNER_IP_TCP | CSUM_INNER_IP_TSO |
CSUM_ENCAP_VXLAN;
}
#if defined(INET) || defined(INET6)
if (needs_outer_tcp_csum(m0)) {
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
tcp = m_advance(&m, &offset, m0->m_pkthdr.l3hlen);
m0->m_pkthdr.l4hlen = tcp->th_off * 4;
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#ifdef RATELIMIT
if (tsclk >= 0 && *(uint32_t *)(tcp + 1) == ntohl(0x0101080a)) {
set_mbuf_eo_tsclk_tsoff(m0,
V_FW_ETH_TX_EO_WR_TSCLK(tsclk) |
V_FW_ETH_TX_EO_WR_TSOFF(sizeof(*tcp) / 2 + 1));
} else
set_mbuf_eo_tsclk_tsoff(m0, 0);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
} else if (needs_outer_udp_csum(m0)) {
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
m0->m_pkthdr.l4hlen = sizeof(struct udphdr);
#endif
}
#ifdef RATELIMIT
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
if (needs_eo(mst)) {
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
u_int immhdrs;
/* EO WRs have the headers in the WR and not the GL. */
immhdrs = m0->m_pkthdr.l2hlen + m0->m_pkthdr.l3hlen +
m0->m_pkthdr.l4hlen;
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
cflags = 0;
nsegs = count_mbuf_nsegs(m0, immhdrs, &cflags);
MPASS(cflags == mbuf_cflags(m0));
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
set_mbuf_eo_nsegs(m0, nsegs);
set_mbuf_eo_len16(m0,
txpkt_eo_len16(nsegs, immhdrs, needs_tso(m0)));
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
}
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#endif
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
#endif
MPASS(m0 == *mp);
return (0);
}
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
void *
start_wrq_wr(struct sge_wrq *wrq, int len16, struct wrq_cookie *cookie)
{
struct sge_eq *eq = &wrq->eq;
struct adapter *sc = wrq->adapter;
int ndesc, available;
struct wrqe *wr;
void *w;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
MPASS(len16 > 0);
cxgbe(4): Add a tx_len16_to_desc helper. No functional change. MFC after: 1 week Sponsored by: Chelsio Communications
2020-06-23 07:33:29 +00:00
ndesc = tx_len16_to_desc(len16);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
MPASS(ndesc > 0 && ndesc <= SGE_MAX_WR_NDESC);
EQ_LOCK(eq);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Fix a couple of problems in the sge_wrq data path. - start_wrq_wr must not drain the wr_list if there are incomplete_wrs pending. This can happen when a t4_wrq_tx runs between two start_wrq_wr. - commit_wrq_wr must examine the cookie's pidx and ndesc with the queue's lock held. Otherwise there is a bad race when incomplete WRs are being completed and commit_wrq_wr for the WR that is ahead in the queue updates the next incomplete WR's cookie's pidx/ndesc but the commit_wrq_wr for the second one is using stale values that it read without the lock. MFC after: 1 week Sponsored by: Chelsio Communications
2017-09-09 05:12:14 +00:00
if (TAILQ_EMPTY(&wrq->incomplete_wrs) && !STAILQ_EMPTY(&wrq->wr_list))
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
drain_wrq_wr_list(sc, wrq);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (!STAILQ_EMPTY(&wrq->wr_list)) {
slowpath:
EQ_UNLOCK(eq);
wr = alloc_wrqe(len16 * 16, wrq);
if (__predict_false(wr == NULL))
return (NULL);
cookie->pidx = -1;
cookie->ndesc = ndesc;
return (&wr->wr);
}
eq->cidx = read_hw_cidx(eq);
if (eq->pidx == eq->cidx)
available = eq->sidx - 1;
else
available = IDXDIFF(eq->cidx, eq->pidx, eq->sidx) - 1;
if (available < ndesc)
goto slowpath;
cookie->pidx = eq->pidx;
cookie->ndesc = ndesc;
TAILQ_INSERT_TAIL(&wrq->incomplete_wrs, cookie, link);
w = &eq->desc[eq->pidx];
IDXINCR(eq->pidx, ndesc, eq->sidx);
cxgbe(4): The wraparound logic in start_wrq_wr() should not get involved in work requests that end at the end of the descriptor ring, even though the pidx wraps around to 0. MFC after: 3 days
2017-01-09 22:18:08 +00:00
if (__predict_false(cookie->pidx + ndesc > eq->sidx)) {
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
w = &wrq->ss[0];
wrq->ss_pidx = cookie->pidx;
wrq->ss_len = len16 * 16;
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
EQ_UNLOCK(eq);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (w);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
void
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
commit_wrq_wr(struct sge_wrq *wrq, void *w, struct wrq_cookie *cookie)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
struct sge_eq *eq = &wrq->eq;
struct adapter *sc = wrq->adapter;
int ndesc, pidx;
struct wrq_cookie *prev, *next;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (cookie->pidx == -1) {
struct wrqe *wr = __containerof(w, struct wrqe, wr);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
t4_wrq_tx(sc, wr);
return;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
- Make note of interface MTU change if the rx queues exist, and not just when the interface is up. - Add a tunable to control the TOE's rx coalesce feature (enabled by default as it always has been). Consider the interface MTU or the coalesce size when deciding which cluster zone to use to fill the offload rx queue's free list. The tunable is: dev.{t4nex,t5nex}.<N>.toe.rx_coalesce MFC after: 1 day
2013-07-04 21:19:01 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (__predict_false(w == &wrq->ss[0])) {
int n = (eq->sidx - wrq->ss_pidx) * EQ_ESIZE;
MPASS(wrq->ss_len > n); /* WR had better wrap around. */
bcopy(&wrq->ss[0], &eq->desc[wrq->ss_pidx], n);
bcopy(&wrq->ss[n], &eq->desc[0], wrq->ss_len - n);
wrq->tx_wrs_ss++;
} else
wrq->tx_wrs_direct++;
EQ_LOCK(eq);
cxgbe(4): Fix a couple of problems in the sge_wrq data path. - start_wrq_wr must not drain the wr_list if there are incomplete_wrs pending. This can happen when a t4_wrq_tx runs between two start_wrq_wr. - commit_wrq_wr must examine the cookie's pidx and ndesc with the queue's lock held. Otherwise there is a bad race when incomplete WRs are being completed and commit_wrq_wr for the WR that is ahead in the queue updates the next incomplete WR's cookie's pidx/ndesc but the commit_wrq_wr for the second one is using stale values that it read without the lock. MFC after: 1 week Sponsored by: Chelsio Communications
2017-09-09 05:12:14 +00:00
ndesc = cookie->ndesc; /* Can be more than SGE_MAX_WR_NDESC here. */
pidx = cookie->pidx;
MPASS(pidx >= 0 && pidx < eq->sidx);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
prev = TAILQ_PREV(cookie, wrq_incomplete_wrs, link);
next = TAILQ_NEXT(cookie, link);
if (prev == NULL) {
MPASS(pidx == eq->dbidx);
cxgbe(4): Make sure that the egress queue's cidx is updated periodically when the driver is writing WRs using start_wrq_wr/commit_wrq_wr all the time. Sponsored by: Chelsio Communications
2018-05-24 06:44:06 +00:00
if (next == NULL || ndesc >= 16) {
int available;
struct fw_eth_tx_pkt_wr *dst; /* any fw WR struct will do */
/*
* Note that the WR via which we'll request tx updates
* is at pidx and not eq->pidx, which has moved on
* already.
*/
dst = (void *)&eq->desc[pidx];
available = IDXDIFF(eq->cidx, eq->pidx, eq->sidx) - 1;
if (available < eq->sidx / 4 &&
atomic_cmpset_int(&eq->equiq, 0, 1)) {
cxgbe(4): Use automatic cidx updates with ofld and ctrl queues. The bits that explicitly request cidx updates do not work reliably with all possible WRs that can be sent over the queue. The F_FW_WR_EQUIQ requests that still remain may also have to be replaced with explicit credit flush WRs in the future. MFC after: 2 days Sponsored by: Chelsio Communications
2018-10-22 23:06:23 +00:00
/*
* XXX: This is not 100% reliable with some
* types of WRs. But this is a very unusual
* situation for an ofld/ctrl queue anyway.
*/
cxgbe(4): Make sure that the egress queue's cidx is updated periodically when the driver is writing WRs using start_wrq_wr/commit_wrq_wr all the time. Sponsored by: Chelsio Communications
2018-05-24 06:44:06 +00:00
dst->equiq_to_len16 |= htobe32(F_FW_WR_EQUIQ |
F_FW_WR_EQUEQ);
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
ring_eq_db(wrq->adapter, eq, ndesc);
cxgbe(4): Make sure that the egress queue's cidx is updated periodically when the driver is writing WRs using start_wrq_wr/commit_wrq_wr all the time. Sponsored by: Chelsio Communications
2018-05-24 06:44:06 +00:00
} else {
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
MPASS(IDXDIFF(next->pidx, pidx, eq->sidx) == ndesc);
next->pidx = pidx;
next->ndesc += ndesc;
}
} else {
MPASS(IDXDIFF(pidx, prev->pidx, eq->sidx) == prev->ndesc);
prev->ndesc += ndesc;
}
TAILQ_REMOVE(&wrq->incomplete_wrs, cookie, link);
if (TAILQ_EMPTY(&wrq->incomplete_wrs) && !STAILQ_EMPTY(&wrq->wr_list))
drain_wrq_wr_list(sc, wrq);
#ifdef INVARIANTS
if (TAILQ_EMPTY(&wrq->incomplete_wrs)) {
/* Doorbell must have caught up to the pidx. */
MPASS(wrq->eq.pidx == wrq->eq.dbidx);
- Make note of interface MTU change if the rx queues exist, and not just when the interface is up. - Add a tunable to control the TOE's rx coalesce feature (enabled by default as it always has been). Consider the interface MTU or the coalesce size when deciding which cluster zone to use to fill the offload rx queue's free list. The tunable is: dev.{t4nex,t5nex}.<N>.toe.rx_coalesce MFC after: 1 day
2013-07-04 21:19:01 +00:00
}
#endif
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
EQ_UNLOCK(eq);
}
static u_int
can_resume_eth_tx(struct mp_ring *r)
{
struct sge_eq *eq = r->cookie;
return (total_available_tx_desc(eq) > eq->sidx / 8);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
static inline bool
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
cannot_use_txpkts(struct mbuf *m)
{
/* maybe put a GL limit too, to avoid silliness? */
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
return (needs_tso(m) || (mbuf_cflags(m) & (MC_RAW_WR | MC_TLS)) != 0);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
cxgbe(4): Do not assume that if_qflush is always followed by inteface-down. MFC after: 3 days Sponsored by: Chelsio Communications
2017-05-09 18:33:41 +00:00
static inline int
discard_tx(struct sge_eq *eq)
{
return ((eq->flags & (EQ_ENABLED | EQ_QFLUSH)) != EQ_ENABLED);
}
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
static inline int
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
wr_can_update_eq(void *p)
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
{
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
struct fw_eth_tx_pkts_wr *wr = p;
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
switch (G_FW_WR_OP(be32toh(wr->op_pkd))) {
case FW_ULPTX_WR:
case FW_ETH_TX_PKT_WR:
case FW_ETH_TX_PKTS_WR:
cxgbe(4): An EQ update can be requested in a TX_PKTS2 work request. MFC after: 1 week Sponsored by: Chelsio Communications
2019-10-15 17:35:39 +00:00
case FW_ETH_TX_PKTS2_WR:
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
case FW_ETH_TX_PKT_VM_WR:
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
case FW_ETH_TX_PKTS_VM_WR:
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
return (1);
default:
return (0);
}
}
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
static inline void
set_txupdate_flags(struct sge_txq *txq, u_int avail,
struct fw_eth_tx_pkt_wr *wr)
{
struct sge_eq *eq = &txq->eq;
struct txpkts *txp = &txq->txp;
if ((txp->npkt > 0 || avail < eq->sidx / 2) &&
atomic_cmpset_int(&eq->equiq, 0, 1)) {
wr->equiq_to_len16 |= htobe32(F_FW_WR_EQUEQ | F_FW_WR_EQUIQ);
eq->equeqidx = eq->pidx;
} else if (IDXDIFF(eq->pidx, eq->equeqidx, eq->sidx) >= 32) {
wr->equiq_to_len16 |= htobe32(F_FW_WR_EQUEQ);
eq->equeqidx = eq->pidx;
}
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/*
* r->items[cidx] to r->items[pidx], with a wraparound at r->size, are ready to
* be consumed. Return the actual number consumed. 0 indicates a stall.
*/
static u_int
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
eth_tx(struct mp_ring *r, u_int cidx, u_int pidx, bool *coalescing)
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
{
struct sge_txq *txq = r->cookie;
struct ifnet *ifp = txq->ifp;
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
struct sge_eq *eq = &txq->eq;
struct txpkts *txp = &txq->txp;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
struct vi_info *vi = ifp->if_softc;
cxgbe(4): Add a pointer to the adapter softc in vi_info. There were quite a few places where port_info was being accessed only to get to the adapter. Reviewed by: jhb@ MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25432
2020-06-25 17:04:22 +00:00
struct adapter *sc = vi->adapter;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
u_int total, remaining; /* # of packets */
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
u_int n, avail, dbdiff; /* # of hardware descriptors */
int i, rc;
struct mbuf *m0;
bool snd;
void *wr; /* start of the last WR written to the ring */
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
TXQ_LOCK_ASSERT_OWNED(txq);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
remaining = IDXDIFF(pidx, cidx, r->size);
cxgbe(4): Do not assume that if_qflush is always followed by inteface-down. MFC after: 3 days Sponsored by: Chelsio Communications
2017-05-09 18:33:41 +00:00
if (__predict_false(discard_tx(eq))) {
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
for (i = 0; i < txp->npkt; i++)
m_freem(txp->mb[i]);
txp->npkt = 0;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
while (cidx != pidx) {
m0 = r->items[cidx];
m_freem(m0);
if (++cidx == r->size)
cidx = 0;
}
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
reclaim_tx_descs(txq, eq->sidx);
*coalescing = false;
return (remaining); /* emptied */
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
/* How many hardware descriptors do we have readily available. */
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
if (eq->pidx == eq->cidx) {
avail = eq->sidx - 1;
if (txp->score++ >= 5)
txp->score = 5; /* tx is completely idle, reset. */
} else
avail = IDXDIFF(eq->cidx, eq->pidx, eq->sidx) - 1;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
total = 0;
if (remaining == 0) {
if (txp->score-- == 1) /* egr_update had to drain txpkts */
txp->score = 1;
goto send_txpkts;
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
dbdiff = 0;
MPASS(remaining > 0);
while (remaining > 0) {
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
m0 = r->items[cidx];
M_ASSERTPKTHDR(m0);
MPASS(m0->m_nextpkt == NULL);
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
if (avail < 2 * SGE_MAX_WR_NDESC)
avail += reclaim_tx_descs(txq, 64);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
if (txp->npkt > 0 || remaining > 1 || txp->score > 3 ||
atomic_load_int(&txq->eq.equiq) != 0) {
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
if (vi->flags & TX_USES_VM_WR)
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
rc = add_to_txpkts_vf(sc, txq, m0, avail, &snd);
else
rc = add_to_txpkts_pf(sc, txq, m0, avail, &snd);
} else {
snd = false;
rc = EINVAL;
}
if (snd) {
MPASS(txp->npkt > 0);
for (i = 0; i < txp->npkt; i++)
ETHER_BPF_MTAP(ifp, txp->mb[i]);
if (txp->npkt > 1) {
if (txp->score++ >= 10)
txp->score = 10;
MPASS(avail >= tx_len16_to_desc(txp->len16));
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
if (vi->flags & TX_USES_VM_WR)
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
n = write_txpkts_vm_wr(sc, txq);
else
n = write_txpkts_wr(sc, txq);
} else {
MPASS(avail >=
tx_len16_to_desc(mbuf_len16(txp->mb[0])));
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
if (vi->flags & TX_USES_VM_WR)
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
n = write_txpkt_vm_wr(sc, txq,
txp->mb[0]);
else
n = write_txpkt_wr(sc, txq, txp->mb[0],
avail);
}
MPASS(n <= SGE_MAX_WR_NDESC);
avail -= n;
dbdiff += n;
wr = &eq->desc[eq->pidx];
IDXINCR(eq->pidx, n, eq->sidx);
txp->npkt = 0; /* emptied */
}
if (rc == 0) {
/* m0 was coalesced into txq->txpkts. */
goto next_mbuf;
}
if (rc == EAGAIN) {
/*
* m0 is suitable for tx coalescing but could not be
* combined with the existing txq->txpkts, which has now
* been transmitted. Start a new txpkts with m0.
*/
MPASS(snd);
MPASS(txp->npkt == 0);
continue;
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
MPASS(rc != 0 && rc != EAGAIN);
MPASS(txp->npkt == 0);
cxgbe(4): Check for descriptors before writing a TLS or raw work request. This fixes a regression in r362905. Submitted by: jhb@ Sponsored by: Chelsio Communications
2020-08-31 22:44:59 +00:00
n = tx_len16_to_desc(mbuf_len16(m0));
if (__predict_false(avail < n)) {
avail += reclaim_tx_descs(txq, min(n, 32));
if (avail < n)
break; /* out of descriptors */
}
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
wr = &eq->desc[eq->pidx];
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
if (mbuf_cflags(m0) & MC_RAW_WR) {
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
n = write_raw_wr(txq, wr, m0, avail);
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
#ifdef KERN_TLS
} else if (mbuf_cflags(m0) & MC_TLS) {
ETHER_BPF_MTAP(ifp, m0);
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
n = t6_ktls_write_wr(txq, wr, m0, mbuf_nsegs(m0),
avail);
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
#endif
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
} else {
cxgbe(4): Fix a bug (introduced in r362905) where some tx traffic wasn't being reported to BPF.
2020-07-05 05:14:33 +00:00
ETHER_BPF_MTAP(ifp, m0);
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
if (vi->flags & TX_USES_VM_WR)
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
n = write_txpkt_vm_wr(sc, txq, m0);
else
n = write_txpkt_wr(sc, txq, m0, avail);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
MPASS(n >= 1 && n <= avail);
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
if (!(mbuf_cflags(m0) & MC_TLS))
MPASS(n <= SGE_MAX_WR_NDESC);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
avail -= n;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
dbdiff += n;
IDXINCR(eq->pidx, n, eq->sidx);
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
if (dbdiff >= 512 / EQ_ESIZE) { /* X_FETCHBURSTMAX_512B */
if (wr_can_update_eq(wr))
set_txupdate_flags(txq, avail, wr);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
ring_eq_db(sc, eq, dbdiff);
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
avail += reclaim_tx_descs(txq, 32);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
dbdiff = 0;
}
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
next_mbuf:
total++;
remaining--;
if (__predict_false(++cidx == r->size))
cidx = 0;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
if (dbdiff != 0) {
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
if (wr_can_update_eq(wr))
set_txupdate_flags(txq, avail, wr);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
ring_eq_db(sc, eq, dbdiff);
reclaim_tx_descs(txq, 32);
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
} else if (eq->pidx == eq->cidx && txp->npkt > 0 &&
atomic_load_int(&txq->eq.equiq) == 0) {
/*
* If nothing was submitted to the chip for tx (it was coalesced
* into txpkts instead) and there is no tx update outstanding
* then we need to send txpkts now.
*/
send_txpkts:
MPASS(txp->npkt > 0);
for (i = 0; i < txp->npkt; i++)
ETHER_BPF_MTAP(ifp, txp->mb[i]);
if (txp->npkt > 1) {
MPASS(avail >= tx_len16_to_desc(txp->len16));
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
if (vi->flags & TX_USES_VM_WR)
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
n = write_txpkts_vm_wr(sc, txq);
else
n = write_txpkts_wr(sc, txq);
} else {
MPASS(avail >=
tx_len16_to_desc(mbuf_len16(txp->mb[0])));
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
if (vi->flags & TX_USES_VM_WR)
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
n = write_txpkt_vm_wr(sc, txq, txp->mb[0]);
else
n = write_txpkt_wr(sc, txq, txp->mb[0], avail);
}
MPASS(n <= SGE_MAX_WR_NDESC);
wr = &eq->desc[eq->pidx];
IDXINCR(eq->pidx, n, eq->sidx);
txp->npkt = 0; /* emptied */
MPASS(wr_can_update_eq(wr));
set_txupdate_flags(txq, avail - n, wr);
ring_eq_db(sc, eq, n);
reclaim_tx_descs(txq, 32);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
*coalescing = txp->npkt > 0;
Consider the total number of descriptors available (and not just those that are ready to be reclaimed) when deciding whether to resume tx after a stall. MFC after: 3 days
2014-06-20 20:28:46 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (total);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
static inline void
init_iq(struct sge_iq *iq, struct adapter *sc, int tmr_idx, int pktc_idx,
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
int qsize)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
KASSERT(tmr_idx >= 0 && tmr_idx < SGE_NTIMERS,
("%s: bad tmr_idx %d", __func__, tmr_idx));
KASSERT(pktc_idx < SGE_NCOUNTERS, /* -ve is ok, means don't use */
("%s: bad pktc_idx %d", __func__, pktc_idx));
iq->flags = 0;
iq->adapter = sc;
Change the default to not use packet counters to generate rx interrupts. Rely solely on the timer based mechanism. Update man page to reflect this change. MFC after: 1 week
2012-04-30 09:46:05 +00:00
iq->intr_params = V_QINTR_TIMER_IDX(tmr_idx);
iq->intr_pktc_idx = SGE_NCOUNTERS - 1;
if (pktc_idx >= 0) {
iq->intr_params |= F_QINTR_CNT_EN;
iq->intr_pktc_idx = pktc_idx;
}
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
iq->qsize = roundup2(qsize, 16); /* See FW_IQ_CMD/iqsize */
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
iq->sidx = iq->qsize - sc->params.sge.spg_len / IQ_ESIZE;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
static inline void
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
init_fl(struct adapter *sc, struct sge_fl *fl, int qsize, int maxp, char *name)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
fl->qsize = qsize;
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
fl->sidx = qsize - sc->params.sge.spg_len / EQ_ESIZE;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
strlcpy(fl->lockname, name, sizeof(fl->lockname));
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
if (sc->flags & BUF_PACKING_OK &&
((!is_t4(sc) && buffer_packing) || /* T5+: enabled unless 0 */
(is_t4(sc) && buffer_packing == 1)))/* T4: disabled unless 1 */
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
fl->flags |= FL_BUF_PACKING;
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
fl->zidx = find_refill_source(sc, maxp, fl->flags & FL_BUF_PACKING);
fl->safe_zidx = sc->sge.safe_zidx;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
static inline void
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
init_eq(struct adapter *sc, struct sge_eq *eq, int eqtype, int qsize,
uint8_t tx_chan, uint16_t iqid, char *name)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
KASSERT(eqtype <= EQ_TYPEMASK, ("%s: bad qtype %d", __func__, eqtype));
eq->flags = eqtype & EQ_TYPEMASK;
eq->tx_chan = tx_chan;
eq->iqid = iqid;
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
eq->sidx = qsize - sc->params.sge.spg_len / EQ_ESIZE;
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
strlcpy(eq->lockname, name, sizeof(eq->lockname));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
static int
alloc_ring(struct adapter *sc, size_t len, bus_dma_tag_t *tag,
bus_dmamap_t *map, bus_addr_t *pa, void **va)
{
int rc;
rc = bus_dma_tag_create(sc->dmat, 512, 0, BUS_SPACE_MAXADDR,
BUS_SPACE_MAXADDR, NULL, NULL, len, 1, len, 0, NULL, NULL, tag);
if (rc != 0) {
device_printf(sc->dev, "cannot allocate DMA tag: %d\n", rc);
goto done;
}
rc = bus_dmamem_alloc(*tag, va,
BUS_DMA_WAITOK | BUS_DMA_COHERENT | BUS_DMA_ZERO, map);
if (rc != 0) {
device_printf(sc->dev, "cannot allocate DMA memory: %d\n", rc);
goto done;
}
rc = bus_dmamap_load(*tag, *map, *va, len, oneseg_dma_callback, pa, 0);
if (rc != 0) {
device_printf(sc->dev, "cannot load DMA map: %d\n", rc);
goto done;
}
done:
if (rc)
free_ring(sc, *tag, *map, *pa, *va);
return (rc);
}
static int
free_ring(struct adapter *sc, bus_dma_tag_t tag, bus_dmamap_t map,
bus_addr_t pa, void *va)
{
if (pa)
bus_dmamap_unload(tag, map);
if (va)
bus_dmamem_free(tag, va, map);
if (tag)
bus_dma_tag_destroy(tag);
return (0);
}
/*
* Allocates the ring for an ingress queue and an optional freelist. If the
* freelist is specified it will be allocated and then associated with the
* ingress queue.
*
* Returns errno on failure. Resources allocated up to that point may still be
* allocated. Caller is responsible for cleanup in case this function fails.
*
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
* If the ingress queue will take interrupts directly then the intr_idx
* specifies the vector, starting from 0. -1 means the interrupts for this
* queue should be forwarded to the fwq.
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
*/
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
alloc_iq_fl(struct vi_info *vi, struct sge_iq *iq, struct sge_fl *fl,
- Enable per-channel congestion notification. - Enable PCIe relaxed ordering for all egress queues and rx data buffers. MFC after: 3 days
2011-05-18 22:09:04 +00:00
int intr_idx, int cong)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
int rc, i, cntxt_id;
size_t len;
struct fw_iq_cmd c;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
struct port_info *pi = vi->pi;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct adapter *sc = iq->adapter;
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
struct sge_params *sp = &sc->params.sge;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
__be32 v = 0;
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
len = iq->qsize * IQ_ESIZE;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
rc = alloc_ring(sc, len, &iq->desc_tag, &iq->desc_map, &iq->ba,
(void **)&iq->desc);
if (rc != 0)
return (rc);
bzero(&c, sizeof(c));
c.op_to_vfn = htobe32(V_FW_CMD_OP(FW_IQ_CMD) | F_FW_CMD_REQUEST |
F_FW_CMD_WRITE | F_FW_CMD_EXEC | V_FW_IQ_CMD_PFN(sc->pf) |
V_FW_IQ_CMD_VFN(0));
c.alloc_to_len16 = htobe32(F_FW_IQ_CMD_ALLOC | F_FW_IQ_CMD_IQSTART |
FW_LEN16(c));
/* Special handling for firmware event queue */
if (iq == &sc->sge.fwq)
v |= F_FW_IQ_CMD_IQASYNCH;
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
if (intr_idx < 0) {
/* Forwarded interrupts, all headed to fwq */
v |= F_FW_IQ_CMD_IQANDST;
v |= V_FW_IQ_CMD_IQANDSTINDEX(sc->sge.fwq.cntxt_id);
} else {
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
KASSERT(intr_idx < sc->intr_count,
("%s: invalid direct intr_idx %d", __func__, intr_idx));
cxgbe(4): Do not forward interrupts to queues with freelists. This leaves the firmware event queue (fwq) as the only queue that can take interrupts for others. This simplifies cfg_itype_and_nqueues and queue allocation in the driver at the cost of a little (never?) used configuration. It also allows service_iq to be split into two specialized variants in the future. MFC after: 2 months Sponsored by: Chelsio Communications
2017-12-22 19:10:19 +00:00
v |= V_FW_IQ_CMD_IQANDSTINDEX(intr_idx);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
c.type_to_iqandstindex = htobe32(v |
V_FW_IQ_CMD_TYPE(FW_IQ_TYPE_FL_INT_CAP) |
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
V_FW_IQ_CMD_VIID(vi->viid) |
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
V_FW_IQ_CMD_IQANUD(X_UPDATEDELIVERY_INTERRUPT));
c.iqdroprss_to_iqesize = htobe16(V_FW_IQ_CMD_IQPCIECH(pi->tx_chan) |
F_FW_IQ_CMD_IQGTSMODE |
V_FW_IQ_CMD_IQINTCNTTHRESH(iq->intr_pktc_idx) |
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
V_FW_IQ_CMD_IQESIZE(ilog2(IQ_ESIZE) - 4));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
c.iqsize = htobe16(iq->qsize);
c.iqaddr = htobe64(iq->ba);
- Enable per-channel congestion notification. - Enable PCIe relaxed ordering for all egress queues and rx data buffers. MFC after: 3 days
2011-05-18 22:09:04 +00:00
if (cong >= 0)
c.iqns_to_fl0congen = htobe32(F_FW_IQ_CMD_IQFLINTCONGEN);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
if (fl) {
mtx_init(&fl->fl_lock, fl->lockname, NULL, MTX_DEF);
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
len = fl->qsize * EQ_ESIZE;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
rc = alloc_ring(sc, len, &fl->desc_tag, &fl->desc_map,
&fl->ba, (void **)&fl->desc);
if (rc)
return (rc);
/* Allocate space for one software descriptor per buffer. */
rc = alloc_fl_sdesc(fl);
if (rc != 0) {
device_printf(sc->dev,
"failed to setup fl software descriptors: %d\n",
rc);
return (rc);
}
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
if (fl->flags & FL_BUF_PACKING) {
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
fl->lowat = roundup2(sp->fl_starve_threshold2, 8);
fl->buf_boundary = sp->pack_boundary;
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
} else {
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
fl->lowat = roundup2(sp->fl_starve_threshold, 8);
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
fl->buf_boundary = 16;
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
}
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
if (fl_pad && fl->buf_boundary < sp->pad_boundary)
fl->buf_boundary = sp->pad_boundary;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Do not clobber the ingress queue's congestion setting. MFC after: 1 month
2011-12-14 05:34:23 +00:00
c.iqns_to_fl0congen |=
- Enable per-channel congestion notification. - Enable PCIe relaxed ordering for all egress queues and rx data buffers. MFC after: 3 days
2011-05-18 22:09:04 +00:00
htobe32(V_FW_IQ_CMD_FL0HOSTFCMODE(X_HOSTFCMODE_NONE) |
F_FW_IQ_CMD_FL0FETCHRO | F_FW_IQ_CMD_FL0DATARO |
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
(fl_pad ? F_FW_IQ_CMD_FL0PADEN : 0) |
(fl->flags & FL_BUF_PACKING ? F_FW_IQ_CMD_FL0PACKEN :
0));
- Enable per-channel congestion notification. - Enable PCIe relaxed ordering for all egress queues and rx data buffers. MFC after: 3 days
2011-05-18 22:09:04 +00:00
if (cong >= 0) {
c.iqns_to_fl0congen |=
htobe32(V_FW_IQ_CMD_FL0CNGCHMAP(cong) |
F_FW_IQ_CMD_FL0CONGCIF |
F_FW_IQ_CMD_FL0CONGEN);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
c.fl0dcaen_to_fl0cidxfthresh =
cxgbe(4): Use smaller min/max bursts for fl descriptors with a T6. Sponsored by: Chelsio Communications
2016-09-11 17:51:17 +00:00
htobe16(V_FW_IQ_CMD_FL0FBMIN(chip_id(sc) <= CHELSIO_T5 ?
cxgbe(4): Use correct FetchBurstMin values for T6. MFC after: 1 week Sponsored by: Chelsio Communications
2019-10-25 21:53:05 +00:00
X_FETCHBURSTMIN_128B : X_FETCHBURSTMIN_64B_T6) |
cxgbe(4): Use smaller min/max bursts for fl descriptors with a T6. Sponsored by: Chelsio Communications
2016-09-11 17:51:17 +00:00
V_FW_IQ_CMD_FL0FBMAX(chip_id(sc) <= CHELSIO_T5 ?
X_FETCHBURSTMAX_512B : X_FETCHBURSTMAX_256B));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
c.fl0size = htobe16(fl->qsize);
c.fl0addr = htobe64(fl->ba);
}
rc = -t4_wr_mbox(sc, sc->mbox, &c, sizeof(c), &c);
if (rc != 0) {
device_printf(sc->dev,
"failed to create ingress queue: %d\n", rc);
return (rc);
}
iq->cidx = 0;
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
iq->gen = F_RSPD_GEN;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
iq->intr_next = iq->intr_params;
iq->cntxt_id = be16toh(c.iqid);
iq->abs_id = be16toh(c.physiqid);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
iq->flags |= IQ_ALLOCATED;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cntxt_id = iq->cntxt_id - sc->sge.iq_start;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (cntxt_id >= sc->sge.niq) {
panic ("%s: iq->cntxt_id (%d) more than the max (%d)", __func__,
cntxt_id, sc->sge.niq - 1);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
sc->sge.iqmap[cntxt_id] = iq;
if (fl) {
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
u_int qid;
iq->flags |= IQ_HAS_FL;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
fl->cntxt_id = be16toh(c.fl0id);
fl->pidx = fl->cidx = 0;
Fix an error while constructing the table that maps context id -> egress queue. MFC after: 1 day
2011-03-22 21:05:56 +00:00
cntxt_id = fl->cntxt_id - sc->sge.eq_start;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (cntxt_id >= sc->sge.neq) {
panic("%s: fl->cntxt_id (%d) more than the max (%d)",
__func__, cntxt_id, sc->sge.neq - 1);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
sc->sge.eqmap[cntxt_id] = (void *)fl;
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
qid = fl->cntxt_id;
if (isset(&sc->doorbells, DOORBELL_UDB)) {
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
uint32_t s_qpp = sc->params.sge.eq_s_qpp;
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
uint32_t mask = (1 << s_qpp) - 1;
volatile uint8_t *udb;
udb = sc->udbs_base + UDBS_DB_OFFSET;
udb += (qid >> s_qpp) << PAGE_SHIFT;
qid &= mask;
if (qid < PAGE_SIZE / UDBS_SEG_SIZE) {
udb += qid << UDBS_SEG_SHIFT;
qid = 0;
}
fl->udb = (volatile void *)udb;
}
cxgbe(4): Very basic T6 awareness. This is part of ongoing work to update to the latest internal shared code. - Add a chip_params structure to keep track of hardware constants for all generations of Terminators handled by cxgbe. - Update t4_hw_pci_read_cfg4 to work with T6. - Update the hardware debug sysctls (hidden within dev.<tNnex>.<n>.misc.*) to work with T6. Most of the changes are in the decoders for the CIM logic analyzer and the MPS TCAM. - Acquire the regwin lock around indirect register accesses. Obtained from: Chelsio Communications Sponsored by: Chelsio Communications
2016-03-04 13:11:13 +00:00
fl->dbval = V_QID(qid) | sc->chip_params->sge_fl_db;
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
FL_LOCK(fl);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
/* Enough to make sure the SGE doesn't think it's starved */
refill_fl(sc, fl, fl->lowat);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
FL_UNLOCK(fl);
}
cxgbe(4): Setup congestion response for T6 rx queues.
2016-09-21 00:50:22 +00:00
if (chip_id(sc) >= CHELSIO_T5 && !(sc->flags & IS_VF) && cong >= 0) {
Set up congestion manager context properly for T5 based cards. MFC after: 3 days (will check with re@)
2013-08-01 23:38:30 +00:00
uint32_t param, val;
param = V_FW_PARAMS_MNEM(FW_PARAMS_MNEM_DMAQ) |
V_FW_PARAMS_PARAM_X(FW_PARAMS_PARAM_DMAQ_CONM_CTXT) |
V_FW_PARAMS_PARAM_YZ(iq->cntxt_id);
Fix previous commit (r253873). "cong" has one bit per channel but the congestion channel map has 1 nibble per channel. So bits wxyz need to be blown up into 000w000x000y000z.
2013-08-02 17:44:19 +00:00
if (cong == 0)
val = 1 << 19;
else {
val = 2 << 19;
for (i = 0; i < 4; i++) {
if (cong & (1 << i))
val |= 1 << (i << 2);
}
}
Set up congestion manager context properly for T5 based cards. MFC after: 3 days (will check with re@)
2013-08-01 23:38:30 +00:00
rc = -t4_set_params(sc, sc->mbox, sc->pf, 0, 1, &param, &val);
if (rc != 0) {
/* report error but carry on */
device_printf(sc->dev,
"failed to set congestion manager context for "
"ingress queue %d: %d\n", iq->cntxt_id, rc);
}
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
/* Enable IQ interrupts */
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
atomic_store_rel_int(&iq->state, IQS_IDLE);
Store the offset of the KDOORBELL and GTS registers in the softc. VF devices use a different register layout than PF devices. Storing the offset in a value in the softc allows code to be shared between the PF and VF drivers. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7389
2016-08-01 22:39:51 +00:00
t4_write_reg(sc, sc->sge_gts_reg, V_SEINTARM(iq->intr_params) |
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
V_INGRESSQID(iq->cntxt_id));
return (0);
}
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
free_iq_fl(struct vi_info *vi, struct sge_iq *iq, struct sge_fl *fl)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
int rc;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct adapter *sc = iq->adapter;
device_t dev;
if (sc == NULL)
return (0); /* nothing to do */
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
dev = vi ? vi->dev : sc->dev;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
if (iq->flags & IQ_ALLOCATED) {
rc = -t4_iq_free(sc, sc->mbox, sc->pf, 0,
FW_IQ_TYPE_FL_INT_CAP, iq->cntxt_id,
fl ? fl->cntxt_id : 0xffff, 0xffff);
if (rc != 0) {
device_printf(dev,
"failed to free queue %p: %d\n", iq, rc);
return (rc);
}
iq->flags &= ~IQ_ALLOCATED;
}
free_ring(sc, iq->desc_tag, iq->desc_map, iq->ba, iq->desc);
bzero(iq, sizeof(*iq));
if (fl) {
free_ring(sc, fl->desc_tag, fl->desc_map, fl->ba,
fl->desc);
There is no need to hold the freelist lock around alloc/free of software descriptors. This also silences WITNESS warnings when the software descriptors are allocated with M_WAITOK. MFC after: 1 week
2013-08-23 18:03:18 +00:00
if (fl->sdesc)
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
free_fl_sdesc(sc, fl);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
if (mtx_initialized(&fl->fl_lock))
mtx_destroy(&fl->fl_lock);
bzero(fl, sizeof(*fl));
}
return (0);
}
cxgbe(4): Reduce duplication by consolidating minor variations of the same code into a single routine. Sponsored by: Chelsio Communications
2017-12-29 02:30:21 +00:00
static void
add_iq_sysctls(struct sysctl_ctx_list *ctx, struct sysctl_oid *oid,
struct sge_iq *iq)
{
struct sysctl_oid_list *children = SYSCTL_CHILDREN(oid);
SYSCTL_ADD_UAUTO(ctx, children, OID_AUTO, "ba", CTLFLAG_RD, &iq->ba,
"bus address of descriptor ring");
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "dmalen", CTLFLAG_RD, NULL,
iq->qsize * IQ_ESIZE, "descriptor ring size in bytes");
SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "abs_id",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &iq->abs_id, 0,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
sysctl_uint16, "I", "absolute id of the queue");
cxgbe(4): Reduce duplication by consolidating minor variations of the same code into a single routine. Sponsored by: Chelsio Communications
2017-12-29 02:30:21 +00:00
SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "cntxt_id",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &iq->cntxt_id, 0,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
sysctl_uint16, "I", "SGE context id of the queue");
cxgbe(4): Reduce duplication by consolidating minor variations of the same code into a single routine. Sponsored by: Chelsio Communications
2017-12-29 02:30:21 +00:00
SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "cidx",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &iq->cidx, 0,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
sysctl_uint16, "I", "consumer index");
cxgbe(4): Reduce duplication by consolidating minor variations of the same code into a single routine. Sponsored by: Chelsio Communications
2017-12-29 02:30:21 +00:00
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
static void
cxgbe(4): Make the location/length of all descriptor rings available in the sysctl MIB.
2016-09-23 20:03:28 +00:00
add_fl_sysctls(struct adapter *sc, struct sysctl_ctx_list *ctx,
struct sysctl_oid *oid, struct sge_fl *fl)
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
{
struct sysctl_oid_list *children = SYSCTL_CHILDREN(oid);
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
oid = SYSCTL_ADD_NODE(ctx, children, OID_AUTO, "fl",
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "freelist");
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
children = SYSCTL_CHILDREN(oid);
cxgbe(4): Make the location/length of all descriptor rings available in the sysctl MIB.
2016-09-23 20:03:28 +00:00
SYSCTL_ADD_UAUTO(ctx, children, OID_AUTO, "ba", CTLFLAG_RD,
&fl->ba, "bus address of descriptor ring");
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "dmalen", CTLFLAG_RD, NULL,
fl->sidx * EQ_ESIZE + sc->params.sge.spg_len,
"desc ring size in bytes");
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "cntxt_id",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &fl->cntxt_id, 0,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
sysctl_uint16, "I", "SGE context id of the freelist");
cxgbe(4): Allow for different pad and pack boundaries for different adapters. Set the pack boundary for T5 cards to be the same as the PCIe max payload size. The chip likes it this way. In this revision the driver allocate rx buffers that align on both boundaries. This is not a strict requirement and a followup commit will switch the driver to a more relaxed allocation strategy. MFC after: 2 weeks
2014-12-06 00:13:56 +00:00
SYSCTL_ADD_UINT(ctx, children, OID_AUTO, "padding", CTLFLAG_RD, NULL,
fl_pad ? 1 : 0, "padding enabled");
SYSCTL_ADD_UINT(ctx, children, OID_AUTO, "packing", CTLFLAG_RD, NULL,
fl->flags & FL_BUF_PACKING ? 1 : 0, "packing enabled");
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
SYSCTL_ADD_UINT(ctx, children, OID_AUTO, "cidx", CTLFLAG_RD, &fl->cidx,
0, "consumer index");
if (fl->flags & FL_BUF_PACKING) {
SYSCTL_ADD_UINT(ctx, children, OID_AUTO, "rx_offset",
CTLFLAG_RD, &fl->rx_offset, 0, "packing rx offset");
}
SYSCTL_ADD_UINT(ctx, children, OID_AUTO, "pidx", CTLFLAG_RD, &fl->pidx,
0, "producer index");
SYSCTL_ADD_UQUAD(ctx, children, OID_AUTO, "cluster_allocated",
CTLFLAG_RD, &fl->cl_allocated, "# of clusters allocated");
SYSCTL_ADD_UQUAD(ctx, children, OID_AUTO, "cluster_recycled",
CTLFLAG_RD, &fl->cl_recycled, "# of clusters recycled");
SYSCTL_ADD_UQUAD(ctx, children, OID_AUTO, "cluster_fast_recycled",
CTLFLAG_RD, &fl->cl_fast_recycled, "# of clusters recycled (fast)");
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
static int
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
alloc_fwq(struct adapter *sc)
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
{
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
int rc, intr_idx;
struct sge_iq *fwq = &sc->sge.fwq;
struct sysctl_oid *oid = device_get_sysctl_tree(sc->dev);
struct sysctl_oid_list *children = SYSCTL_CHILDREN(oid);
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
cxgbe(4): minor optimizations in ingress queue processing. Reorganize struct sge_iq. Make the iq entry size a compile time constant. While here, eliminate RX_FL_ESIZE and use EQ_ESIZE directly. MFC after: 2 weeks
2014-08-02 00:56:34 +00:00
init_iq(fwq, sc, 0, 0, FW_IQ_QSIZE);
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
if (sc->flags & IS_VF)
intr_idx = 0;
cxgbe(4): Use opaque cookies or tid range-checks to determine the intended recipient of a CPL when it can't be determined solely from the opcode. Retire the per-queue handlers for such CPLs in favor of the new scheme. Sponsored by: Chelsio Communications
2018-04-30 15:18:38 +00:00
else
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
intr_idx = sc->intr_count > 1 ? 1 : 0;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
rc = alloc_iq_fl(&sc->port[0]->vi[0], fwq, NULL, intr_idx, -1);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (rc != 0) {
device_printf(sc->dev,
"failed to create firmware event queue: %d\n", rc);
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
return (rc);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
}
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
oid = SYSCTL_ADD_NODE(&sc->ctx, children, OID_AUTO, "fwq",
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "firmware event queue");
cxgbe(4): Reduce duplication by consolidating minor variations of the same code into a single routine. Sponsored by: Chelsio Communications
2017-12-29 02:30:21 +00:00
add_iq_sysctls(&sc->ctx, oid, fwq);
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
return (0);
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
}
static int
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
free_fwq(struct adapter *sc)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
return free_iq_fl(NULL, &sc->sge.fwq, NULL);
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
}
static int
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
alloc_ctrlq(struct adapter *sc, struct sge_wrq *ctrlq, int idx,
struct sysctl_oid *oid)
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
{
int rc;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
char name[16];
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
struct sysctl_oid_list *children;
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
snprintf(name, sizeof(name), "%s ctrlq%d", device_get_nameunit(sc->dev),
idx);
init_eq(sc, &ctrlq->eq, EQ_CTRL, CTRL_EQ_QSIZE, sc->port[idx]->tx_chan,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
sc->sge.fwq.cntxt_id, name);
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
children = SYSCTL_CHILDREN(oid);
snprintf(name, sizeof(name), "%d", idx);
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
oid = SYSCTL_ADD_NODE(&sc->ctx, children, OID_AUTO, name,
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "ctrl queue");
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
rc = alloc_wrq(sc, NULL, ctrlq, oid);
- Updated TOE support in the kernel. - Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs. These are available as t3_tom and t4_tom modules that augment cxgb(4) and cxgbe(4) respectively. The cxgb/cxgbe drivers continue to work as usual with or without these extra features. - iWARP driver for Terminator 3 ASIC (kernel verbs). T4 iWARP in the works and will follow soon. Build-tested with make universe. 30s overview ============ What interfaces support TCP offload? Look for TOE4 and/or TOE6 in the capabilities of an interface: # ifconfig -m | grep TOE Enable/disable TCP offload on an interface (just like any other ifnet capability): # ifconfig cxgbe0 toe # ifconfig cxgbe0 -toe Which connections are offloaded? Look for toe4 and/or toe6 in the output of netstat and sockstat: # netstat -np tcp | grep toe # sockstat -46c | grep toe Reviewed by: bz, gnn Sponsored by: Chelsio communications. MFC after: ~3 months (after 9.1, and after ensuring MFC is feasible)
2012-06-19 07:34:13 +00:00
cxgbe(4): Move all control queues to the adapter. There used to be one control queue per adapter (the mgmtq) that was initialized during adapter init and one per port that was initialized later during port init. This change moves all the control queues (one per port/channel) to the adapter so that they are initialized during adapter init and are available before any port is up. This allows the driver to issue ctrlq work requests over any channel without having to bring up any port. MFH: 2 weeks Sponsored by: Chelsio Communications
2018-08-11 21:10:08 +00:00
return (rc);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): set up congestion management for netmap rx queues. The hw.cxgbe.cong_drop knob controls the response of the chip when netmap queues are congested.
2015-02-24 18:40:10 +00:00
int
cxgbe(4): Add a new knob that controls the congestion response of netmap rx queues. The default is to drop rather than backpressure. This decouples the congestion settings of NIC and netmap rx queues. MFC after: 3 days
2015-07-06 20:56:59 +00:00
tnl_cong(struct port_info *pi, int drop)
Convert some fixed parameters to tunables (with reasonable default values). - cong_drop specifies what to do on congestion: nothing, backpressure, or drop. - fl_pktshift specifies the padding before Ethernet payload. - fl_pad specifies the boundary upto which to pad Ethernet payload. - spg_len controls the length of the status page. MFC after: 2 weeks
2012-08-14 21:47:41 +00:00
{
cxgbe(4): Add a new knob that controls the congestion response of netmap rx queues. The default is to drop rather than backpressure. This decouples the congestion settings of NIC and netmap rx queues. MFC after: 3 days
2015-07-06 20:56:59 +00:00
if (drop == -1)
Convert some fixed parameters to tunables (with reasonable default values). - cong_drop specifies what to do on congestion: nothing, backpressure, or drop. - fl_pktshift specifies the padding before Ethernet payload. - fl_pad specifies the boundary upto which to pad Ethernet payload. - spg_len controls the length of the status page. MFC after: 2 weeks
2012-08-14 21:47:41 +00:00
return (-1);
cxgbe(4): Add a new knob that controls the congestion response of netmap rx queues. The default is to drop rather than backpressure. This decouples the congestion settings of NIC and netmap rx queues. MFC after: 3 days
2015-07-06 20:56:59 +00:00
else if (drop == 1)
Convert some fixed parameters to tunables (with reasonable default values). - cong_drop specifies what to do on congestion: nothing, backpressure, or drop. - fl_pktshift specifies the padding before Ethernet payload. - fl_pad specifies the boundary upto which to pad Ethernet payload. - spg_len controls the length of the status page. MFC after: 2 weeks
2012-08-14 21:47:41 +00:00
return (0);
else
cxgbe(4): Read the MPS buffer group map from the firmware as it could be different from hardware defaults. The congestion channel map, which is still fixed, needs to be tracked separately now. Change the congestion setting for TOE rx queues to match the drivers on other OSes while here. MFC after: 2 weeks Sponsored by: Chelsio Communications
2017-10-24 05:41:48 +00:00
return (pi->rx_e_chan_map);
Convert some fixed parameters to tunables (with reasonable default values). - cong_drop specifies what to do on congestion: nothing, backpressure, or drop. - fl_pktshift specifies the padding before Ethernet payload. - fl_pad specifies the boundary upto which to pad Ethernet payload. - spg_len controls the length of the status page. MFC after: 2 weeks
2012-08-14 21:47:41 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
alloc_rxq(struct vi_info *vi, struct sge_rxq *rxq, int intr_idx, int idx,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sysctl_oid *oid)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
int rc;
cxgbe(4): Add a pointer to the adapter softc in vi_info. There were quite a few places where port_info was being accessed only to get to the adapter. Reviewed by: jhb@ MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25432
2020-06-25 17:04:22 +00:00
struct adapter *sc = vi->adapter;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct sysctl_oid_list *children;
char name[16];
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
rc = alloc_iq_fl(vi, &rxq->iq, &rxq->fl, intr_idx,
tnl_cong(vi->pi, cong_drop));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
if (rc != 0)
return (rc);
Track the base absolute ID of ingress and egress queues. Use this to map an absolute queue ID to a logical queue ID in interrupt handlers. For the regular cxgbe/cxl drivers this should be a no-op as the base absolute ID should be zero. VF devices have a non-zero base absolute ID and require this change. While here, export the absolute ID of egress queues via a sysctl. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7446
2016-08-09 17:49:42 +00:00
if (idx == 0)
sc->sge.iq_base = rxq->iq.abs_id - rxq->iq.cntxt_id;
else
KASSERT(rxq->iq.cntxt_id + sc->sge.iq_base == rxq->iq.abs_id,
("iq_base mismatch"));
KASSERT(sc->sge.iq_base == 0 || sc->flags & IS_VF,
("PF with non-zero iq_base"));
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
/*
* The freelist is just barely above the starvation threshold right now,
* fill it up a bit more.
*/
Allow lazy fill up of freelists. MFC after: 3 days
2011-06-04 23:31:33 +00:00
FL_LOCK(&rxq->fl);
Track the base absolute ID of ingress and egress queues. Use this to map an absolute queue ID to a logical queue ID in interrupt handlers. For the regular cxgbe/cxl drivers this should be a no-op as the base absolute ID should be zero. VF devices have a non-zero base absolute ID and require this change. While here, export the absolute ID of egress queues via a sysctl. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7446
2016-08-09 17:49:42 +00:00
refill_fl(sc, &rxq->fl, 128);
Allow lazy fill up of freelists. MFC after: 3 days
2011-06-04 23:31:33 +00:00
FL_UNLOCK(&rxq->fl);
cxgbe(4): support for IPv6 TSO and LRO. Submitted by: bz (this is a modified version of that patch)
2012-06-29 19:51:06 +00:00
#if defined(INET) || defined(INET6)
cxgbe: Add tunables to control the number of LRO entries and the number of rx mbufs that should be presorted before LRO. There is no change in default behavior. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-17 09:00:20 +00:00
rc = tcp_lro_init_args(&rxq->lro, vi->ifp, lro_entries, lro_mbufs);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
if (rc != 0)
return (rc);
cxgbe: Add tunables to control the number of LRO entries and the number of rx mbufs that should be presorted before LRO. There is no change in default behavior. MFC after: 1 week Sponsored by: Chelsio Communications
2017-04-17 09:00:20 +00:00
MPASS(rxq->lro.ifp == vi->ifp); /* also indicates LRO init'ed */
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
if (vi->ifp->if_capenable & IFCAP_LRO)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
rxq->iq.flags |= IQ_LRO_ENABLED;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
#endif
cxgbe(4): Make sure the rx queues start off with the correct timestamp settings on initialization. Sponsored by: Chelsio Communications
2018-12-20 20:34:21 +00:00
if (vi->ifp->if_capenable & IFCAP_HWRXTSTMP)
rxq->iq.flags |= IQ_RX_TIMESTAMP;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
rxq->ifp = vi->ifp;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
children = SYSCTL_CHILDREN(oid);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
snprintf(name, sizeof(name), "%d", idx);
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
oid = SYSCTL_ADD_NODE(&vi->ctx, children, OID_AUTO, name,
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "rx queue");
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
children = SYSCTL_CHILDREN(oid);
cxgbe(4): Reduce duplication by consolidating minor variations of the same code into a single routine. Sponsored by: Chelsio Communications
2017-12-29 02:30:21 +00:00
add_iq_sysctls(&vi->ctx, oid, &rxq->iq);
cxgbe(4): support for IPv6 TSO and LRO. Submitted by: bz (this is a modified version of that patch)
2012-06-29 19:51:06 +00:00
#if defined(INET) || defined(INET6)
Add optimizing LRO wrapper: - Add optimizing LRO wrapper which pre-sorts all incoming packets according to the hash type and flowid. This prevents exhaustion of the LRO entries due to too many connections at the same time. Testing using a larger number of higher bandwidth TCP connections showed that the incoming ACK packet aggregation rate increased from ~1.3:1 to almost 3:1. Another test showed that for a number of TCP connections greater than 16 per hardware receive ring, where 8 TCP connections was the LRO active entry limit, there was a significant improvement in throughput due to being able to fully aggregate more than 8 TCP stream. For very few very high bandwidth TCP streams, the optimizing LRO wrapper will add CPU usage instead of reducing CPU usage. This is expected. Network drivers which want to use the optimizing LRO wrapper needs to call "tcp_lro_queue_mbuf()" instead of "tcp_lro_rx()" and "tcp_lro_flush_all()" instead of "tcp_lro_flush()". Further the LRO control structure must be initialized using "tcp_lro_init_args()" passing a non-zero number into the "lro_mbufs" argument. - Make LRO statistics 64-bit. Previously 32-bit integers were used for statistics which can be prone to wrap-around. Fix this while at it and update all SYSCTL's which expose LRO statistics. - Ensure all data is freed when destroying a LRO control structures, especially leftover LRO entries. - Reduce number of memory allocations needed when setting up a LRO control structure by precomputing the total amount of memory needed. - Add own memory allocation counter for LRO. - Bump the FreeBSD version to force recompilation of all KLDs due to change of the LRO control structure size. Sponsored by: Mellanox Technologies Reviewed by: gallatin, sbruno, rrs, gnn, transport Tested by: Netflix Differential Revision: https://reviews.freebsd.org/D4914
2016-01-19 15:33:28 +00:00
SYSCTL_ADD_U64(&vi->ctx, children, OID_AUTO, "lro_queued", CTLFLAG_RD,
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
&rxq->lro.lro_queued, 0, NULL);
Add optimizing LRO wrapper: - Add optimizing LRO wrapper which pre-sorts all incoming packets according to the hash type and flowid. This prevents exhaustion of the LRO entries due to too many connections at the same time. Testing using a larger number of higher bandwidth TCP connections showed that the incoming ACK packet aggregation rate increased from ~1.3:1 to almost 3:1. Another test showed that for a number of TCP connections greater than 16 per hardware receive ring, where 8 TCP connections was the LRO active entry limit, there was a significant improvement in throughput due to being able to fully aggregate more than 8 TCP stream. For very few very high bandwidth TCP streams, the optimizing LRO wrapper will add CPU usage instead of reducing CPU usage. This is expected. Network drivers which want to use the optimizing LRO wrapper needs to call "tcp_lro_queue_mbuf()" instead of "tcp_lro_rx()" and "tcp_lro_flush_all()" instead of "tcp_lro_flush()". Further the LRO control structure must be initialized using "tcp_lro_init_args()" passing a non-zero number into the "lro_mbufs" argument. - Make LRO statistics 64-bit. Previously 32-bit integers were used for statistics which can be prone to wrap-around. Fix this while at it and update all SYSCTL's which expose LRO statistics. - Ensure all data is freed when destroying a LRO control structures, especially leftover LRO entries. - Reduce number of memory allocations needed when setting up a LRO control structure by precomputing the total amount of memory needed. - Add own memory allocation counter for LRO. - Bump the FreeBSD version to force recompilation of all KLDs due to change of the LRO control structure size. Sponsored by: Mellanox Technologies Reviewed by: gallatin, sbruno, rrs, gnn, transport Tested by: Netflix Differential Revision: https://reviews.freebsd.org/D4914
2016-01-19 15:33:28 +00:00
SYSCTL_ADD_U64(&vi->ctx, children, OID_AUTO, "lro_flushed", CTLFLAG_RD,
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
&rxq->lro.lro_flushed, 0, NULL);
Tweaks for rx: - everything related to LRO should be in #ifdef INET blocks - reorder sge_iq's fields so that the most frequently used are all together - pull all rx code into t4_intr_data directly - let go of the ingress queue lock when passing up data - refill the freelist only if it is short of at least 32 buffers
2011-03-05 03:42:03 +00:00
#endif
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "rxcsum", CTLFLAG_RD,
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
&rxq->rxcsum, "# of times hardware assisted with checksum");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "vlan_extraction",
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
CTLFLAG_RD, &rxq->vlan_extraction,
"# of times hardware extracted 802.1Q tag");
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "vxlan_rxcsum",
CTLFLAG_RD, &rxq->vxlan_rxcsum,
"# of times hardware assisted with inner checksum (VXLAN) ");
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Make the location/length of all descriptor rings available in the sysctl MIB.
2016-09-23 20:03:28 +00:00
add_fl_sysctls(sc, &vi->ctx, oid, &rxq->fl);
- driver ioctl to get SGE context for any given queue. - sysctls to display the context id, cidx, and pidx of all kinds of queues. MFC after: 3 days
2011-06-11 04:50:54 +00:00
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
return (rc);
}
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
free_rxq(struct vi_info *vi, struct sge_rxq *rxq)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
int rc;
cxgbe(4): support for IPv6 TSO and LRO. Submitted by: bz (this is a modified version of that patch)
2012-06-29 19:51:06 +00:00
#if defined(INET) || defined(INET6)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (rxq->lro.ifp) {
tcp_lro_free(&rxq->lro);
rxq->lro.ifp = NULL;
}
#endif
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
rc = free_iq_fl(vi, &rxq->iq, &rxq->fl);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (rc == 0)
bzero(rxq, sizeof(*rxq));
return (rc);
}
- Updated TOE support in the kernel. - Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs. These are available as t3_tom and t4_tom modules that augment cxgb(4) and cxgbe(4) respectively. The cxgb/cxgbe drivers continue to work as usual with or without these extra features. - iWARP driver for Terminator 3 ASIC (kernel verbs). T4 iWARP in the works and will follow soon. Build-tested with make universe. 30s overview ============ What interfaces support TCP offload? Look for TOE4 and/or TOE6 in the capabilities of an interface: # ifconfig -m | grep TOE Enable/disable TCP offload on an interface (just like any other ifnet capability): # ifconfig cxgbe0 toe # ifconfig cxgbe0 -toe Which connections are offloaded? Look for toe4 and/or toe6 in the output of netstat and sockstat: # netstat -np tcp | grep toe # sockstat -46c | grep toe Reviewed by: bz, gnn Sponsored by: Chelsio communications. MFC after: ~3 months (after 9.1, and after ensuring MFC is feasible)
2012-06-19 07:34:13 +00:00
#ifdef TCP_OFFLOAD
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
alloc_ofld_rxq(struct vi_info *vi, struct sge_ofld_rxq *ofld_rxq,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
int intr_idx, int idx, struct sysctl_oid *oid)
{
cxgbe(4): Make the location/length of all descriptor rings available in the sysctl MIB.
2016-09-23 20:03:28 +00:00
struct port_info *pi = vi->pi;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
int rc;
struct sysctl_oid_list *children;
char name[16];
cxgbe(4): Read the MPS buffer group map from the firmware as it could be different from hardware defaults. The congestion channel map, which is still fixed, needs to be tracked separately now. Change the congestion setting for TOE rx queues to match the drivers on other OSes while here. MFC after: 2 weeks Sponsored by: Chelsio Communications
2017-10-24 05:41:48 +00:00
rc = alloc_iq_fl(vi, &ofld_rxq->iq, &ofld_rxq->fl, intr_idx, 0);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (rc != 0)
return (rc);
children = SYSCTL_CHILDREN(oid);
snprintf(name, sizeof(name), "%d", idx);
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
oid = SYSCTL_ADD_NODE(&vi->ctx, children, OID_AUTO, name,
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "rx queue");
cxgbe(4): Reduce duplication by consolidating minor variations of the same code into a single routine. Sponsored by: Chelsio Communications
2017-12-29 02:30:21 +00:00
add_iq_sysctls(&vi->ctx, oid, &ofld_rxq->iq);
cxgbe(4): Make the location/length of all descriptor rings available in the sysctl MIB.
2016-09-23 20:03:28 +00:00
add_fl_sysctls(pi->adapter, &vi->ctx, oid, &ofld_rxq->fl);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
return (rc);
}
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
free_ofld_rxq(struct vi_info *vi, struct sge_ofld_rxq *ofld_rxq)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
{
int rc;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
rc = free_iq_fl(vi, &ofld_rxq->iq, &ofld_rxq->fl);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (rc == 0)
bzero(ofld_rxq, sizeof(*ofld_rxq));
return (rc);
}
#endif
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
#ifdef DEV_NETMAP
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
alloc_nm_rxq(struct vi_info *vi, struct sge_nm_rxq *nm_rxq, int intr_idx,
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
int idx, struct sysctl_oid *oid)
{
int rc;
struct sysctl_oid_list *children;
struct sysctl_ctx_list *ctx;
char name[16];
size_t len;
cxgbe(4): Add a pointer to the adapter softc in vi_info. There were quite a few places where port_info was being accessed only to get to the adapter. Reviewed by: jhb@ MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25432
2020-06-25 17:04:22 +00:00
struct adapter *sc = vi->adapter;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
struct netmap_adapter *na = NA(vi->ifp);
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
MPASS(na != NULL);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
len = vi->qsize_rxq * IQ_ESIZE;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
rc = alloc_ring(sc, len, &nm_rxq->iq_desc_tag, &nm_rxq->iq_desc_map,
&nm_rxq->iq_ba, (void **)&nm_rxq->iq_desc);
if (rc != 0)
return (rc);
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
len = na->num_rx_desc * EQ_ESIZE + sc->params.sge.spg_len;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
rc = alloc_ring(sc, len, &nm_rxq->fl_desc_tag, &nm_rxq->fl_desc_map,
&nm_rxq->fl_ba, (void **)&nm_rxq->fl_desc);
if (rc != 0)
return (rc);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
nm_rxq->vi = vi;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
nm_rxq->nid = idx;
nm_rxq->iq_cidx = 0;
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
nm_rxq->iq_sidx = vi->qsize_rxq - sc->params.sge.spg_len / IQ_ESIZE;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
nm_rxq->iq_gen = F_RSPD_GEN;
nm_rxq->fl_pidx = nm_rxq->fl_cidx = 0;
nm_rxq->fl_sidx = na->num_rx_desc;
cxgbe(4): Split sge_nm_rxq into three cachelines. This reduces the lines bouncing around between the driver rx ithread and the netmap rxsync thread. There is no net change in the size of the struct (it continues to waste a lot of space). This kind of split was originally proposed in D17869 by Marc De La Gueronniere @ Verisign, Inc. MFC after: 1 week Sponsored by: Chelsio Communications
2020-03-20 05:12:16 +00:00
nm_rxq->fl_sidx2 = nm_rxq->fl_sidx; /* copy for rxsync cacheline */
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
nm_rxq->intr_idx = intr_idx;
cxgbe(4): Fix per-queue netmap operation. Do not attempt to initialize netmap queues that are already initialized or aren't supposed to be initialized. Similarly, do not free queues that are not initialized or aren't supposed to be freed. PR: 217156 Sponsored by: Chelsio Communications
2017-06-15 19:56:59 +00:00
nm_rxq->iq_cntxt_id = INVALID_NM_RXQ_CNTXT_ID;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
ctx = &vi->ctx;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
children = SYSCTL_CHILDREN(oid);
snprintf(name, sizeof(name), "%d", idx);
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
oid = SYSCTL_ADD_NODE(ctx, children, OID_AUTO, name,
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "rx queue");
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
children = SYSCTL_CHILDREN(oid);
SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "abs_id",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &nm_rxq->iq_abs_id,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
0, sysctl_uint16, "I", "absolute id of the queue");
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "cntxt_id",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &nm_rxq->iq_cntxt_id,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
0, sysctl_uint16, "I", "SGE context id of the queue");
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "cidx",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &nm_rxq->iq_cidx, 0,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
sysctl_uint16, "I", "consumer index");
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
children = SYSCTL_CHILDREN(oid);
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
oid = SYSCTL_ADD_NODE(ctx, children, OID_AUTO, "fl",
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "freelist");
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
children = SYSCTL_CHILDREN(oid);
SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "cntxt_id",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &nm_rxq->fl_cntxt_id,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
0, sysctl_uint16, "I", "SGE context id of the freelist");
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
SYSCTL_ADD_UINT(ctx, children, OID_AUTO, "cidx", CTLFLAG_RD,
&nm_rxq->fl_cidx, 0, "consumer index");
SYSCTL_ADD_UINT(ctx, children, OID_AUTO, "pidx", CTLFLAG_RD,
&nm_rxq->fl_pidx, 0, "producer index");
return (rc);
}
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
free_nm_rxq(struct vi_info *vi, struct sge_nm_rxq *nm_rxq)
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
{
cxgbe(4): Add a pointer to the adapter softc in vi_info. There were quite a few places where port_info was being accessed only to get to the adapter. Reviewed by: jhb@ MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25432
2020-06-25 17:04:22 +00:00
struct adapter *sc = vi->adapter;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
cxgbe(4): Fix some assertions during driver detach. The netmap queues can't be initialized if the VI isn't. MFC after: 3 days
2017-08-28 03:25:41 +00:00
if (vi->flags & VI_INIT_DONE)
MPASS(nm_rxq->iq_cntxt_id == INVALID_NM_RXQ_CNTXT_ID);
else
MPASS(nm_rxq->iq_cntxt_id == 0);
cxgbe(4): Fix per-queue netmap operation. Do not attempt to initialize netmap queues that are already initialized or aren't supposed to be initialized. Similarly, do not free queues that are not initialized or aren't supposed to be freed. PR: 217156 Sponsored by: Chelsio Communications
2017-06-15 19:56:59 +00:00
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
free_ring(sc, nm_rxq->iq_desc_tag, nm_rxq->iq_desc_map, nm_rxq->iq_ba,
nm_rxq->iq_desc);
free_ring(sc, nm_rxq->fl_desc_tag, nm_rxq->fl_desc_map, nm_rxq->fl_ba,
nm_rxq->fl_desc);
return (0);
}
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
alloc_nm_txq(struct vi_info *vi, struct sge_nm_txq *nm_txq, int iqidx, int idx,
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
struct sysctl_oid *oid)
{
int rc;
size_t len;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
struct port_info *pi = vi->pi;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
struct adapter *sc = pi->adapter;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
struct netmap_adapter *na = NA(vi->ifp);
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
char name[16];
struct sysctl_oid_list *children = SYSCTL_CHILDREN(oid);
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
len = na->num_tx_desc * EQ_ESIZE + sc->params.sge.spg_len;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
rc = alloc_ring(sc, len, &nm_txq->desc_tag, &nm_txq->desc_map,
&nm_txq->ba, (void **)&nm_txq->desc);
if (rc)
return (rc);
nm_txq->pidx = nm_txq->cidx = 0;
nm_txq->sidx = na->num_tx_desc;
nm_txq->nid = idx;
nm_txq->iqidx = iqidx;
nm_txq->cpl_ctrl0 = htobe32(V_TXPKT_OPCODE(CPL_TX_PKT) |
cxgbe(4): Treat the viid as an opaque identifier. Recent firmwares prefer to use a different format for viid internally and this change allows them to do so. MFC after: 1 week Sponsored by: Chelsio Communications
2019-03-20 17:27:11 +00:00
V_TXPKT_INTF(pi->tx_chan) | V_TXPKT_PF(sc->pf) |
V_TXPKT_VF(vi->vin) | V_TXPKT_VF_VLD(vi->vfvld));
cxgbe(4): Use TX_PKTS2 work requests in netmap Tx if it's available. TX_PKTS2 is more efficient within the firmware and this improves netmap Tx by a few Mpps in some common scenarios. MFC after: 1 week Sponsored by: Chelsio Communications
2019-12-10 08:16:19 +00:00
if (sc->params.fw_vers >= FW_VERSION32(1, 24, 11, 0))
nm_txq->op_pkd = htobe32(V_FW_WR_OP(FW_ETH_TX_PKTS2_WR));
else
nm_txq->op_pkd = htobe32(V_FW_WR_OP(FW_ETH_TX_PKTS_WR));
cxgbe(4): Fix per-queue netmap operation. Do not attempt to initialize netmap queues that are already initialized or aren't supposed to be initialized. Similarly, do not free queues that are not initialized or aren't supposed to be freed. PR: 217156 Sponsored by: Chelsio Communications
2017-06-15 19:56:59 +00:00
nm_txq->cntxt_id = INVALID_NM_TXQ_CNTXT_ID;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
snprintf(name, sizeof(name), "%d", idx);
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
oid = SYSCTL_ADD_NODE(&vi->ctx, children, OID_AUTO, name,
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "netmap tx queue");
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
children = SYSCTL_CHILDREN(oid);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UINT(&vi->ctx, children, OID_AUTO, "cntxt_id", CTLFLAG_RD,
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
&nm_txq->cntxt_id, 0, "SGE context id of the queue");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_PROC(&vi->ctx, children, OID_AUTO, "cidx",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &nm_txq->cidx, 0,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
sysctl_uint16, "I", "consumer index");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_PROC(&vi->ctx, children, OID_AUTO, "pidx",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &nm_txq->pidx, 0,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
sysctl_uint16, "I", "producer index");
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
return (rc);
}
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
free_nm_txq(struct vi_info *vi, struct sge_nm_txq *nm_txq)
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
{
cxgbe(4): Add a pointer to the adapter softc in vi_info. There were quite a few places where port_info was being accessed only to get to the adapter. Reviewed by: jhb@ MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25432
2020-06-25 17:04:22 +00:00
struct adapter *sc = vi->adapter;
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
cxgbe(4): Fix some assertions during driver detach. The netmap queues can't be initialized if the VI isn't. MFC after: 3 days
2017-08-28 03:25:41 +00:00
if (vi->flags & VI_INIT_DONE)
MPASS(nm_txq->cntxt_id == INVALID_NM_TXQ_CNTXT_ID);
else
MPASS(nm_txq->cntxt_id == 0);
cxgbe(4): Fix per-queue netmap operation. Do not attempt to initialize netmap queues that are already initialized or aren't supposed to be initialized. Similarly, do not free queues that are not initialized or aren't supposed to be freed. PR: 217156 Sponsored by: Chelsio Communications
2017-06-15 19:56:59 +00:00
cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards. Netmap gets its own hardware-assisted virtual interface and won't take over or disrupt the "normal" interface in any way. You can use both simultaneously. For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface (note the 'n' prefix) in the hardware to accompany each cxl<N> interface. These two ifnet's per port share the same wire but really are separate interfaces in the hardware and software. Each gets its own L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc. You should run netmap on the 'n' interfaces only, that's what they are for. With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port of a T580 card. 2 port tx is at ~56Mpps total (28M + 28M) as of now. Single port receive is at 33Mpps but this is very much a work in progress. I expect it to be closer to 40Mpps once done. In any case the current effort can already saturate multiple 10G ports of a T5 card at the smallest legal packet size. T4 gear is totally untested. trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef 881.952141 main [1621] interface is ncxl0 881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0 881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0 881.962540 main [1804] mapped 334980KB at 0x801dff000 Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef) 881.962562 main [1882] Sending 512 packets every 0.000000000 s 881.962563 main [1884] Wait 2 secs for phy reset 884.088516 main [1886] Ready... 884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1 884.088607 sender_body [996] start 884.093246 sender_body [1064] drop copy 885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec) 886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec) 887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec) 888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec) 889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec) 890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec) 891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec) 892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec) 893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec) 894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec) 895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec) 896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec) ... Relnotes: Yes Sponsored by: Chelsio Communications.
2014-05-27 18:18:41 +00:00
free_ring(sc, nm_txq->desc_tag, nm_txq->desc_map, nm_txq->ba,
nm_txq->desc);
return (0);
}
#endif
cxgbe(4): Use automatic cidx updates with ofld and ctrl queues. The bits that explicitly request cidx updates do not work reliably with all possible WRs that can be sent over the queue. The F_FW_WR_EQUIQ requests that still remain may also have to be replaced with explicit credit flush WRs in the future. MFC after: 2 days Sponsored by: Chelsio Communications
2018-10-22 23:06:23 +00:00
/*
* Returns a reasonable automatic cidx flush threshold for a given queue size.
*/
static u_int
qsize_to_fthresh(int qsize)
{
u_int fthresh;
while (!powerof2(qsize))
qsize++;
fthresh = ilog2(qsize);
if (fthresh > X_CIDXFLUSHTHRESH_128)
fthresh = X_CIDXFLUSHTHRESH_128;
return (fthresh);
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static int
ctrl_eq_alloc(struct adapter *sc, struct sge_eq *eq)
{
int rc, cntxt_id;
struct fw_eq_ctrl_cmd c;
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
int qsize = eq->sidx + sc->params.sge.spg_len / EQ_ESIZE;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
bzero(&c, sizeof(c));
c.op_to_vfn = htobe32(V_FW_CMD_OP(FW_EQ_CTRL_CMD) | F_FW_CMD_REQUEST |
F_FW_CMD_WRITE | F_FW_CMD_EXEC | V_FW_EQ_CTRL_CMD_PFN(sc->pf) |
V_FW_EQ_CTRL_CMD_VFN(0));
c.alloc_to_len16 = htobe32(F_FW_EQ_CTRL_CMD_ALLOC |
F_FW_EQ_CTRL_CMD_EQSTART | FW_LEN16(c));
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
c.cmpliqid_eqid = htonl(V_FW_EQ_CTRL_CMD_CMPLIQID(eq->iqid));
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
c.physeqid_pkd = htobe32(0);
c.fetchszm_to_iqid =
cxgbe(4): Enable automatic cidx flush for all control queues. MFC after: 3 days
2017-01-09 22:20:09 +00:00
htobe32(V_FW_EQ_CTRL_CMD_HOSTFCMODE(X_HOSTFCMODE_STATUS_PAGE) |
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
V_FW_EQ_CTRL_CMD_PCIECHN(eq->tx_chan) |
F_FW_EQ_CTRL_CMD_FETCHRO | V_FW_EQ_CTRL_CMD_IQID(eq->iqid));
c.dcaen_to_eqsize =
cxgbe(4): Use correct FetchBurstMin values for T6. MFC after: 1 week Sponsored by: Chelsio Communications
2019-10-25 21:53:05 +00:00
htobe32(V_FW_EQ_CTRL_CMD_FBMIN(chip_id(sc) <= CHELSIO_T5 ?
X_FETCHBURSTMIN_64B : X_FETCHBURSTMIN_64B_T6) |
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
V_FW_EQ_CTRL_CMD_FBMAX(X_FETCHBURSTMAX_512B) |
cxgbe(4): Use automatic cidx updates with ofld and ctrl queues. The bits that explicitly request cidx updates do not work reliably with all possible WRs that can be sent over the queue. The F_FW_WR_EQUIQ requests that still remain may also have to be replaced with explicit credit flush WRs in the future. MFC after: 2 days Sponsored by: Chelsio Communications
2018-10-22 23:06:23 +00:00
V_FW_EQ_CTRL_CMD_CIDXFTHRESH(qsize_to_fthresh(qsize)) |
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
V_FW_EQ_CTRL_CMD_EQSIZE(qsize));
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
c.eqaddr = htobe64(eq->ba);
rc = -t4_wr_mbox(sc, sc->mbox, &c, sizeof(c), &c);
if (rc != 0) {
device_printf(sc->dev,
"failed to create control queue %d: %d\n", eq->tx_chan, rc);
return (rc);
}
eq->flags |= EQ_ALLOCATED;
eq->cntxt_id = G_FW_EQ_CTRL_CMD_EQID(be32toh(c.cmpliqid_eqid));
cntxt_id = eq->cntxt_id - sc->sge.eq_start;
if (cntxt_id >= sc->sge.neq)
panic("%s: eq->cntxt_id (%d) more than the max (%d)", __func__,
cntxt_id, sc->sge.neq - 1);
sc->sge.eqmap[cntxt_id] = eq;
return (rc);
}
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
eth_eq_alloc(struct adapter *sc, struct vi_info *vi, struct sge_eq *eq)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
{
int rc, cntxt_id;
struct fw_eq_eth_cmd c;
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
int qsize = eq->sidx + sc->params.sge.spg_len / EQ_ESIZE;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
bzero(&c, sizeof(c));
c.op_to_vfn = htobe32(V_FW_CMD_OP(FW_EQ_ETH_CMD) | F_FW_CMD_REQUEST |
F_FW_CMD_WRITE | F_FW_CMD_EXEC | V_FW_EQ_ETH_CMD_PFN(sc->pf) |
V_FW_EQ_ETH_CMD_VFN(0));
c.alloc_to_len16 = htobe32(F_FW_EQ_ETH_CMD_ALLOC |
F_FW_EQ_ETH_CMD_EQSTART | FW_LEN16(c));
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
c.autoequiqe_to_viid = htobe32(F_FW_EQ_ETH_CMD_AUTOEQUIQE |
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
F_FW_EQ_ETH_CMD_AUTOEQUEQE | V_FW_EQ_ETH_CMD_VIID(vi->viid));
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
c.fetchszm_to_iqid =
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
htobe32(V_FW_EQ_ETH_CMD_HOSTFCMODE(X_HOSTFCMODE_NONE) |
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
V_FW_EQ_ETH_CMD_PCIECHN(eq->tx_chan) | F_FW_EQ_ETH_CMD_FETCHRO |
V_FW_EQ_ETH_CMD_IQID(eq->iqid));
cxgbe(4): Use correct FetchBurstMin values for T6. MFC after: 1 week Sponsored by: Chelsio Communications
2019-10-25 21:53:05 +00:00
c.dcaen_to_eqsize =
htobe32(V_FW_EQ_ETH_CMD_FBMIN(chip_id(sc) <= CHELSIO_T5 ?
X_FETCHBURSTMIN_64B : X_FETCHBURSTMIN_64B_T6) |
V_FW_EQ_ETH_CMD_FBMAX(X_FETCHBURSTMAX_512B) |
V_FW_EQ_ETH_CMD_EQSIZE(qsize));
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
c.eqaddr = htobe64(eq->ba);
rc = -t4_wr_mbox(sc, sc->mbox, &c, sizeof(c), &c);
if (rc != 0) {
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
device_printf(vi->dev,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
"failed to create Ethernet egress queue: %d\n", rc);
return (rc);
}
eq->flags |= EQ_ALLOCATED;
eq->cntxt_id = G_FW_EQ_ETH_CMD_EQID(be32toh(c.eqid_pkd));
Track the base absolute ID of ingress and egress queues. Use this to map an absolute queue ID to a logical queue ID in interrupt handlers. For the regular cxgbe/cxl drivers this should be a no-op as the base absolute ID should be zero. VF devices have a non-zero base absolute ID and require this change. While here, export the absolute ID of egress queues via a sysctl. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7446
2016-08-09 17:49:42 +00:00
eq->abs_id = G_FW_EQ_ETH_CMD_PHYSEQID(be32toh(c.physeqid_pkd));
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cntxt_id = eq->cntxt_id - sc->sge.eq_start;
if (cntxt_id >= sc->sge.neq)
panic("%s: eq->cntxt_id (%d) more than the max (%d)", __func__,
cntxt_id, sc->sge.neq - 1);
sc->sge.eqmap[cntxt_id] = eq;
return (rc);
}
cxgbe(4): Allocate offload Tx queues when a card has resources provisioned for NIC_ETHOFLD and the kernel has option RATELIMIT. It is possible to use the chip's offload queues for normal NIC Tx and not just TOE Tx. The difference is that these queues support out of order processing of work requests and have a per-"flowid" mechanism for tracking credits between the driver and hardware. This allows Tx for any number of flows bound to different rate limits to be submitted to a single Tx queue and the work requests for slow flows won't cause HOL blocking for the rest. Sponsored by: Chelsio Communications
2018-05-17 01:42:18 +00:00
#if defined(TCP_OFFLOAD) || defined(RATELIMIT)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
ofld_eq_alloc(struct adapter *sc, struct vi_info *vi, struct sge_eq *eq)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
{
int rc, cntxt_id;
struct fw_eq_ofld_cmd c;
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
int qsize = eq->sidx + sc->params.sge.spg_len / EQ_ESIZE;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
bzero(&c, sizeof(c));
c.op_to_vfn = htonl(V_FW_CMD_OP(FW_EQ_OFLD_CMD) | F_FW_CMD_REQUEST |
F_FW_CMD_WRITE | F_FW_CMD_EXEC | V_FW_EQ_OFLD_CMD_PFN(sc->pf) |
V_FW_EQ_OFLD_CMD_VFN(0));
c.alloc_to_len16 = htonl(F_FW_EQ_OFLD_CMD_ALLOC |
F_FW_EQ_OFLD_CMD_EQSTART | FW_LEN16(c));
c.fetchszm_to_iqid =
cxgbe(4): Use automatic cidx updates with ofld and ctrl queues. The bits that explicitly request cidx updates do not work reliably with all possible WRs that can be sent over the queue. The F_FW_WR_EQUIQ requests that still remain may also have to be replaced with explicit credit flush WRs in the future. MFC after: 2 days Sponsored by: Chelsio Communications
2018-10-22 23:06:23 +00:00
htonl(V_FW_EQ_OFLD_CMD_HOSTFCMODE(X_HOSTFCMODE_STATUS_PAGE) |
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
V_FW_EQ_OFLD_CMD_PCIECHN(eq->tx_chan) |
F_FW_EQ_OFLD_CMD_FETCHRO | V_FW_EQ_OFLD_CMD_IQID(eq->iqid));
c.dcaen_to_eqsize =
cxgbe(4): Use correct FetchBurstMin values for T6. MFC after: 1 week Sponsored by: Chelsio Communications
2019-10-25 21:53:05 +00:00
htobe32(V_FW_EQ_OFLD_CMD_FBMIN(chip_id(sc) <= CHELSIO_T5 ?
X_FETCHBURSTMIN_64B : X_FETCHBURSTMIN_64B_T6) |
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
V_FW_EQ_OFLD_CMD_FBMAX(X_FETCHBURSTMAX_512B) |
cxgbe(4): Use automatic cidx updates with ofld and ctrl queues. The bits that explicitly request cidx updates do not work reliably with all possible WRs that can be sent over the queue. The F_FW_WR_EQUIQ requests that still remain may also have to be replaced with explicit credit flush WRs in the future. MFC after: 2 days Sponsored by: Chelsio Communications
2018-10-22 23:06:23 +00:00
V_FW_EQ_OFLD_CMD_CIDXFTHRESH(qsize_to_fthresh(qsize)) |
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
V_FW_EQ_OFLD_CMD_EQSIZE(qsize));
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
c.eqaddr = htobe64(eq->ba);
rc = -t4_wr_mbox(sc, sc->mbox, &c, sizeof(c), &c);
if (rc != 0) {
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
device_printf(vi->dev,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
"failed to create egress queue for TCP offload: %d\n", rc);
return (rc);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
eq->flags |= EQ_ALLOCATED;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
eq->cntxt_id = G_FW_EQ_OFLD_CMD_EQID(be32toh(c.eqid_pkd));
cntxt_id = eq->cntxt_id - sc->sge.eq_start;
if (cntxt_id >= sc->sge.neq)
panic("%s: eq->cntxt_id (%d) more than the max (%d)", __func__,
cntxt_id, sc->sge.neq - 1);
sc->sge.eqmap[cntxt_id] = eq;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
return (rc);
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
#endif
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
alloc_eq(struct adapter *sc, struct vi_info *vi, struct sge_eq *eq)
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
{
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
int rc, qsize;
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
size_t len;
mtx_init(&eq->eq_lock, eq->lockname, NULL, MTX_DEF);
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
qsize = eq->sidx + sc->params.sge.spg_len / EQ_ESIZE;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
len = qsize * EQ_ESIZE;
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
rc = alloc_ring(sc, len, &eq->desc_tag, &eq->desc_map,
&eq->ba, (void **)&eq->desc);
if (rc)
return (rc);
cxgbe(4): Use automatic cidx updates with ofld and ctrl queues. The bits that explicitly request cidx updates do not work reliably with all possible WRs that can be sent over the queue. The F_FW_WR_EQUIQ requests that still remain may also have to be replaced with explicit credit flush WRs in the future. MFC after: 2 days Sponsored by: Chelsio Communications
2018-10-22 23:06:23 +00:00
eq->pidx = eq->cidx = eq->dbidx = 0;
/* Note that equeqidx is not used with sge_wrq (OFLD/CTRL) queues. */
eq->equeqidx = 0;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
eq->doorbells = sc->doorbells;
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
switch (eq->flags & EQ_TYPEMASK) {
case EQ_CTRL:
rc = ctrl_eq_alloc(sc, eq);
break;
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
case EQ_ETH:
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
rc = eth_eq_alloc(sc, vi, eq);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
break;
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
cxgbe(4): Allocate offload Tx queues when a card has resources provisioned for NIC_ETHOFLD and the kernel has option RATELIMIT. It is possible to use the chip's offload queues for normal NIC Tx and not just TOE Tx. The difference is that these queues support out of order processing of work requests and have a per-"flowid" mechanism for tracking credits between the driver and hardware. This allows Tx for any number of flows bound to different rate limits to be submitted to a single Tx queue and the work requests for slow flows won't cause HOL blocking for the rest. Sponsored by: Chelsio Communications
2018-05-17 01:42:18 +00:00
#if defined(TCP_OFFLOAD) || defined(RATELIMIT)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
case EQ_OFLD:
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
rc = ofld_eq_alloc(sc, vi, eq);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
break;
#endif
default:
panic("%s: invalid eq type %d.", __func__,
eq->flags & EQ_TYPEMASK);
}
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
if (rc != 0) {
device_printf(sc->dev,
Add missing newline to an error message. MFC after: 3 days
2014-07-22 19:48:21 +00:00
"failed to allocate egress queue(%d): %d\n",
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
eq->flags & EQ_TYPEMASK, rc);
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
}
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
if (isset(&eq->doorbells, DOORBELL_UDB) ||
isset(&eq->doorbells, DOORBELL_UDBWC) ||
Cosmetic change (s/wrwc/wcwr/;s/WRWC/WCWR/). MFC after: 3 days.
2013-04-11 22:49:29 +00:00
isset(&eq->doorbells, DOORBELL_WCWR)) {
cxgbe(4): Add a struct sge_params to store per-adapter SGE parameters. Move the code that reads all the parameters to t4_init_sge_params in the shared code. Use these per-adapter values instead of globals. Sponsored by: Chelsio Communications
2016-03-08 00:23:56 +00:00
uint32_t s_qpp = sc->params.sge.eq_s_qpp;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
uint32_t mask = (1 << s_qpp) - 1;
volatile uint8_t *udb;
udb = sc->udbs_base + UDBS_DB_OFFSET;
udb += (eq->cntxt_id >> s_qpp) << PAGE_SHIFT; /* pg offset */
eq->udb_qid = eq->cntxt_id & mask; /* id in page */
cxgbe(4): Fix an off by one error when looking for the BAR2 doorbell address of an egress queue. MFC after: 2 weeks
2014-08-02 01:48:25 +00:00
if (eq->udb_qid >= PAGE_SIZE / UDBS_SEG_SIZE)
Cosmetic change (s/wrwc/wcwr/;s/WRWC/WCWR/). MFC after: 3 days.
2013-04-11 22:49:29 +00:00
clrbit(&eq->doorbells, DOORBELL_WCWR);
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
else {
udb += eq->udb_qid << UDBS_SEG_SHIFT; /* seg offset */
eq->udb_qid = 0;
}
eq->udb = (volatile void *)udb;
}
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
return (rc);
}
static int
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
free_eq(struct adapter *sc, struct sge_eq *eq)
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
{
int rc;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (eq->flags & EQ_ALLOCATED) {
switch (eq->flags & EQ_TYPEMASK) {
case EQ_CTRL:
rc = -t4_ctrl_eq_free(sc, sc->mbox, sc->pf, 0,
eq->cntxt_id);
break;
case EQ_ETH:
rc = -t4_eth_eq_free(sc, sc->mbox, sc->pf, 0,
eq->cntxt_id);
break;
cxgbe(4): Allocate offload Tx queues when a card has resources provisioned for NIC_ETHOFLD and the kernel has option RATELIMIT. It is possible to use the chip's offload queues for normal NIC Tx and not just TOE Tx. The difference is that these queues support out of order processing of work requests and have a per-"flowid" mechanism for tracking credits between the driver and hardware. This allows Tx for any number of flows bound to different rate limits to be submitted to a single Tx queue and the work requests for slow flows won't cause HOL blocking for the rest. Sponsored by: Chelsio Communications
2018-05-17 01:42:18 +00:00
#if defined(TCP_OFFLOAD) || defined(RATELIMIT)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
case EQ_OFLD:
rc = -t4_ofld_eq_free(sc, sc->mbox, sc->pf, 0,
eq->cntxt_id);
break;
#endif
default:
panic("%s: invalid eq type %d.", __func__,
eq->flags & EQ_TYPEMASK);
}
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
if (rc != 0) {
device_printf(sc->dev,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
"failed to free egress queue (%d): %d\n",
eq->flags & EQ_TYPEMASK, rc);
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
return (rc);
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
eq->flags &= ~EQ_ALLOCATED;
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
}
free_ring(sc, eq->desc_tag, eq->desc_map, eq->ba, eq->desc);
if (mtx_initialized(&eq->eq_lock))
mtx_destroy(&eq->eq_lock);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
bzero(eq, sizeof(*eq));
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
return (0);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
alloc_wrq(struct adapter *sc, struct vi_info *vi, struct sge_wrq *wrq,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sysctl_oid *oid)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
int rc;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
struct sysctl_ctx_list *ctx = vi ? &vi->ctx : &sc->ctx;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sysctl_oid_list *children = SYSCTL_CHILDREN(oid);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
rc = alloc_eq(sc, vi, &wrq->eq);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
if (rc)
return (rc);
wrq->adapter = sc;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
TASK_INIT(&wrq->wrq_tx_task, 0, wrq_tx_drain, wrq);
TAILQ_INIT(&wrq->incomplete_wrs);
- Updated TOE support in the kernel. - Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs. These are available as t3_tom and t4_tom modules that augment cxgb(4) and cxgbe(4) respectively. The cxgb/cxgbe drivers continue to work as usual with or without these extra features. - iWARP driver for Terminator 3 ASIC (kernel verbs). T4 iWARP in the works and will follow soon. Build-tested with make universe. 30s overview ============ What interfaces support TCP offload? Look for TOE4 and/or TOE6 in the capabilities of an interface: # ifconfig -m | grep TOE Enable/disable TCP offload on an interface (just like any other ifnet capability): # ifconfig cxgbe0 toe # ifconfig cxgbe0 -toe Which connections are offloaded? Look for toe4 and/or toe6 in the output of netstat and sockstat: # netstat -np tcp | grep toe # sockstat -46c | grep toe Reviewed by: bz, gnn Sponsored by: Chelsio communications. MFC after: ~3 months (after 9.1, and after ensuring MFC is feasible)
2012-06-19 07:34:13 +00:00
STAILQ_INIT(&wrq->wr_list);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
wrq->nwr_pending = 0;
wrq->ndesc_needed = 0;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): Make the location/length of all descriptor rings available in the sysctl MIB.
2016-09-23 20:03:28 +00:00
SYSCTL_ADD_UAUTO(ctx, children, OID_AUTO, "ba", CTLFLAG_RD,
&wrq->eq.ba, "bus address of descriptor ring");
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "dmalen", CTLFLAG_RD, NULL,
wrq->eq.sidx * EQ_ESIZE + sc->params.sge.spg_len,
"desc ring size in bytes");
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
SYSCTL_ADD_UINT(ctx, children, OID_AUTO, "cntxt_id", CTLFLAG_RD,
&wrq->eq.cntxt_id, 0, "SGE context id of the queue");
SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "cidx",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &wrq->eq.cidx, 0,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
sysctl_uint16, "I", "consumer index");
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "pidx",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &wrq->eq.pidx, 0,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
sysctl_uint16, "I", "producer index");
cxgbe(4): Make the location/length of all descriptor rings available in the sysctl MIB.
2016-09-23 20:03:28 +00:00
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "sidx", CTLFLAG_RD, NULL,
wrq->eq.sidx, "status page index");
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
SYSCTL_ADD_UQUAD(ctx, children, OID_AUTO, "tx_wrs_direct", CTLFLAG_RD,
&wrq->tx_wrs_direct, "# of work requests (direct)");
SYSCTL_ADD_UQUAD(ctx, children, OID_AUTO, "tx_wrs_copied", CTLFLAG_RD,
&wrq->tx_wrs_copied, "# of work requests (copied)");
cxgbe(4): Fixes to wrq stats. - Increment tx_wrs_copied in the correct place. - Add tx_wrs_sspace to the sysctl MIB. Sponsored by: Chelsio Communications
2016-09-19 17:16:51 +00:00
SYSCTL_ADD_UQUAD(ctx, children, OID_AUTO, "tx_wrs_sspace", CTLFLAG_RD,
&wrq->tx_wrs_ss, "# of work requests (copied from scratch space)");
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
return (rc);
}
static int
free_wrq(struct adapter *sc, struct sge_wrq *wrq)
{
int rc;
rc = free_eq(sc, &wrq->eq);
if (rc)
return (rc);
bzero(wrq, sizeof(*wrq));
return (0);
}
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
alloc_txq(struct vi_info *vi, struct sge_txq *txq, int idx,
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sysctl_oid *oid)
{
int rc;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
struct port_info *pi = vi->pi;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct adapter *sc = pi->adapter;
struct sge_eq *eq = &txq->eq;
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
struct txpkts *txp;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
char name[16];
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
struct sysctl_oid_list *children = SYSCTL_CHILDREN(oid);
Resume tx immediately in response to an SGE egress update from the hardware. MFC after: 1 week
2011-03-05 03:06:38 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
rc = mp_ring_alloc(&txq->r, eq->sidx, txq, eth_tx, can_resume_eth_tx,
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
M_CXGBE, &eq->eq_lock, M_WAITOK);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
if (rc != 0) {
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
device_printf(sc->dev, "failed to allocate mp_ring: %d\n", rc);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
return (rc);
}
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
rc = alloc_eq(sc, vi, eq);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
if (rc != 0) {
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
mp_ring_free(txq->r);
txq->r = NULL;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
return (rc);
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/* Can't fail after this point. */
Track the base absolute ID of ingress and egress queues. Use this to map an absolute queue ID to a logical queue ID in interrupt handlers. For the regular cxgbe/cxl drivers this should be a no-op as the base absolute ID should be zero. VF devices have a non-zero base absolute ID and require this change. While here, export the absolute ID of egress queues via a sysctl. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7446
2016-08-09 17:49:42 +00:00
if (idx == 0)
sc->sge.eq_base = eq->abs_id - eq->cntxt_id;
else
KASSERT(eq->cntxt_id + sc->sge.eq_base == eq->abs_id,
("eq_base mismatch"));
KASSERT(sc->sge.eq_base == 0 || sc->flags & IS_VF,
("PF with non-zero eq_base"));
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
TASK_INIT(&txq->tx_reclaim_task, 0, tx_reclaim, eq);
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
txq->ifp = vi->ifp;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
txq->gl = sglist_alloc(TX_SGL_SEGS, M_WAITOK);
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
if (vi->flags & TX_USES_VM_WR)
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
txq->cpl_ctrl0 = htobe32(V_TXPKT_OPCODE(CPL_TX_PKT_XT) |
V_TXPKT_INTF(pi->tx_chan));
else
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
txq->cpl_ctrl0 = htobe32(V_TXPKT_OPCODE(CPL_TX_PKT_XT) |
cxgbe(4): Treat the viid as an opaque identifier. Recent firmwares prefer to use a different format for viid internally and this change allows them to do so. MFC after: 1 week Sponsored by: Chelsio Communications
2019-03-20 17:27:11 +00:00
V_TXPKT_INTF(pi->tx_chan) | V_TXPKT_PF(sc->pf) |
V_TXPKT_VF(vi->vin) | V_TXPKT_VF_VLD(vi->vfvld));
cxgbe(4): Add a sysctl to manage the binding of a txq to a traffic class. Sponsored by: Chelsio Communications
2016-06-08 14:15:29 +00:00
txq->tc_idx = -1;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
txq->sdesc = malloc(eq->sidx * sizeof(struct tx_sdesc), M_CXGBE,
M_ZERO | M_WAITOK);
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
txp = &txq->txp;
txp->score = 5;
MPASS(nitems(txp->mb) >= sc->params.max_pkts_per_eth_tx_pkts_wr);
txq->txp.max_npkt = min(nitems(txp->mb),
sc->params.max_pkts_per_eth_tx_pkts_wr);
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
if (vi->flags & TX_USES_VM_WR && !(sc->flags & IS_VF))
txq->txp.max_npkt--;
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
snprintf(name, sizeof(name), "%d", idx);
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
oid = SYSCTL_ADD_NODE(&vi->ctx, children, OID_AUTO, name,
CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "tx queue");
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
children = SYSCTL_CHILDREN(oid);
cxgbe(4): Make the location/length of all descriptor rings available in the sysctl MIB.
2016-09-23 20:03:28 +00:00
SYSCTL_ADD_UAUTO(&vi->ctx, children, OID_AUTO, "ba", CTLFLAG_RD,
&eq->ba, "bus address of descriptor ring");
SYSCTL_ADD_INT(&vi->ctx, children, OID_AUTO, "dmalen", CTLFLAG_RD, NULL,
eq->sidx * EQ_ESIZE + sc->params.sge.spg_len,
"desc ring size in bytes");
Track the base absolute ID of ingress and egress queues. Use this to map an absolute queue ID to a logical queue ID in interrupt handlers. For the regular cxgbe/cxl drivers this should be a no-op as the base absolute ID should be zero. VF devices have a non-zero base absolute ID and require this change. While here, export the absolute ID of egress queues via a sysctl. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7446
2016-08-09 17:49:42 +00:00
SYSCTL_ADD_UINT(&vi->ctx, children, OID_AUTO, "abs_id", CTLFLAG_RD,
&eq->abs_id, 0, "absolute id of the queue");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UINT(&vi->ctx, children, OID_AUTO, "cntxt_id", CTLFLAG_RD,
- driver ioctl to get SGE context for any given queue. - sysctls to display the context id, cidx, and pidx of all kinds of queues. MFC after: 3 days
2011-06-11 04:50:54 +00:00
&eq->cntxt_id, 0, "SGE context id of the queue");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_PROC(&vi->ctx, children, OID_AUTO, "cidx",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &eq->cidx, 0,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
sysctl_uint16, "I", "consumer index");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_PROC(&vi->ctx, children, OID_AUTO, "pidx",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RD | CTLFLAG_MPSAFE, &eq->pidx, 0,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
sysctl_uint16, "I", "producer index");
cxgbe(4): Make the location/length of all descriptor rings available in the sysctl MIB.
2016-09-23 20:03:28 +00:00
SYSCTL_ADD_INT(&vi->ctx, children, OID_AUTO, "sidx", CTLFLAG_RD, NULL,
eq->sidx, "status page index");
- driver ioctl to get SGE context for any given queue. - sysctls to display the context id, cidx, and pidx of all kinds of queues. MFC after: 3 days
2011-06-11 04:50:54 +00:00
cxgbe(4): Add a sysctl to manage the binding of a txq to a traffic class. Sponsored by: Chelsio Communications
2016-06-08 14:15:29 +00:00
SYSCTL_ADD_PROC(&vi->ctx, children, OID_AUTO, "tc",
cxgbe(4) sysctls do not need Giant. Sponsored by: Chelsio Communications
2020-10-05 22:18:04 +00:00
CTLTYPE_INT | CTLFLAG_RW | CTLFLAG_MPSAFE, vi, idx, sysctl_tc,
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
"I", "traffic class (-1 means none)");
cxgbe(4): Add a sysctl to manage the binding of a txq to a traffic class. Sponsored by: Chelsio Communications
2016-06-08 14:15:29 +00:00
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "txcsum", CTLFLAG_RD,
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
&txq->txcsum, "# of times hardware assisted with checksum");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "vlan_insertion",
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
CTLFLAG_RD, &txq->vlan_insertion,
"# of times hardware inserted 802.1Q tag");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "tso_wrs", CTLFLAG_RD,
cxgbe(4): support for IPv6 TSO and LRO. Submitted by: bz (this is a modified version of that patch)
2012-06-29 19:51:06 +00:00
&txq->tso_wrs, "# of TSO work requests");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "imm_wrs", CTLFLAG_RD,
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
&txq->imm_wrs, "# of work requests with immediate data");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "sgl_wrs", CTLFLAG_RD,
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
&txq->sgl_wrs, "# of work requests with direct SGL");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "txpkt_wrs", CTLFLAG_RD,
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
&txq->txpkt_wrs, "# of txpkt work requests (one pkt/WR)");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "txpkts0_wrs",
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
CTLFLAG_RD, &txq->txpkts0_wrs,
"# of txpkts (type 0) work requests");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "txpkts1_wrs",
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
CTLFLAG_RD, &txq->txpkts1_wrs,
"# of txpkts (type 1) work requests");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "txpkts0_pkts",
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
CTLFLAG_RD, &txq->txpkts0_pkts,
"# of frames tx'd using type0 txpkts work requests");
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "txpkts1_pkts",
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
CTLFLAG_RD, &txq->txpkts1_pkts,
"# of frames tx'd using type1 txpkts work requests");
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "raw_wrs", CTLFLAG_RD,
&txq->raw_wrs, "# of raw work requests (non-packets)");
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "vxlan_tso_wrs",
CTLFLAG_RD, &txq->vxlan_tso_wrs, "# of VXLAN TSO work requests");
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO, "vxlan_txcsum",
CTLFLAG_RD, &txq->vxlan_txcsum,
"# of times hardware assisted with inner checksums (VXLAN)");
NIC KTLS for Chelsio T6 adapters. This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters. Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE connections. NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment the encrypted TLS frames output by the crypto engine. Instead, the TOE is placed into a special setup to permit "dummy" connections to be associated with regular sockets using KTLS. This permits using the TOE to segment the encrypted TLS records. However, this approach does have some limitations: 1) Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time. 2) In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset. 3) Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls. Unlike TOE TLS which is able to buffer encrypted TLS records in on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS records for retransmit requests as well as non-retransmit requests that do not include the start of a TLS record but do include the trailer. The T6 NIC KTLS code tries to optimize some of the cases for requests to transmit partial TLS records. In particular it attempts to minimize sending "waste" bytes that have to be given as input to the crypto engine but are not needed on the wire to satisfy mbufs sent from the TCP stack down to the driver. TCP packets for TLS requests are broken down into the following classes (with associated counters): - Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full). - Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes. - Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes. In addition, other per-txq sysctls are provided: - dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC. - dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM. - dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set. - dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records. - dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests. - dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests. To enable NIC KTLS with T6, set the following tunables prior to loading the cxgbe(4) driver: hw.cxgbe.config_file=kern_tls hw.cxgbe.kern_tls=1 Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
#ifdef KERN_TLS
if (sc->flags & KERN_TLS_OK) {
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO,
"kern_tls_records", CTLFLAG_RD, &txq->kern_tls_records,
"# of NIC TLS records transmitted");
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO,
"kern_tls_short", CTLFLAG_RD, &txq->kern_tls_short,
"# of short NIC TLS records transmitted");
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO,
"kern_tls_partial", CTLFLAG_RD, &txq->kern_tls_partial,
"# of partial NIC TLS records transmitted");
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO,
"kern_tls_full", CTLFLAG_RD, &txq->kern_tls_full,
"# of full NIC TLS records transmitted");
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO,
"kern_tls_octets", CTLFLAG_RD, &txq->kern_tls_octets,
"# of payload octets in transmitted NIC TLS records");
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO,
"kern_tls_waste", CTLFLAG_RD, &txq->kern_tls_waste,
"# of octets DMAd but not transmitted in NIC TLS records");
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO,
"kern_tls_options", CTLFLAG_RD, &txq->kern_tls_options,
"# of NIC TLS options-only packets transmitted");
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO,
"kern_tls_header", CTLFLAG_RD, &txq->kern_tls_header,
"# of NIC TLS header-only packets transmitted");
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO,
"kern_tls_fin", CTLFLAG_RD, &txq->kern_tls_fin,
"# of NIC TLS FIN-only packets transmitted");
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO,
"kern_tls_fin_short", CTLFLAG_RD, &txq->kern_tls_fin_short,
"# of NIC TLS padded FIN packets on short TLS records");
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO,
"kern_tls_cbc", CTLFLAG_RD, &txq->kern_tls_cbc,
"# of NIC TLS sessions using AES-CBC");
SYSCTL_ADD_UQUAD(&vi->ctx, children, OID_AUTO,
"kern_tls_gcm", CTLFLAG_RD, &txq->kern_tls_gcm,
"# of NIC TLS sessions using AES-GCM");
}
#endif
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
mp_ring_sysctls(txq->r, &vi->ctx, children);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (0);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
static int
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
free_txq(struct vi_info *vi, struct sge_txq *txq)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
int rc;
cxgbe(4): Add a pointer to the adapter softc in vi_info. There were quite a few places where port_info was being accessed only to get to the adapter. Reviewed by: jhb@ MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25432
2020-06-25 17:04:22 +00:00
struct adapter *sc = vi->adapter;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct sge_eq *eq = &txq->eq;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
rc = free_eq(sc, eq);
if (rc)
return (rc);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
sglist_free(txq->gl);
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
free(txq->sdesc, M_CXGBE);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
mp_ring_free(txq->r);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
bzero(txq, sizeof(*txq));
return (0);
}
static void
oneseg_dma_callback(void *arg, bus_dma_segment_t *segs, int nseg, int error)
{
bus_addr_t *ba = arg;
KASSERT(nseg == 1,
("%s meant for single segment mappings only.", __func__));
*ba = error ? 0 : segs->ds_addr;
}
static inline void
ring_fl_db(struct adapter *sc, struct sge_fl *fl)
{
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
uint32_t n, v;
Ring the freelist doorbell from within refill_fl. While here, fix a bug that could have allowed the hardware pidx to reach the cidx even though the freelist isn't empty. (Haven't actually seen this but it was there waiting to happen..) MFC after: 1 week
2011-04-20 23:20:00 +00:00
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
n = IDXDIFF(fl->pidx >> 3, fl->dbidx, fl->sidx);
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
MPASS(n > 0);
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
wmb();
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
v = fl->dbval | V_PIDX(n);
if (fl->udb)
*fl->udb = htole32(v);
else
Store the offset of the KDOORBELL and GTS registers in the softc. VF devices use a different register layout than PF devices. Storing the offset in a value in the softc allows code to be shared between the PF and VF drivers. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7389
2016-08-01 22:39:51 +00:00
t4_write_reg(sc, sc->sge_kdoorbell_reg, v);
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
IDXINCR(fl->dbidx, n, fl->sidx);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
Ring the freelist doorbell from within refill_fl. While here, fix a bug that could have allowed the hardware pidx to reach the cidx even though the freelist isn't empty. (Haven't actually seen this but it was there waiting to happen..) MFC after: 1 week
2011-04-20 23:20:00 +00:00
/*
sys/dev: minor spelling fixes. Most affect comments, very few have user-visible effects.
2016-05-03 03:41:25 +00:00
* Fills up the freelist by allocating up to 'n' buffers. Buffers that are
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
* recycled do not count towards this allocation budget.
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
*
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
* Returns non-zero to indicate that this freelist should be added to the list
* of starving freelists.
Ring the freelist doorbell from within refill_fl. While here, fix a bug that could have allowed the hardware pidx to reach the cidx even though the freelist isn't empty. (Haven't actually seen this but it was there waiting to happen..) MFC after: 1 week
2011-04-20 23:20:00 +00:00
*/
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static int
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
refill_fl(struct adapter *sc, struct sge_fl *fl, int n)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
__be64 *d;
struct fl_sdesc *sd;
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
uintptr_t pa;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
caddr_t cl;
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
struct rx_buf_info *rxb;
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
struct cluster_metadata *clm;
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
uint16_t max_pidx;
uint16_t hw_cidx = fl->hw_cidx; /* stable snapshot */
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
FL_LOCK_ASSERT_OWNED(fl);
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
/*
sys/dev: minor spelling fixes. Most affect comments, very few have user-visible effects.
2016-05-03 03:41:25 +00:00
* We always stop at the beginning of the hardware descriptor that's just
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
* before the one with the hw cidx. This is to avoid hw pidx = hw cidx,
* which would mean an empty freelist to the chip.
*/
max_pidx = __predict_false(hw_cidx == 0) ? fl->sidx - 1 : hw_cidx - 1;
if (fl->pidx == max_pidx * 8)
return (0);
d = &fl->desc[fl->pidx];
sd = &fl->sdesc[fl->pidx];
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
while (n > 0) {
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
if (sd->cl != NULL) {
Simplify r267600, there's no need to distinguish between allocated and inlined mbufs. MFC after: 1 week
2014-07-22 02:02:39 +00:00
if (sd->nmbuf == 0) {
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
/*
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
* Fast recycle without involving any atomics on
* the cluster's metadata (if the cluster has
* metadata). This happens when all frames
* received in the cluster were small enough to
* fit within a single mbuf each.
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
*/
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
fl->cl_fast_recycled++;
cxgbe(4): Initialize the rx buffer's metadata on first-use and not on allocation. refill_fl doesn't touch any part of a freshly allocated cluster after this change. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-03 23:25:12 +00:00
goto recycled;
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
/*
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
* Cluster is guaranteed to have metadata. Clusters
* without metadata always take the fast recycle path
* when they're recycled.
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
*/
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
clm = cl_metadata(sd);
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
MPASS(clm != NULL);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
if (atomic_fetchadd_int(&clm->refcount, -1) == 1) {
fl->cl_recycled++;
cxgbe(4): Keep track of the clusters that have to be freed by the custom free routine (rxb_free) in the driver. Fail MOD_UNLOAD with EBUSY if any such cluster has been handed up to the kernel but hasn't been freed yet. This prevents a panic later when the cluster finally needs to be freed but rxb_free is gone from the kernel. MFC after: 1 week
2014-07-23 22:29:22 +00:00
counter_u64_add(extfree_rels, 1);
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
goto recycled;
}
sd->cl = NULL; /* gave up my reference */
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
MPASS(sd->cl == NULL);
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
rxb = &sc->sge.rx_buf_info[fl->zidx];
cl = uma_zalloc(rxb->zone, M_NOWAIT);
cxgbe(4): Do not try to use 0 as an rx buffer address when the driver is already allocating from the safe zone and the allocation fails. This bug was introduced in r357481. MFC after: 3 days Sponsored by: Chelsio Communications
2020-03-10 21:44:20 +00:00
if (__predict_false(cl == NULL)) {
if (fl->zidx != fl->safe_zidx) {
rxb = &sc->sge.rx_buf_info[fl->safe_zidx];
cl = uma_zalloc(rxb->zone, M_NOWAIT);
}
if (cl == NULL)
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
break;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
fl->cl_allocated++;
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
n--;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
pa = pmap_kextract((vm_offset_t)cl);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
sd->cl = cl;
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
sd->zidx = fl->zidx;
if (fl->flags & FL_BUF_PACKING) {
*d = htobe64(pa | rxb->hwidx2);
sd->moff = rxb->size2;
} else {
*d = htobe64(pa | rxb->hwidx1);
sd->moff = 0;
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
recycled:
Simplify r267600, there's no need to distinguish between allocated and inlined mbufs. MFC after: 1 week
2014-07-22 02:02:39 +00:00
sd->nmbuf = 0;
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
d++;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
sd++;
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if (__predict_false((++fl->pidx & 7) == 0)) {
uint16_t pidx = fl->pidx >> 3;
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
if (__predict_false(pidx == fl->sidx)) {
fl->pidx = 0;
pidx = 0;
sd = fl->sdesc;
d = fl->desc;
}
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if (n < 8 || pidx == max_pidx)
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
break;
if (IDXDIFF(pidx, fl->dbidx, fl->sidx) >= 4)
ring_fl_db(sc, fl);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
}
Ring the freelist doorbell from within refill_fl. While here, fix a bug that could have allowed the hardware pidx to reach the cidx even though the freelist isn't empty. (Haven't actually seen this but it was there waiting to happen..) MFC after: 1 week
2011-04-20 23:20:00 +00:00
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if ((fl->pidx >> 3) != fl->dbidx)
Ring the freelist doorbell from within refill_fl. While here, fix a bug that could have allowed the hardware pidx to reach the cidx even though the freelist isn't empty. (Haven't actually seen this but it was there waiting to happen..) MFC after: 1 week
2011-04-20 23:20:00 +00:00
ring_fl_db(sc, fl);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
return (FL_RUNNING_LOW(fl) && !(fl->flags & FL_STARVING));
}
/*
* Attempt to refill all starving freelists.
*/
static void
refill_sfl(void *arg)
{
struct adapter *sc = arg;
struct sge_fl *fl, *fl_temp;
Add support for configuring additional virtual interfaces (VIs) on a port. Each virtual interface has its own MAC address, queues, and statistics. The dedicated netmap interfaces (ncxgbeX / ncxlX) were already implemented as additional VIs on each port. This change allows additional non-netmap interfaces to be configured on each port. Additional virtual interfaces use the naming scheme vcxgbeX or vcxlX. Additional VIs are enabled by setting the hw.cxgbe.num_vis tunable to a value greater than 1 before loading the cxgbe(4) or cxl(4) driver. NB: The first VI on each port is the "main" interface (cxgbeX or cxlX). T4/T5 NICs provide a limited number of MAC addresses for each physical port. As a result, a maximum of six VIs can be configured on each port (including the "main" interface and the netmap interface when netmap is enabled). One user-visible result is that when netmap is enabled, packets received or transmitted via the netmap interface are no longer counted in the stats for the "main" interface, but are not accounted to the netmap interface. The netmap interfaces now also have a new-bus device and export various information sysctl nodes via dev.n(cxgbe|cxl).X. The cxgbetool 'clearstats' command clears the stats for all VIs on the specified port along with the port's stats. There is currently no way to clear the stats of an individual VI. Reviewed by: np MFC after: 1 month Sponsored by: Chelsio
2015-12-03 00:02:01 +00:00
mtx_assert(&sc->sfl_lock, MA_OWNED);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
TAILQ_FOREACH_SAFE(fl, &sc->sfl, link, fl_temp) {
FL_LOCK(fl);
refill_fl(sc, fl, 64);
if (FL_NOT_RUNNING_LOW(fl) || fl->flags & FL_DOOMED) {
TAILQ_REMOVE(&sc->sfl, fl, link);
fl->flags &= ~FL_STARVING;
}
FL_UNLOCK(fl);
}
if (!TAILQ_EMPTY(&sc->sfl))
callout_schedule(&sc->sfl_callout, hz / 5);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
static int
alloc_fl_sdesc(struct sge_fl *fl)
{
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
fl->sdesc = malloc(fl->sidx * 8 * sizeof(struct fl_sdesc), M_CXGBE,
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
M_ZERO | M_WAITOK);
return (0);
}
static void
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
free_fl_sdesc(struct adapter *sc, struct sge_fl *fl)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
struct fl_sdesc *sd;
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
struct cluster_metadata *clm;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
int i;
sd = fl->sdesc;
cxgbe(4): some optimizations in freelist handling. MFC after: 2 weeks.
2014-08-02 06:55:36 +00:00
for (i = 0; i < fl->sidx * 8; i++, sd++) {
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
if (sd->cl == NULL)
continue;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Keep track of the clusters that have to be freed by the custom free routine (rxb_free) in the driver. Fail MOD_UNLOAD with EBUSY if any such cluster has been handed up to the kernel but hasn't been freed yet. This prevents a panic later when the cluster finally needs to be freed but rxb_free is gone from the kernel. MFC after: 1 week
2014-07-23 22:29:22 +00:00
if (sd->nmbuf == 0)
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
uma_zfree(sc->sge.rx_buf_info[sd->zidx].zone, sd->cl);
else if (fl->flags & FL_BUF_PACKING) {
clm = cl_metadata(sd);
if (atomic_fetchadd_int(&clm->refcount, -1) == 1) {
uma_zfree(sc->sge.rx_buf_info[sd->zidx].zone,
sd->cl);
counter_u64_add(extfree_rels, 1);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
sd->cl = NULL;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
free(fl->sdesc, M_CXGBE);
fl->sdesc = NULL;
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static inline void
get_pkt_gl(struct mbuf *m, struct sglist *gl)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
int rc;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
M_ASSERTPKTHDR(m);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
sglist_reset(gl);
rc = sglist_append_mbuf(gl, m);
if (__predict_false(rc != 0)) {
panic("%s: mbuf %p (%d segs) was vetted earlier but now fails "
"with %d.", __func__, m, mbuf_nsegs(m), rc);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
KASSERT(gl->sg_nseg == mbuf_nsegs(m),
("%s: nsegs changed for mbuf %p from %d to %d", __func__, m,
mbuf_nsegs(m), gl->sg_nseg));
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
#if 0 /* vm_wr not readily available here. */
KASSERT(gl->sg_nseg > 0 && gl->sg_nseg <= max_nsegs_allowed(m, vm_wr),
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
("%s: %d segments, should have been 1 <= nsegs <= %d", __func__,
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
gl->sg_nseg, max_nsegs_allowed(m, vm_wr)));
#endif
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/*
* len16 for a txpkt WR with a GL. Includes the firmware work request header.
*/
static inline u_int
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
txpkt_len16(u_int nsegs, const u_int extra)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
u_int n;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
MPASS(nsegs > 0);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
nsegs--; /* first segment is part of ulptx_sgl */
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
n = extra + sizeof(struct fw_eth_tx_pkt_wr) +
sizeof(struct cpl_tx_pkt_core) +
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
sizeof(struct ulptx_sgl) + 8 * ((3 * nsegs) / 2 + (nsegs & 1));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (howmany(n, 16));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
/*
* len16 for a txpkt_vm WR with a GL. Includes the firmware work
* request header.
*/
static inline u_int
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
txpkt_vm_len16(u_int nsegs, const u_int extra)
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
{
u_int n;
MPASS(nsegs > 0);
nsegs--; /* first segment is part of ulptx_sgl */
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
n = extra + sizeof(struct fw_eth_tx_pkt_vm_wr) +
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
sizeof(struct cpl_tx_pkt_core) +
sizeof(struct ulptx_sgl) + 8 * ((3 * nsegs) / 2 + (nsegs & 1));
return (howmany(n, 16));
}
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
static inline void
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
calculate_mbuf_len16(struct mbuf *m, bool vm_wr)
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
{
const int lso = sizeof(struct cpl_tx_pkt_lso_core);
const int tnl_lso = sizeof(struct cpl_tx_tnl_lso);
cxgbe(4): let the PF driver use VM work requests for transmit. This allows the PF interfaces to communicate with the VF interfaces over the internal switch in the ASIC. Fix the GL limits for VM work requests while here. MFC after: 3 days Sponsored by: Chelsio Communications
2020-09-22 04:16:40 +00:00
if (vm_wr) {
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (needs_tso(m))
set_mbuf_len16(m, txpkt_vm_len16(mbuf_nsegs(m), lso));
else
set_mbuf_len16(m, txpkt_vm_len16(mbuf_nsegs(m), 0));
return;
}
if (needs_tso(m)) {
if (needs_vxlan_tso(m))
set_mbuf_len16(m, txpkt_len16(mbuf_nsegs(m), tnl_lso));
else
set_mbuf_len16(m, txpkt_len16(mbuf_nsegs(m), lso));
} else
set_mbuf_len16(m, txpkt_len16(mbuf_nsegs(m), 0));
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
/*
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
* len16 for a txpkts type 0 WR with a GL. Does not include the firmware work
* request header.
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
*/
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static inline u_int
txpkts0_len16(u_int nsegs)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
u_int n;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
MPASS(nsegs > 0);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
nsegs--; /* first segment is part of ulptx_sgl */
n = sizeof(struct ulp_txpkt) + sizeof(struct ulptx_idata) +
sizeof(struct cpl_tx_pkt_core) + sizeof(struct ulptx_sgl) +
8 * ((3 * nsegs) / 2 + (nsegs & 1));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (howmany(n, 16));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
/*
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
* len16 for a txpkts type 1 WR with a GL. Does not include the firmware work
* request header.
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
*/
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static inline u_int
txpkts1_len16(void)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
u_int n;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
n = sizeof(struct cpl_tx_pkt_core) + sizeof(struct ulptx_sgl);
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (howmany(n, 16));
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static inline u_int
imm_payload(u_int ndesc)
{
u_int n;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
n = ndesc * EQ_ESIZE - sizeof(struct fw_eth_tx_pkt_wr) -
sizeof(struct cpl_tx_pkt_core);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (n);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
static inline uint64_t
csum_to_ctrl(struct adapter *sc, struct mbuf *m)
{
uint64_t ctrl;
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
int csum_type, l2hlen, l3hlen;
int x, y;
static const int csum_types[3][2] = {
{TX_CSUM_TCPIP, TX_CSUM_TCPIP6},
{TX_CSUM_UDPIP, TX_CSUM_UDPIP6},
{TX_CSUM_IP, 0}
};
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
M_ASSERTPKTHDR(m);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (!needs_hwcsum(m))
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
return (F_TXPKT_IPCSUM_DIS | F_TXPKT_L4CSUM_DIS);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
MPASS(m->m_pkthdr.l2hlen >= ETHER_HDR_LEN);
MPASS(m->m_pkthdr.l3hlen >= sizeof(struct ip));
if (needs_vxlan_csum(m)) {
MPASS(m->m_pkthdr.l4hlen > 0);
MPASS(m->m_pkthdr.l5hlen > 0);
MPASS(m->m_pkthdr.inner_l2hlen >= ETHER_HDR_LEN);
MPASS(m->m_pkthdr.inner_l3hlen >= sizeof(struct ip));
l2hlen = m->m_pkthdr.l2hlen + m->m_pkthdr.l3hlen +
m->m_pkthdr.l4hlen + m->m_pkthdr.l5hlen +
m->m_pkthdr.inner_l2hlen - ETHER_HDR_LEN;
l3hlen = m->m_pkthdr.inner_l3hlen;
} else {
l2hlen = m->m_pkthdr.l2hlen - ETHER_HDR_LEN;
l3hlen = m->m_pkthdr.l3hlen;
}
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
ctrl = 0;
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (!needs_l3_csum(m))
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
ctrl |= F_TXPKT_IPCSUM_DIS;
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (m->m_pkthdr.csum_flags & (CSUM_IP_TCP | CSUM_INNER_IP_TCP |
CSUM_IP6_TCP | CSUM_INNER_IP6_TCP))
x = 0; /* TCP */
else if (m->m_pkthdr.csum_flags & (CSUM_IP_UDP | CSUM_INNER_IP_UDP |
CSUM_IP6_UDP | CSUM_INNER_IP6_UDP))
x = 1; /* UDP */
else
x = 2;
if (m->m_pkthdr.csum_flags & (CSUM_IP | CSUM_IP_TCP | CSUM_IP_UDP |
CSUM_INNER_IP | CSUM_INNER_IP_TCP | CSUM_INNER_IP_UDP))
y = 0; /* IPv4 */
else {
MPASS(m->m_pkthdr.csum_flags & (CSUM_IP6_TCP | CSUM_IP6_UDP |
CSUM_INNER_IP6_TCP | CSUM_INNER_IP6_UDP));
y = 1; /* IPv6 */
}
/*
* needs_hwcsum returned true earlier so there must be some kind of
* checksum to calculate.
*/
csum_type = csum_types[x][y];
MPASS(csum_type != 0);
if (csum_type == TX_CSUM_IP)
ctrl |= F_TXPKT_L4CSUM_DIS;
ctrl |= V_TXPKT_CSUM_TYPE(csum_type) | V_TXPKT_IPHDR_LEN(l3hlen);
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
if (chip_id(sc) <= CHELSIO_T5)
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
ctrl |= V_TXPKT_ETHHDR_LEN(l2hlen);
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
else
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
ctrl |= V_T6_TXPKT_ETHHDR_LEN(l2hlen);
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
return (ctrl);
}
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
static inline void *
write_lso_cpl(void *cpl, struct mbuf *m0)
{
struct cpl_tx_pkt_lso_core *lso;
uint32_t ctrl;
KASSERT(m0->m_pkthdr.l2hlen > 0 && m0->m_pkthdr.l3hlen > 0 &&
m0->m_pkthdr.l4hlen > 0,
("%s: mbuf %p needs TSO but missing header lengths",
__func__, m0));
ctrl = V_LSO_OPCODE(CPL_TX_PKT_LSO) |
F_LSO_FIRST_SLICE | F_LSO_LAST_SLICE |
V_LSO_ETHHDR_LEN((m0->m_pkthdr.l2hlen - ETHER_HDR_LEN) >> 2) |
V_LSO_IPHDR_LEN(m0->m_pkthdr.l3hlen >> 2) |
V_LSO_TCPHDR_LEN(m0->m_pkthdr.l4hlen >> 2);
if (m0->m_pkthdr.l3hlen == sizeof(struct ip6_hdr))
ctrl |= F_LSO_IPV6;
lso = cpl;
lso->lso_ctrl = htobe32(ctrl);
lso->ipid_ofst = htobe16(0);
lso->mss = htobe16(m0->m_pkthdr.tso_segsz);
lso->seqno_offset = htobe32(0);
lso->len = htobe32(m0->m_pkthdr.len);
return (lso + 1);
}
static void *
write_tnl_lso_cpl(void *cpl, struct mbuf *m0)
{
struct cpl_tx_tnl_lso *tnl_lso = cpl;
uint32_t ctrl;
KASSERT(m0->m_pkthdr.inner_l2hlen > 0 &&
m0->m_pkthdr.inner_l3hlen > 0 && m0->m_pkthdr.inner_l4hlen > 0 &&
m0->m_pkthdr.inner_l5hlen > 0,
("%s: mbuf %p needs VXLAN_TSO but missing inner header lengths",
__func__, m0));
KASSERT(m0->m_pkthdr.l2hlen > 0 && m0->m_pkthdr.l3hlen > 0 &&
m0->m_pkthdr.l4hlen > 0 && m0->m_pkthdr.l5hlen > 0,
("%s: mbuf %p needs VXLAN_TSO but missing outer header lengths",
__func__, m0));
/* Outer headers. */
ctrl = V_CPL_TX_TNL_LSO_OPCODE(CPL_TX_TNL_LSO) |
F_CPL_TX_TNL_LSO_FIRST | F_CPL_TX_TNL_LSO_LAST |
V_CPL_TX_TNL_LSO_ETHHDRLENOUT(
(m0->m_pkthdr.l2hlen - ETHER_HDR_LEN) >> 2) |
V_CPL_TX_TNL_LSO_IPHDRLENOUT(m0->m_pkthdr.l3hlen >> 2) |
F_CPL_TX_TNL_LSO_IPLENSETOUT;
if (m0->m_pkthdr.l3hlen == sizeof(struct ip6_hdr))
ctrl |= F_CPL_TX_TNL_LSO_IPV6OUT;
else {
ctrl |= F_CPL_TX_TNL_LSO_IPHDRCHKOUT |
F_CPL_TX_TNL_LSO_IPIDINCOUT;
}
tnl_lso->op_to_IpIdSplitOut = htobe32(ctrl);
tnl_lso->IpIdOffsetOut = 0;
tnl_lso->UdpLenSetOut_to_TnlHdrLen =
htobe16(F_CPL_TX_TNL_LSO_UDPCHKCLROUT |
F_CPL_TX_TNL_LSO_UDPLENSETOUT |
V_CPL_TX_TNL_LSO_TNLHDRLEN(m0->m_pkthdr.l2hlen +
m0->m_pkthdr.l3hlen + m0->m_pkthdr.l4hlen +
m0->m_pkthdr.l5hlen) |
V_CPL_TX_TNL_LSO_TNLTYPE(TX_TNL_TYPE_VXLAN));
tnl_lso->r1 = 0;
/* Inner headers. */
ctrl = V_CPL_TX_TNL_LSO_ETHHDRLEN(
(m0->m_pkthdr.inner_l2hlen - ETHER_HDR_LEN) >> 2) |
V_CPL_TX_TNL_LSO_IPHDRLEN(m0->m_pkthdr.inner_l3hlen >> 2) |
V_CPL_TX_TNL_LSO_TCPHDRLEN(m0->m_pkthdr.inner_l4hlen >> 2);
if (m0->m_pkthdr.inner_l3hlen == sizeof(struct ip6_hdr))
ctrl |= F_CPL_TX_TNL_LSO_IPV6;
tnl_lso->Flow_to_TcpHdrLen = htobe32(ctrl);
tnl_lso->IpIdOffset = 0;
tnl_lso->IpIdSplit_to_Mss =
htobe16(V_CPL_TX_TNL_LSO_MSS(m0->m_pkthdr.tso_segsz));
tnl_lso->TCPSeqOffset = 0;
tnl_lso->EthLenOffset_Size =
htobe32(V_CPL_TX_TNL_LSO_SIZE(m0->m_pkthdr.len));
return (tnl_lso + 1);
}
cxgbev(4): Compare at most 16 bytes of the Ethernet header when trying to coalesce tx work requests. Note that Coverity will still treat this as an out-of-bounds access. We do want to compare 16B starting from ethmacdst but cmp_l2hdr was was going beyond that by 2B. cmp_l2hdr was introduced in r362905. Reported by: Coverity (CID 1430284) Sponsored by: Chelsio Communications
2020-07-13 19:15:29 +00:00
#define VM_TX_L2HDR_LEN 16 /* ethmacdst to vlantci */
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
/*
* Write a VM txpkt WR for this packet to the hardware descriptors, update the
* software descriptor, and advance the pidx. It is guaranteed that enough
* descriptors are available.
*
* The return value is the # of hardware descriptors used.
*/
static u_int
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
write_txpkt_vm_wr(struct adapter *sc, struct sge_txq *txq, struct mbuf *m0)
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
{
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
struct sge_eq *eq;
struct fw_eth_tx_pkt_vm_wr *wr;
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
struct tx_sdesc *txsd;
struct cpl_tx_pkt_core *cpl;
uint32_t ctrl; /* used in many unrelated places */
uint64_t ctrl1;
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
int len16, ndesc, pktlen, nsegs;
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
caddr_t dst;
TXQ_LOCK_ASSERT_OWNED(txq);
M_ASSERTPKTHDR(m0);
len16 = mbuf_len16(m0);
nsegs = mbuf_nsegs(m0);
pktlen = m0->m_pkthdr.len;
ctrl = sizeof(struct cpl_tx_pkt_core);
if (needs_tso(m0))
ctrl += sizeof(struct cpl_tx_pkt_lso_core);
cxgbe(4): Add a tx_len16_to_desc helper. No functional change. MFC after: 1 week Sponsored by: Chelsio Communications
2020-06-23 07:33:29 +00:00
ndesc = tx_len16_to_desc(len16);
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
/* Firmware work request header */
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
eq = &txq->eq;
wr = (void *)&eq->desc[eq->pidx];
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
wr->op_immdlen = htobe32(V_FW_WR_OP(FW_ETH_TX_PKT_VM_WR) |
V_FW_ETH_TX_PKT_WR_IMMDLEN(ctrl));
ctrl = V_FW_WR_LEN16(len16);
wr->equiq_to_len16 = htobe32(ctrl);
wr->r3[0] = 0;
wr->r3[1] = 0;
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
/*
* Copy over ethmacdst, ethmacsrc, ethtype, and vlantci.
* vlantci is ignored unless the ethtype is 0x8100, so it's
* simpler to always copy it rather than making it
* conditional. Also, it seems that we do not have to set
* vlantci or fake the ethtype when doing VLAN tag insertion.
*/
cxgbev(4): Compare at most 16 bytes of the Ethernet header when trying to coalesce tx work requests. Note that Coverity will still treat this as an out-of-bounds access. We do want to compare 16B starting from ethmacdst but cmp_l2hdr was was going beyond that by 2B. cmp_l2hdr was introduced in r362905. Reported by: Coverity (CID 1430284) Sponsored by: Chelsio Communications
2020-07-13 19:15:29 +00:00
m_copydata(m0, 0, VM_TX_L2HDR_LEN, wr->ethmacdst);
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
if (needs_tso(m0)) {
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
cpl = write_lso_cpl(wr + 1, m0);
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
txq->tso_wrs++;
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
} else
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
cpl = (void *)(wr + 1);
/* Checksum offload */
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
ctrl1 = csum_to_ctrl(sc, m0);
if (ctrl1 != (F_TXPKT_IPCSUM_DIS | F_TXPKT_L4CSUM_DIS))
Chelsio T4/T5 VF driver. The cxgbev/cxlv driver supports Virtual Function devices for Chelsio T4 and T4 adapters. The VF devices share most of their code with the existing PF4 driver (cxgbe/cxl) and as such the VF device driver currently depends on the PF4 driver. Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf PCI device driver that attaches to the VF device. It then creates child cxgbev/cxlv devices representing ports assigned to the VF. By default, the PF driver assigns a single port to each VF. t4vf_hw.c contains VF-specific routines from the shared code used to fetch VF-specific parameters from the firmware. t4_vf.c contains the VF-specific PCI device driver and includes its own attach routine. VF devices are required to use a different firmware request when transmitting packets (which in turn requires a different CPL message to encapsulate messages). This alternate firmware request does not permit chaining multiple packets in a single message, so each packet results in a firmware request. In addition, the different CPL message requires more detailed information when enabling hardware checksums, so parse_pkt() on VF devices must examine L2 and L3 headers for all packets (not just TSO packets) for VF devices. Finally, L2 checksums on non-UDP/non-TCP packets do not work reliably (the firmware trashes the IPv4 fragment field), so IPv4 checksums for such packets are calculated in software. Most of the other changes in the non-VF-specific code are to expose various variables and functions private to the PF driver so that they can be used by the VF driver. Note that a limited subset of cxgbetool functions are supported on VF devices including register dumps, scheduler classes, and clearing of statistics. In addition, TOE is not supported on VF devices, only for the PF interfaces. Reviewed by: np MFC after: 2 months Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
txq->txcsum++; /* some hardware assistance provided */
/* VLAN tag insertion */
if (needs_vlan_insertion(m0)) {
ctrl1 |= F_TXPKT_VLAN_VLD |
V_TXPKT_VLAN(m0->m_pkthdr.ether_vtag);
txq->vlan_insertion++;
}
/* CPL header */
cpl->ctrl0 = txq->cpl_ctrl0;
cpl->pack = 0;
cpl->len = htobe16(pktlen);
cpl->ctrl1 = htobe64(ctrl1);
/* SGL */
dst = (void *)(cpl + 1);
/*
* A packet using TSO will use up an entire descriptor for the
* firmware work request header, LSO CPL, and TX_PKT_XT CPL.
* If this descriptor is the last descriptor in the ring, wrap
* around to the front of the ring explicitly for the start of
* the sgl.
*/
if (dst == (void *)&eq->desc[eq->sidx]) {
dst = (void *)&eq->desc[0];
write_gl_to_txd(txq, m0, &dst, 0);
} else
write_gl_to_txd(txq, m0, &dst, eq->sidx - ndesc < eq->pidx);
txq->sgl_wrs++;
txq->txpkt_wrs++;
txsd = &txq->sdesc[eq->pidx];
txsd->m = m0;
txsd->desc_used = ndesc;
return (ndesc);
}
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
/*
* Write a raw WR to the hardware descriptors, update the software
* descriptor, and advance the pidx. It is guaranteed that enough
* descriptors are available.
*
* The return value is the # of hardware descriptors used.
*/
static u_int
write_raw_wr(struct sge_txq *txq, void *wr, struct mbuf *m0, u_int available)
{
struct sge_eq *eq = &txq->eq;
struct tx_sdesc *txsd;
struct mbuf *m;
caddr_t dst;
int len16, ndesc;
len16 = mbuf_len16(m0);
cxgbe(4): Add a tx_len16_to_desc helper. No functional change. MFC after: 1 week Sponsored by: Chelsio Communications
2020-06-23 07:33:29 +00:00
ndesc = tx_len16_to_desc(len16);
Add a facility for transmitting "raw" work requests on regular NIC queues. - Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to describe properties of a queued packet. The MC_RAW_WR flag indicates an mbuf holding a raw work request. mbuf_cflags() returns the current flags. - Raw work request mbufs are allocated via alloc_wr_mbuf() which will allocate a single contiguous range to hold the mbuf data. The consumer can use mtod() to obtain the start of the work request and write the required work request in the buffer. The mbuf can then be enqueued directly to the txq via mp_ring_enqueue(). - Since raw work requests might potentially send arbitrary work requests, only set the EQUIQ and EQUEQ bits on work requests that support them such as the normal tunneled Ethernet packet work requests. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D17811
2018-11-06 00:11:36 +00:00
MPASS(ndesc <= available);
dst = wr;
for (m = m0; m != NULL; m = m->m_next)
copy_to_txd(eq, mtod(m, caddr_t), &dst, m->m_len);
txq->raw_wrs++;
txsd = &txq->sdesc[eq->pidx];
txsd->m = m0;
txsd->desc_used = ndesc;
return (ndesc);
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/*
* Write a txpkt WR for this packet to the hardware descriptors, update the
* software descriptor, and advance the pidx. It is guaranteed that enough
* descriptors are available.
*
* The return value is the # of hardware descriptors used.
*/
static u_int
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
write_txpkt_wr(struct adapter *sc, struct sge_txq *txq, struct mbuf *m0,
u_int available)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
struct sge_eq *eq;
struct fw_eth_tx_pkt_wr *wr;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
struct tx_sdesc *txsd;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct cpl_tx_pkt_core *cpl;
uint32_t ctrl; /* used in many unrelated places */
uint64_t ctrl1;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
int len16, ndesc, pktlen, nsegs;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
caddr_t dst;
TXQ_LOCK_ASSERT_OWNED(txq);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
M_ASSERTPKTHDR(m0);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
len16 = mbuf_len16(m0);
nsegs = mbuf_nsegs(m0);
pktlen = m0->m_pkthdr.len;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
ctrl = sizeof(struct cpl_tx_pkt_core);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (needs_tso(m0)) {
if (needs_vxlan_tso(m0))
ctrl += sizeof(struct cpl_tx_tnl_lso);
else
ctrl += sizeof(struct cpl_tx_pkt_lso_core);
} else if (!(mbuf_cflags(m0) & MC_NOMAP) && pktlen <= imm_payload(2) &&
Add support for IFCAP_NOMAP to cxgbe(4). Since cxgbe(4) uses sglist instead of bus_dma, this required updates to the code that generates scatter/gather lists for packets. Also, unmapped mbufs are always sent via DMA and never as immediate data in the payload of a work request. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: np Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:52:21 +00:00
available >= 2) {
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/* Immediate data. Recalculate len16 and set nsegs to 0. */
Resume tx immediately in response to an SGE egress update from the hardware. MFC after: 1 week
2011-03-05 03:06:38 +00:00
ctrl += pktlen;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
len16 = howmany(sizeof(struct fw_eth_tx_pkt_wr) +
sizeof(struct cpl_tx_pkt_core) + pktlen, 16);
nsegs = 0;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): Add a tx_len16_to_desc helper. No functional change. MFC after: 1 week Sponsored by: Chelsio Communications
2020-06-23 07:33:29 +00:00
ndesc = tx_len16_to_desc(len16);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
MPASS(ndesc <= available);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
/* Firmware work request header */
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
eq = &txq->eq;
wr = (void *)&eq->desc[eq->pidx];
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
wr->op_immdlen = htobe32(V_FW_WR_OP(FW_ETH_TX_PKT_WR) |
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
V_FW_ETH_TX_PKT_WR_IMMDLEN(ctrl));
There is no need to request a tx credit flush if such a request is already pending. MFC after: 3 days
2011-04-14 20:06:23 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
ctrl = V_FW_WR_LEN16(len16);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
wr->equiq_to_len16 = htobe32(ctrl);
wr->r3 = 0;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (needs_tso(m0)) {
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (needs_vxlan_tso(m0)) {
cpl = write_tnl_lso_cpl(wr + 1, m0);
txq->vxlan_tso_wrs++;
} else {
cpl = write_lso_cpl(wr + 1, m0);
txq->tso_wrs++;
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
} else
cpl = (void *)(wr + 1);
/* Checksum offload */
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
ctrl1 = csum_to_ctrl(sc, m0);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (ctrl1 != (F_TXPKT_IPCSUM_DIS | F_TXPKT_L4CSUM_DIS)) {
/* some hardware assistance provided */
if (needs_vxlan_csum(m0))
txq->vxlan_txcsum++;
else
txq->txcsum++;
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
/* VLAN tag insertion */
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (needs_vlan_insertion(m0)) {
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
ctrl1 |= F_TXPKT_VLAN_VLD |
V_TXPKT_VLAN(m0->m_pkthdr.ether_vtag);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
txq->vlan_insertion++;
}
/* CPL header */
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
cpl->ctrl0 = txq->cpl_ctrl0;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cpl->pack = 0;
Resume tx immediately in response to an SGE egress update from the hardware. MFC after: 1 week
2011-03-05 03:06:38 +00:00
cpl->len = htobe16(pktlen);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cpl->ctrl1 = htobe64(ctrl1);
/* SGL */
dst = (void *)(cpl + 1);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (__predict_false((uintptr_t)dst == (uintptr_t)&eq->desc[eq->sidx]))
dst = (caddr_t)&eq->desc[0];
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (nsegs > 0) {
write_gl_to_txd(txq, m0, &dst, eq->sidx - ndesc < eq->pidx);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
txq->sgl_wrs++;
} else {
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
struct mbuf *m;
for (m = m0; m != NULL; m = m->m_next) {
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
copy_to_txd(eq, mtod(m, caddr_t), &dst, m->m_len);
Resume tx immediately in response to an SGE egress update from the hardware. MFC after: 1 week
2011-03-05 03:06:38 +00:00
#ifdef INVARIANTS
pktlen -= m->m_len;
#endif
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
Resume tx immediately in response to an SGE egress update from the hardware. MFC after: 1 week
2011-03-05 03:06:38 +00:00
#ifdef INVARIANTS
KASSERT(pktlen == 0, ("%s: %d bytes left.", __func__, pktlen));
#endif
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
txq->imm_wrs++;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
txq->txpkt_wrs++;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
txsd = &txq->sdesc[eq->pidx];
txsd->m = m0;
txsd->desc_used = ndesc;
return (ndesc);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
static inline bool
cmp_l2hdr(struct txpkts *txp, struct mbuf *m)
{
int len;
MPASS(txp->npkt > 0);
cxgbev(4): Compare at most 16 bytes of the Ethernet header when trying to coalesce tx work requests. Note that Coverity will still treat this as an out-of-bounds access. We do want to compare 16B starting from ethmacdst but cmp_l2hdr was was going beyond that by 2B. cmp_l2hdr was introduced in r362905. Reported by: Coverity (CID 1430284) Sponsored by: Chelsio Communications
2020-07-13 19:15:29 +00:00
MPASS(m->m_len >= VM_TX_L2HDR_LEN);
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
if (txp->ethtype == be16toh(ETHERTYPE_VLAN))
cxgbev(4): Compare at most 16 bytes of the Ethernet header when trying to coalesce tx work requests. Note that Coverity will still treat this as an out-of-bounds access. We do want to compare 16B starting from ethmacdst but cmp_l2hdr was was going beyond that by 2B. cmp_l2hdr was introduced in r362905. Reported by: Coverity (CID 1430284) Sponsored by: Chelsio Communications
2020-07-13 19:15:29 +00:00
len = VM_TX_L2HDR_LEN;
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
else
len = sizeof(struct ether_header);
return (memcmp(m->m_data, &txp->ethmacdst[0], len) != 0);
}
static inline void
save_l2hdr(struct txpkts *txp, struct mbuf *m)
{
cxgbev(4): Compare at most 16 bytes of the Ethernet header when trying to coalesce tx work requests. Note that Coverity will still treat this as an out-of-bounds access. We do want to compare 16B starting from ethmacdst but cmp_l2hdr was was going beyond that by 2B. cmp_l2hdr was introduced in r362905. Reported by: Coverity (CID 1430284) Sponsored by: Chelsio Communications
2020-07-13 19:15:29 +00:00
MPASS(m->m_len >= VM_TX_L2HDR_LEN);
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
cxgbev(4): Compare at most 16 bytes of the Ethernet header when trying to coalesce tx work requests. Note that Coverity will still treat this as an out-of-bounds access. We do want to compare 16B starting from ethmacdst but cmp_l2hdr was was going beyond that by 2B. cmp_l2hdr was introduced in r362905. Reported by: Coverity (CID 1430284) Sponsored by: Chelsio Communications
2020-07-13 19:15:29 +00:00
memcpy(&txp->ethmacdst[0], mtod(m, const void *), VM_TX_L2HDR_LEN);
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
static int
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
add_to_txpkts_vf(struct adapter *sc, struct sge_txq *txq, struct mbuf *m,
int avail, bool *send)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
struct txpkts *txp = &txq->txp;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
/* Cannot have TSO and coalesce at the same time. */
if (cannot_use_txpkts(m)) {
cannot_coalesce:
*send = txp->npkt > 0;
return (EINVAL);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
/* VF allows coalescing of type 1 (1 GL) only */
if (mbuf_nsegs(m) > 1)
goto cannot_coalesce;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
*send = false;
if (txp->npkt > 0) {
MPASS(tx_len16_to_desc(txp->len16) <= avail);
MPASS(txp->npkt < txp->max_npkt);
MPASS(txp->wr_type == 1); /* VF supports type 1 only */
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
if (tx_len16_to_desc(txp->len16 + txpkts1_len16()) > avail) {
retry_after_send:
*send = true;
return (EAGAIN);
}
if (m->m_pkthdr.len + txp->plen > 65535)
goto retry_after_send;
if (cmp_l2hdr(txp, m))
goto retry_after_send;
txp->len16 += txpkts1_len16();
txp->plen += m->m_pkthdr.len;
txp->mb[txp->npkt++] = m;
if (txp->npkt == txp->max_npkt)
*send = true;
} else {
txp->len16 = howmany(sizeof(struct fw_eth_tx_pkts_vm_wr), 16) +
txpkts1_len16();
if (tx_len16_to_desc(txp->len16) > avail)
goto cannot_coalesce;
txp->npkt = 1;
txp->wr_type = 1;
txp->plen = m->m_pkthdr.len;
txp->mb[0] = m;
save_l2hdr(txp, m);
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (0);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static int
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
add_to_txpkts_pf(struct adapter *sc, struct sge_txq *txq, struct mbuf *m,
int avail, bool *send)
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
{
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
struct txpkts *txp = &txq->txp;
int nsegs;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
MPASS(!(sc->flags & IS_VF));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
/* Cannot have TSO and coalesce at the same time. */
if (cannot_use_txpkts(m)) {
cannot_coalesce:
*send = txp->npkt > 0;
return (EINVAL);
}
Check cannot_use_txpkts() rather than needs_tso() in add_to_txpkts(). Currently this is a no-op, but will matter in the future when cannot_use_txpkts() starts checking other conditions than just needs_tso(). Sponsored by: Chelsio Communications
2018-11-01 21:49:49 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
*send = false;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
nsegs = mbuf_nsegs(m);
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
if (txp->npkt == 0) {
if (m->m_pkthdr.len > 65535)
goto cannot_coalesce;
if (nsegs > 1) {
txp->wr_type = 0;
txp->len16 =
howmany(sizeof(struct fw_eth_tx_pkts_wr), 16) +
txpkts0_len16(nsegs);
} else {
txp->wr_type = 1;
txp->len16 =
howmany(sizeof(struct fw_eth_tx_pkts_wr), 16) +
txpkts1_len16();
}
if (tx_len16_to_desc(txp->len16) > avail)
goto cannot_coalesce;
txp->npkt = 1;
txp->plen = m->m_pkthdr.len;
txp->mb[0] = m;
} else {
MPASS(tx_len16_to_desc(txp->len16) <= avail);
MPASS(txp->npkt < txp->max_npkt);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
if (m->m_pkthdr.len + txp->plen > 65535) {
retry_after_send:
*send = true;
return (EAGAIN);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
MPASS(txp->wr_type == 0 || txp->wr_type == 1);
if (txp->wr_type == 0) {
if (tx_len16_to_desc(txp->len16 +
txpkts0_len16(nsegs)) > min(avail, SGE_MAX_WR_NDESC))
goto retry_after_send;
txp->len16 += txpkts0_len16(nsegs);
} else {
if (nsegs != 1)
goto retry_after_send;
if (tx_len16_to_desc(txp->len16 + txpkts1_len16()) >
avail)
goto retry_after_send;
txp->len16 += txpkts1_len16();
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
txp->plen += m->m_pkthdr.len;
txp->mb[txp->npkt++] = m;
if (txp->npkt == txp->max_npkt)
*send = true;
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
return (0);
}
/*
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
* Write a txpkts WR for the packets in txp to the hardware descriptors, update
* the software descriptor, and advance the pidx. It is guaranteed that enough
* descriptors are available.
*
* The return value is the # of hardware descriptors used.
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
*/
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static u_int
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
write_txpkts_wr(struct adapter *sc, struct sge_txq *txq)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
const struct txpkts *txp = &txq->txp;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct sge_eq *eq = &txq->eq;
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
struct fw_eth_tx_pkts_wr *wr;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct tx_sdesc *txsd;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
struct cpl_tx_pkt_core *cpl;
uint64_t ctrl1;
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
int ndesc, i, checkwrap;
struct mbuf *m, *last;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
void *flitp;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
TXQ_LOCK_ASSERT_OWNED(txq);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
MPASS(txp->npkt > 0);
MPASS(txp->len16 <= howmany(SGE_MAX_WR_LEN, 16));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
wr = (void *)&eq->desc[eq->pidx];
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
wr->op_pkd = htobe32(V_FW_WR_OP(FW_ETH_TX_PKTS_WR));
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
wr->equiq_to_len16 = htobe32(V_FW_WR_LEN16(txp->len16));
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
wr->plen = htobe16(txp->plen);
wr->npkt = txp->npkt;
wr->r3 = 0;
wr->type = txp->wr_type;
flitp = wr + 1;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/*
* At this point we are 16B into a hardware descriptor. If checkwrap is
* set then we know the WR is going to wrap around somewhere. We'll
* check for that at appropriate points.
*/
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
ndesc = tx_len16_to_desc(txp->len16);
last = NULL;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
checkwrap = eq->sidx - ndesc < eq->pidx;
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
for (i = 0; i < txp->npkt; i++) {
m = txp->mb[i];
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (txp->wr_type == 0) {
struct ulp_txpkt *ulpmc;
struct ulptx_idata *ulpsc;
/* ULP master command */
ulpmc = flitp;
ulpmc->cmd_dest = htobe32(V_ULPTX_CMD(ULP_TX_PKT) |
V_ULP_TXPKT_DEST(0) | V_ULP_TXPKT_FID(eq->iqid));
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
ulpmc->len = htobe32(txpkts0_len16(mbuf_nsegs(m)));
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/* ULP subcommand */
ulpsc = (void *)(ulpmc + 1);
ulpsc->cmd_more = htobe32(V_ULPTX_CMD(ULP_TX_SC_IMM) |
F_ULP_TX_SC_MORE);
ulpsc->len = htobe32(sizeof(struct cpl_tx_pkt_core));
cpl = (void *)(ulpsc + 1);
if (checkwrap &&
(uintptr_t)cpl == (uintptr_t)&eq->desc[eq->sidx])
cpl = (void *)&eq->desc[0];
} else {
cpl = flitp;
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/* Checksum offload */
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
ctrl1 = csum_to_ctrl(sc, m);
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (ctrl1 != (F_TXPKT_IPCSUM_DIS | F_TXPKT_L4CSUM_DIS)) {
/* some hardware assistance provided */
if (needs_vxlan_csum(m))
txq->vxlan_txcsum++;
else
txq->txcsum++;
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/* VLAN tag insertion */
if (needs_vlan_insertion(m)) {
ctrl1 |= F_TXPKT_VLAN_VLD |
V_TXPKT_VLAN(m->m_pkthdr.ether_vtag);
txq->vlan_insertion++;
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
/* CPL header */
cpl->ctrl0 = txq->cpl_ctrl0;
cpl->pack = 0;
cpl->len = htobe16(m->m_pkthdr.len);
cpl->ctrl1 = htobe64(ctrl1);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
flitp = cpl + 1;
if (checkwrap &&
(uintptr_t)flitp == (uintptr_t)&eq->desc[eq->sidx])
flitp = (void *)&eq->desc[0];
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
write_gl_to_txd(txq, m, (caddr_t *)(&flitp), checkwrap);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
if (last != NULL)
last->m_nextpkt = m;
last = m;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
txq->sgl_wrs++;
cxgbe(4): Update the statistics for compound tx work requests once per work request, not once per frame. MFC after: 1 week Sponsored by: Chelsio Communications
2017-06-02 17:57:27 +00:00
if (txp->wr_type == 0) {
txq->txpkts0_pkts += txp->npkt;
txq->txpkts0_wrs++;
} else {
txq->txpkts1_pkts += txp->npkt;
txq->txpkts1_wrs++;
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
txsd = &txq->sdesc[eq->pidx];
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
txsd->m = txp->mb[0];
txsd->desc_used = ndesc;
return (ndesc);
}
static u_int
write_txpkts_vm_wr(struct adapter *sc, struct sge_txq *txq)
{
const struct txpkts *txp = &txq->txp;
struct sge_eq *eq = &txq->eq;
struct fw_eth_tx_pkts_vm_wr *wr;
struct tx_sdesc *txsd;
struct cpl_tx_pkt_core *cpl;
uint64_t ctrl1;
int ndesc, i;
struct mbuf *m, *last;
void *flitp;
TXQ_LOCK_ASSERT_OWNED(txq);
MPASS(txp->npkt > 0);
MPASS(txp->wr_type == 1); /* VF supports type 1 only */
MPASS(txp->mb[0] != NULL);
MPASS(txp->len16 <= howmany(SGE_MAX_WR_LEN, 16));
wr = (void *)&eq->desc[eq->pidx];
wr->op_pkd = htobe32(V_FW_WR_OP(FW_ETH_TX_PKTS_VM_WR));
wr->equiq_to_len16 = htobe32(V_FW_WR_LEN16(txp->len16));
wr->r3 = 0;
wr->plen = htobe16(txp->plen);
wr->npkt = txp->npkt;
wr->r4 = 0;
memcpy(&wr->ethmacdst[0], &txp->ethmacdst[0], 16);
flitp = wr + 1;
/*
* At this point we are 32B into a hardware descriptor. Each mbuf in
* the WR will take 32B so we check for the end of the descriptor ring
* before writing odd mbufs (mb[1], 3, 5, ..)
*/
ndesc = tx_len16_to_desc(txp->len16);
last = NULL;
for (i = 0; i < txp->npkt; i++) {
m = txp->mb[i];
if (i & 1 && (uintptr_t)flitp == (uintptr_t)&eq->desc[eq->sidx])
flitp = &eq->desc[0];
cpl = flitp;
/* Checksum offload */
ctrl1 = csum_to_ctrl(sc, m);
if (ctrl1 != (F_TXPKT_IPCSUM_DIS | F_TXPKT_L4CSUM_DIS))
txq->txcsum++; /* some hardware assistance provided */
/* VLAN tag insertion */
if (needs_vlan_insertion(m)) {
ctrl1 |= F_TXPKT_VLAN_VLD |
V_TXPKT_VLAN(m->m_pkthdr.ether_vtag);
txq->vlan_insertion++;
}
/* CPL header */
cpl->ctrl0 = txq->cpl_ctrl0;
cpl->pack = 0;
cpl->len = htobe16(m->m_pkthdr.len);
cpl->ctrl1 = htobe64(ctrl1);
flitp = cpl + 1;
MPASS(mbuf_nsegs(m) == 1);
write_gl_to_txd(txq, m, (caddr_t *)(&flitp), 0);
if (last != NULL)
last->m_nextpkt = m;
last = m;
}
txq->sgl_wrs++;
txq->txpkts1_pkts += txp->npkt;
txq->txpkts1_wrs++;
txsd = &txq->sdesc[eq->pidx];
txsd->m = txp->mb[0];
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
txsd->desc_used = ndesc;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (ndesc);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
/*
* If the SGL ends on an address that is not 16 byte aligned, this function will
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
* add a 0 filled flit at the end.
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
*/
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static void
write_gl_to_txd(struct sge_txq *txq, struct mbuf *m, caddr_t *to, int checkwrap)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
struct sge_eq *eq = &txq->eq;
struct sglist *gl = txq->gl;
struct sglist_seg *seg;
__be64 *flitp, *wrap;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
struct ulptx_sgl *usgl;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
int i, nflits, nsegs;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
KASSERT(((uintptr_t)(*to) & 0xf) == 0,
("%s: SGL must start at a 16 byte boundary: %p", __func__, *to));
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
MPASS((uintptr_t)(*to) >= (uintptr_t)&eq->desc[0]);
MPASS((uintptr_t)(*to) < (uintptr_t)&eq->desc[eq->sidx]);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
get_pkt_gl(m, gl);
nsegs = gl->sg_nseg;
MPASS(nsegs > 0);
nflits = (3 * (nsegs - 1)) / 2 + ((nsegs - 1) & 1) + 2;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
flitp = (__be64 *)(*to);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
wrap = (__be64 *)(&eq->desc[eq->sidx]);
seg = &gl->sg_segs[0];
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
usgl = (void *)flitp;
/*
* We start at a 16 byte boundary somewhere inside the tx descriptor
* ring, so we're at least 16 bytes away from the status page. There is
* no chance of a wrap around in the middle of usgl (which is 16 bytes).
*/
usgl->cmd_nsge = htobe32(V_ULPTX_CMD(ULP_TX_SC_DSGL) |
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
V_ULPTX_NSGE(nsegs));
usgl->len0 = htobe32(seg->ss_len);
usgl->addr0 = htobe64(seg->ss_paddr);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
seg++;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (checkwrap == 0 || (uintptr_t)(flitp + nflits) <= (uintptr_t)wrap) {
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
/* Won't wrap around at all */
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
for (i = 0; i < nsegs - 1; i++, seg++) {
usgl->sge[i / 2].len[i & 1] = htobe32(seg->ss_len);
usgl->sge[i / 2].addr[i & 1] = htobe64(seg->ss_paddr);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
if (i & 1)
usgl->sge[i / 2].len[1] = htobe32(0);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
flitp += nflits;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
} else {
/* Will wrap somewhere in the rest of the SGL */
/* 2 flits already written, write the rest flit by flit */
flitp = (void *)(usgl + 1);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
for (i = 0; i < nflits - 2; i++) {
if (flitp == wrap)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
flitp = (void *)eq->desc;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
*flitp++ = get_flit(seg, nsegs - 1, i);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (nflits & 1) {
MPASS(((uintptr_t)flitp) & 0xf);
*flitp++ = 0;
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
MPASS((((uintptr_t)flitp) & 0xf) == 0);
if (__predict_false(flitp == wrap))
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
*to = (void *)eq->desc;
else
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
*to = (void *)flitp;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
static inline void
copy_to_txd(struct sge_eq *eq, caddr_t from, caddr_t *to, int len)
{
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
MPASS((uintptr_t)(*to) >= (uintptr_t)&eq->desc[0]);
MPASS((uintptr_t)(*to) < (uintptr_t)&eq->desc[eq->sidx]);
if (__predict_true((uintptr_t)(*to) + len <=
(uintptr_t)&eq->desc[eq->sidx])) {
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
bcopy(from, *to, len);
(*to) += len;
} else {
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
int portion = (uintptr_t)&eq->desc[eq->sidx] - (uintptr_t)(*to);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
bcopy(from, *to, portion);
from += portion;
portion = len - portion; /* remaining */
bcopy(from, (void *)eq->desc, portion);
(*to) = (caddr_t)eq->desc + portion;
}
}
static inline void
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
ring_eq_db(struct adapter *sc, struct sge_eq *eq, u_int n)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
u_int db;
MPASS(n > 0);
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
db = eq->doorbells;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (n > 1)
Cosmetic change (s/wrwc/wcwr/;s/WRWC/WCWR/). MFC after: 3 days.
2013-04-11 22:49:29 +00:00
clrbit(&db, DOORBELL_WCWR);
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
wmb();
switch (ffs(db) - 1) {
case DOORBELL_UDB:
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
*eq->udb = htole32(V_QID(eq->udb_qid) | V_PIDX(n));
break;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
Cosmetic change (s/wrwc/wcwr/;s/WRWC/WCWR/). MFC after: 3 days.
2013-04-11 22:49:29 +00:00
case DOORBELL_WCWR: {
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
volatile uint64_t *dst, *src;
int i;
/*
* Queues whose 128B doorbell segment fits in the page do not
* use relative qid (udb_qid is always 0). Only queues with
Cosmetic change (s/wrwc/wcwr/;s/WRWC/WCWR/). MFC after: 3 days.
2013-04-11 22:49:29 +00:00
* doorbell segments can do WCWR.
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
*/
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
KASSERT(eq->udb_qid == 0 && n == 1,
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
("%s: inappropriate doorbell (0x%x, %d, %d) for eq %p",
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
__func__, eq->doorbells, n, eq->dbidx, eq));
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
dst = (volatile void *)((uintptr_t)eq->udb + UDBS_WR_OFFSET -
UDBS_DB_OFFSET);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
i = eq->dbidx;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
src = (void *)&eq->desc[i];
while (src != (void *)&eq->desc[i + 1])
*dst++ = *src++;
wmb();
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
break;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
}
case DOORBELL_UDBWC:
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
*eq->udb = htole32(V_QID(eq->udb_qid) | V_PIDX(n));
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
wmb();
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
break;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
case DOORBELL_KDB:
Store the offset of the KDOORBELL and GTS registers in the softc. VF devices use a different register layout than PF devices. Storing the offset in a value in the softc allows code to be shared between the PF and VF drivers. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7389
2016-08-01 22:39:51 +00:00
t4_write_reg(sc, sc->sge_kdoorbell_reg,
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
V_QID(eq->cntxt_id) | V_PIDX(n));
break;
cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This includes support for the NIC and TOE features of the 40G, 10G, and 1G/100M cards based on the T5. The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4) has been updated instead of writing a brand new driver. T5 cards will show up as cxl (short for cxlgb) ports attached to the t5nex bus driver. Sponsored by: Chelsio
2013-03-30 02:26:20 +00:00
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
IDXINCR(eq->dbidx, n, eq->sidx);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static inline u_int
reclaimable_tx_desc(struct sge_eq *eq)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
uint16_t hw_cidx;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
hw_cidx = read_hw_cidx(eq);
return (IDXDIFF(hw_cidx, eq->cidx, eq->sidx));
}
static inline u_int
total_available_tx_desc(struct sge_eq *eq)
{
uint16_t hw_cidx, pidx;
hw_cidx = read_hw_cidx(eq);
pidx = eq->pidx;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
if (pidx == hw_cidx)
return (eq->sidx - 1);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
else
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (IDXDIFF(hw_cidx, pidx, eq->sidx) - 1);
}
static inline uint16_t
read_hw_cidx(struct sge_eq *eq)
{
struct sge_qstat *spg = (void *)&eq->desc[eq->sidx];
uint16_t cidx = spg->cidx; /* stable snapshot */
return (be16toh(cidx));
Calculate how many descriptors can be reclaimed before calling reclaim_tx_descs
2011-03-05 03:54:37 +00:00
}
/*
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
* Reclaim 'n' descriptors approximately.
Calculate how many descriptors can be reclaimed before calling reclaim_tx_descs
2011-03-05 03:54:37 +00:00
*/
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static u_int
reclaim_tx_descs(struct sge_txq *txq, u_int n)
Calculate how many descriptors can be reclaimed before calling reclaim_tx_descs
2011-03-05 03:54:37 +00:00
{
struct tx_sdesc *txsd;
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
struct sge_eq *eq = &txq->eq;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
u_int can_reclaim, reclaimed;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
TXQ_LOCK_ASSERT_OWNED(txq);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
MPASS(n > 0);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
reclaimed = 0;
can_reclaim = reclaimable_tx_desc(eq);
while (can_reclaim && reclaimed < n) {
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
int ndesc;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
struct mbuf *m, *nextpkt;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
txsd = &txq->sdesc[eq->cidx];
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
ndesc = txsd->desc_used;
/* Firmware doesn't return "partial" credits. */
KASSERT(can_reclaim >= ndesc,
("%s: unexpected number of credits: %d, %d",
__func__, can_reclaim, ndesc));
Assert that reclaim_tx_descs() is always making forward progress. MFC after: 2 months Sponsored by: Chelsio Communications
2018-11-01 21:39:33 +00:00
KASSERT(ndesc != 0,
("%s: descriptor with no credits: cidx %d",
__func__, eq->cidx));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
for (m = txsd->m; m != NULL; m = nextpkt) {
nextpkt = m->m_nextpkt;
m->m_nextpkt = NULL;
m_freem(m);
}
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
reclaimed += ndesc;
Calculate how many descriptors can be reclaimed before calling reclaim_tx_descs
2011-03-05 03:54:37 +00:00
can_reclaim -= ndesc;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
IDXINCR(eq->cidx, ndesc, eq->sidx);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
return (reclaimed);
}
static void
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
tx_reclaim(void *arg, int n)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
struct sge_txq *txq = arg;
struct sge_eq *eq = &txq->eq;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
do {
if (TXQ_TRYLOCK(txq) == 0)
break;
n = reclaim_tx_descs(txq, 32);
if (eq->cidx == eq->pidx)
eq->equeqidx = eq->pidx;
TXQ_UNLOCK(txq);
} while (n > 0);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
static __be64
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
get_flit(struct sglist_seg *segs, int nsegs, int idx)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
int i = (idx / 3) * 2;
switch (idx % 3) {
case 0: {
CXGBE: fix get_filt to be endianness-aware Unconditional 32-bit shift is not endianness-safe. Modify the logic to work both on LE and BE. Submitted by: Wojciech Macek <wma@freebsd.org> Reviewed by: np Obtained from: Semihalf Sponsored by: IBM, QCM Technologies Differential revision: https://reviews.freebsd.org/D13102
2018-01-11 09:17:02 +00:00
uint64_t rc;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
CXGBE: fix get_filt to be endianness-aware Unconditional 32-bit shift is not endianness-safe. Modify the logic to work both on LE and BE. Submitted by: Wojciech Macek <wma@freebsd.org> Reviewed by: np Obtained from: Semihalf Sponsored by: IBM, QCM Technologies Differential revision: https://reviews.freebsd.org/D13102
2018-01-11 09:17:02 +00:00
rc = (uint64_t)segs[i].ss_len << 32;
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
if (i + 1 < nsegs)
CXGBE: fix get_filt to be endianness-aware Unconditional 32-bit shift is not endianness-safe. Modify the logic to work both on LE and BE. Submitted by: Wojciech Macek <wma@freebsd.org> Reviewed by: np Obtained from: Semihalf Sponsored by: IBM, QCM Technologies Differential revision: https://reviews.freebsd.org/D13102
2018-01-11 09:17:02 +00:00
rc |= (uint64_t)(segs[i + 1].ss_len);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
CXGBE: fix get_filt to be endianness-aware Unconditional 32-bit shift is not endianness-safe. Modify the logic to work both on LE and BE. Submitted by: Wojciech Macek <wma@freebsd.org> Reviewed by: np Obtained from: Semihalf Sponsored by: IBM, QCM Technologies Differential revision: https://reviews.freebsd.org/D13102
2018-01-11 09:17:02 +00:00
return (htobe64(rc));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
case 1:
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (htobe64(segs[i].ss_paddr));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
case 2:
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
return (htobe64(segs[i + 1].ss_paddr));
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
return (0);
}
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
static int
find_refill_source(struct adapter *sc, int maxp, bool packing)
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
{
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
int i, zidx = -1;
struct rx_buf_info *rxb = &sc->sge.rx_buf_info[0];
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if (packing) {
for (i = 0; i < SW_ZONE_SIZES; i++, rxb++) {
if (rxb->hwidx2 == -1)
continue;
if (rxb->size1 < PAGE_SIZE &&
rxb->size1 < largest_rx_cluster)
continue;
if (rxb->size1 > largest_rx_cluster)
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
break;
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
MPASS(rxb->size1 - rxb->size2 >= CL_METADATA_SIZE);
if (rxb->size2 >= maxp)
return (i);
zidx = i;
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
}
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
} else {
for (i = 0; i < SW_ZONE_SIZES; i++, rxb++) {
if (rxb->hwidx1 == -1)
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
continue;
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
if (rxb->size1 > largest_rx_cluster)
break;
if (rxb->size1 >= maxp)
return (i);
zidx = i;
Implement support for rx buffer packing. Enable it by default for T5 cards. This is a T4 and T5 chip feature which lets the chip deliver multiple Ethernet frames in a single buffer. This is more efficient within the chip, in the driver, and reduces wastage of space in rx buffers. - Always allocate rx buffers from the jumbop zone, no matter what the MTU is. Do not use the normal cluster refcounting mechanism. - Reserve space for an mbuf and a refcount in the cluster itself and let the chip DMA multiple frames in the rest. - Use the embedded mbuf for the first frame and allocate mbufs on the fly for any additional frames delivered in the cluster. Each of these mbufs has a reference on the underlying cluster.
2013-08-30 01:45:36 +00:00
}
}
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
return (zidx);
cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters. MFC after: 3 weeks
2011-02-18 08:00:26 +00:00
}
Resume tx immediately in response to an SGE egress update from the hardware. MFC after: 1 week
2011-03-05 03:06:38 +00:00
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
static void
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
add_fl_to_sfl(struct adapter *sc, struct sge_fl *fl)
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
{
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
mtx_lock(&sc->sfl_lock);
FL_LOCK(fl);
if ((fl->flags & FL_DOOMED) == 0) {
fl->flags |= FL_STARVING;
TAILQ_INSERT_TAIL(&sc->sfl, fl, link);
callout_reset(&sc->sfl_callout, hz / 5, refill_sfl, sc);
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
}
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
FL_UNLOCK(fl);
mtx_unlock(&sc->sfl_lock);
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
}
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static void
handle_wrq_egr_update(struct adapter *sc, struct sge_eq *eq)
{
struct sge_wrq *wrq = (void *)eq;
atomic_readandclear_int(&eq->equiq);
taskqueue_enqueue(sc->tq[eq->tx_chan], &wrq->wrq_tx_task);
}
static void
handle_eth_egr_update(struct adapter *sc, struct sge_eq *eq)
{
struct sge_txq *txq = (void *)eq;
MPASS((eq->flags & EQ_TYPEMASK) == EQ_ETH);
atomic_readandclear_int(&eq->equiq);
cxgbe(4): changes in the Tx path to help increase tx coalescing. - Ask the firmware for the number of frames that can be stuffed in one work request. - Modify mp_ring to increase the likelihood of tx coalescing when there are just one or two threads that are doing most of the tx. Add teeth to the abdication mechanism by pushing the consumer lock into mp_ring. This reduces the likelihood that a consumer will get stuck with all the work even though it is above its budget. - Add support for coalesced tx WR to the VF driver. This, with the changes above, results in a 7x improvement in the tx pps of the VF driver for some common cases. The firmware vets the L2 headers submitted by the VF driver and it's a big win if the checks are performed for a batch of packets and not each one individually. Reviewed by: jhb@ MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25454
2020-07-03 04:44:23 +00:00
if (mp_ring_is_idle(txq->r))
taskqueue_enqueue(sc->tq[eq->tx_chan], &txq->tx_reclaim_task);
else
mp_ring_check_drainage(txq->r, 64);
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
}
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
static int
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
handle_sge_egr_update(struct sge_iq *iq, const struct rss_header *rss,
struct mbuf *m)
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
{
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
const struct cpl_sge_egr_update *cpl = (const void *)(rss + 1);
unsigned int qid = G_EGR_QID(ntohl(cpl->opcode_qid));
struct adapter *sc = iq->adapter;
struct sge *s = &sc->sge;
struct sge_eq *eq;
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
static void (*h[])(struct adapter *, struct sge_eq *) = {NULL,
&handle_wrq_egr_update, &handle_eth_egr_update,
&handle_wrq_egr_update};
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
KASSERT(m == NULL, ("%s: payload with opcode %02x", __func__,
rss->opcode));
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
Track the base absolute ID of ingress and egress queues. Use this to map an absolute queue ID to a logical queue ID in interrupt handlers. For the regular cxgbe/cxl drivers this should be a no-op as the base absolute ID should be zero. VF devices have a non-zero base absolute ID and require this change. While here, export the absolute ID of egress queues via a sysctl. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7446
2016-08-09 17:49:42 +00:00
eq = s->eqmap[qid - s->eq_start - s->eq_base];
cxgbe(4): major tx rework. a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
2014-12-31 23:19:16 +00:00
(*h[eq->flags & EQ_TYPEMASK])(sc, eq);
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
return (0);
}
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
cxgbe(4): Ask the card's firmware to pad up tiny CPLs by encapsulating them in a firmware message if it is able to do so. This works out better for one of the FIFOs in the chip. MFC after: 5 days
2013-02-26 00:27:27 +00:00
/* handle_fw_msg works for both fw4_msg and fw6_msg because this is valid */
CTASSERT(offsetof(struct cpl_fw4_msg, data) == \
offsetof(struct cpl_fw6_msg, data));
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
static int
Allow for a different handler for each type of firmware message. MFC after: 2 weeks
2012-08-16 18:31:50 +00:00
handle_fw_msg(struct sge_iq *iq, const struct rss_header *rss, struct mbuf *m)
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
{
Allow for a different handler for each type of firmware message. MFC after: 2 weeks
2012-08-16 18:31:50 +00:00
struct adapter *sc = iq->adapter;
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
const struct cpl_fw6_msg *cpl = (const void *)(rss + 1);
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
Many updates to cxgbe(4) - Device configuration via plain text config file. Also able to operate when not attached to the chip as the master driver. - Generic "work request" queue that serves as the base for both ctrl and ofld tx queues. - Generic interrupt handler routine that can process any event on any kind of ingress queue (via a dispatch table). - A couple of new driver ioctls. cxgbetool can now install a firmware to the card ("loadfw" command) and can read the card's memory ("memdump" and "tcb" commands). - Lots of assorted information within dev.t4nex.X.misc.* This is primarily for debugging and won't show up in sysctl -a. - Code to manage the L2 tables on the chip. - Updates to cxgbe(4) man page to go with the tunables that have changed. - Updates to the shared code in common/ - Updates to the driver-firmware interface (now at fw 1.4.16.0) MFC after: 1 month
2011-12-16 02:09:51 +00:00
KASSERT(m == NULL, ("%s: payload with opcode %02x", __func__,
rss->opcode));
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
cxgbe(4): Ask the card's firmware to pad up tiny CPLs by encapsulating them in a firmware message if it is able to do so. This works out better for one of the FIFOs in the chip. MFC after: 5 days
2013-02-26 00:27:27 +00:00
if (cpl->type == FW_TYPE_RSSCPL || cpl->type == FW6_TYPE_RSSCPL) {
const struct rss_header *rss2;
rss2 = (const struct rss_header *)&cpl->data[0];
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
return (t4_cpl_handler[rss2->opcode](iq, rss2, m));
cxgbe(4): Ask the card's firmware to pad up tiny CPLs by encapsulating them in a firmware message if it is able to do so. This works out better for one of the FIFOs in the chip. MFC after: 5 days
2013-02-26 00:27:27 +00:00
}
cxgbe(4): Changes to the CPL-handler registration mechanism and code related to "shared" CPLs. a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single function. Allow callers to direct the response to any iq. Tidy up set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of magic constants. b) Remove all CPL handler tables from struct adapter. This reduces its size by around 2KB. All handlers are now registered at MOD_LOAD instead of attach or some kind of initialization/activation. The registration functions do not need an adapter parameter any more. c) Add per-iq handlers to deal with CPLs whose destination cannot be determined solely from the opcode. There are 2 such CPLs in use right now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq. t4_tom (including the DDP code) now uses the port's ctrlq to send L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq. fwq and ofld_rxq have different handlers that know what kind of tid to expect in the reply. Update t4_write_l2e and callers to to support any wrq/iq combination. Approved by: re@ (kib@) Sponsored by: Chelsio Communications
2016-07-05 01:29:24 +00:00
return (t4_fw_msg_handler[cpl->type](sc, &cpl->data[0]));
- Move all Ethernet specific items from sge_eq to sge_txq. sge_eq is now a suitable base for all kinds of egress queues. - Add control queues (sge_ctrlq) and allocate one of these per hardware channel. They can be used to program filters and steer traffic (and more). MFC after: 1 week
2011-04-19 22:08:28 +00:00
}
sysctl that displays the absolute queue id of an rxq.
2011-05-14 19:27:15 +00:00
Install a handler for firmware work request error messages. If a driver sends an malformed or disallowed work request, the firmware responds with a work request error. Previously the driver treated this is as an unexpected message and panicked. Now it decodes the error message to aid in debugging. Reviewed by: np (older version) MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D6950
2016-07-22 21:52:07 +00:00
/**
* t4_handle_wrerr_rpl - process a FW work request error message
* @adap: the adapter
* @rpl: start of the FW message
*/
static int
t4_handle_wrerr_rpl(struct adapter *adap, const __be64 *rpl)
{
u8 opcode = *(const u8 *)rpl;
const struct fw_error_cmd *e = (const void *)rpl;
unsigned int i;
if (opcode != FW_ERROR_CMD) {
log(LOG_ERR,
"%s: Received WRERR_RPL message with opcode %#x\n",
device_get_nameunit(adap->dev), opcode);
return (EINVAL);
}
log(LOG_ERR, "%s: FW_ERROR (%s) ", device_get_nameunit(adap->dev),
G_FW_ERROR_CMD_FATAL(be32toh(e->op_to_type)) ? "fatal" :
"non-fatal");
switch (G_FW_ERROR_CMD_TYPE(be32toh(e->op_to_type))) {
case FW_ERROR_TYPE_EXCEPTION:
log(LOG_ERR, "exception info:\n");
for (i = 0; i < nitems(e->u.exception.info); i++)
log(LOG_ERR, "%s%08x", i == 0 ? "\t" : " ",
be32toh(e->u.exception.info[i]));
log(LOG_ERR, "\n");
break;
case FW_ERROR_TYPE_HWMODULE:
log(LOG_ERR, "HW module regaddr %08x regval %08x\n",
be32toh(e->u.hwmodule.regaddr),
be32toh(e->u.hwmodule.regval));
break;
case FW_ERROR_TYPE_WR:
log(LOG_ERR, "WR cidx %d PF %d VF %d eqid %d hdr:\n",
be16toh(e->u.wr.cidx),
G_FW_ERROR_CMD_PFN(be16toh(e->u.wr.pfn_vfn)),
G_FW_ERROR_CMD_VFN(be16toh(e->u.wr.pfn_vfn)),
be32toh(e->u.wr.eqid));
for (i = 0; i < nitems(e->u.wr.wrhdr); i++)
log(LOG_ERR, "%s%02x", i == 0 ? "\t" : " ",
e->u.wr.wrhdr[i]);
log(LOG_ERR, "\n");
break;
case FW_ERROR_TYPE_ACL:
log(LOG_ERR, "ACL cidx %d PF %d VF %d eqid %d %s",
be16toh(e->u.acl.cidx),
G_FW_ERROR_CMD_PFN(be16toh(e->u.acl.pfn_vfn)),
G_FW_ERROR_CMD_VFN(be16toh(e->u.acl.pfn_vfn)),
be32toh(e->u.acl.eqid),
G_FW_ERROR_CMD_MV(be16toh(e->u.acl.mv_pkd)) ? "vlanid" :
"MAC");
for (i = 0; i < nitems(e->u.acl.val); i++)
log(LOG_ERR, " %02x", e->u.acl.val[i]);
log(LOG_ERR, "\n");
break;
default:
log(LOG_ERR, "type %#x\n",
G_FW_ERROR_CMD_TYPE(be32toh(e->op_to_type)));
return (EINVAL);
}
return (0);
}
sysctl that displays the absolute queue id of an rxq.
2011-05-14 19:27:15 +00:00
static int
- Specialized ingress queues that take interrupts for other ingress queues. Try to have a set of these per port when possible, fall back to sharing a common pool between all ports otherwise. - One control queue per port (used to be one per hardware channel). - t4_eth_rx now handles Ethernet rx only. - sysctls to display pidx/cidx for some queues. MFC after: 1 week
2011-05-30 21:34:44 +00:00
sysctl_uint16(SYSCTL_HANDLER_ARGS)
sysctl that displays the absolute queue id of an rxq.
2011-05-14 19:27:15 +00:00
{
uint16_t *id = arg1;
int i = *id;
return sysctl_handle_int(oidp, &i, 0, req);
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
static inline bool
bufidx_used(struct adapter *sc, int idx)
{
struct rx_buf_info *rxb = &sc->sge.rx_buf_info[0];
int i;
for (i = 0; i < SW_ZONE_SIZES; i++, rxb++) {
if (rxb->size1 > largest_rx_cluster)
continue;
if (rxb->hwidx1 == idx || rxb->hwidx2 == idx)
return (true);
}
return (false);
}
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
static int
sysctl_bufsizes(SYSCTL_HANDLER_ARGS)
{
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
struct adapter *sc = arg1;
struct sge_params *sp = &sc->params.sge;
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
int i, rc;
struct sbuf sb;
char c;
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
sbuf_new(&sb, NULL, 128, SBUF_AUTOEXTEND);
for (i = 0; i < SGE_FLBUF_SIZES; i++) {
if (bufidx_used(sc, i))
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
c = '*';
else
c = '\0';
cxgbe(4): Retire the allow_mbufs_in_cluster optimization. This simplifies the driver's rx fast path as well as the bookkeeping code that tracks various rx buffer sizes and layouts. MFC after: 1 week Sponsored by: Chelsio Communications
2020-02-04 00:51:10 +00:00
sbuf_printf(&sb, "%u%c ", sp->sge_fl_buffer_size[i], c);
cxgbe(4): significant rx rework. - More flexible cluster size selection, including the ability to fall back to a safe cluster size (PAGE_SIZE from zone_jumbop by default) in case an allocation of a larger size fails. - A single get_fl_payload() function that assembles the payload into an mbuf chain for any kind of freelist. This replaces two variants: one for freelists with buffer packing enabled and another for those without. - Buffer packing with any sized cluster. It was limited to 4K clusters only before this change. - Enable buffer packing for TOE rx queues as well. - Statistics and tunables to go with all these changes. The driver's man page will be updated separately. MFC after: 5 weeks
2014-03-18 20:14:13 +00:00
}
sbuf_trim(&sb);
sbuf_finish(&sb);
rc = sysctl_handle_string(oidp, sbuf_data(&sb), sbuf_len(&sb), req);
sbuf_delete(&sb);
return (rc);
}
cxgbe(4): Add a sysctl to manage the binding of a txq to a traffic class. Sponsored by: Chelsio Communications
2016-06-08 14:15:29 +00:00
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
#ifdef RATELIMIT
/*
* len16 for a txpkt WR with a GL. Includes the firmware work request header.
*/
static inline u_int
txpkt_eo_len16(u_int nsegs, u_int immhdrs, u_int tso)
{
u_int n;
MPASS(immhdrs > 0);
n = roundup2(sizeof(struct fw_eth_tx_eo_wr) +
sizeof(struct cpl_tx_pkt_core) + immhdrs, 16);
if (__predict_false(nsegs == 0))
goto done;
nsegs--; /* first segment is part of ulptx_sgl */
n += sizeof(struct ulptx_sgl) + 8 * ((3 * nsegs) / 2 + (nsegs & 1));
if (tso)
n += sizeof(struct cpl_tx_pkt_lso_core);
done:
return (howmany(n, 16));
}
#define ETID_FLOWC_NPARAMS 6
#define ETID_FLOWC_LEN (roundup2((sizeof(struct fw_flowc_wr) + \
ETID_FLOWC_NPARAMS * sizeof(struct fw_flowc_mnemval)), 16))
#define ETID_FLOWC_LEN16 (howmany(ETID_FLOWC_LEN, 16))
static int
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
send_etid_flowc_wr(struct cxgbe_rate_tag *cst, struct port_info *pi,
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
struct vi_info *vi)
{
struct wrq_cookie cookie;
cxgbe(4): Treat the viid as an opaque identifier. Recent firmwares prefer to use a different format for viid internally and this change allows them to do so. MFC after: 1 week Sponsored by: Chelsio Communications
2019-03-20 17:27:11 +00:00
u_int pfvf = pi->adapter->pf << S_FW_VIID_PFN;
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
struct fw_flowc_wr *flowc;
mtx_assert(&cst->lock, MA_OWNED);
MPASS((cst->flags & (EO_FLOWC_PENDING | EO_FLOWC_RPL_PENDING)) ==
EO_FLOWC_PENDING);
flowc = start_wrq_wr(cst->eo_txq, ETID_FLOWC_LEN16, &cookie);
if (__predict_false(flowc == NULL))
return (ENOMEM);
bzero(flowc, ETID_FLOWC_LEN);
flowc->op_to_nparams = htobe32(V_FW_WR_OP(FW_FLOWC_WR) |
V_FW_FLOWC_WR_NPARAMS(ETID_FLOWC_NPARAMS) | V_FW_WR_COMPL(0));
flowc->flowid_len16 = htonl(V_FW_WR_LEN16(ETID_FLOWC_LEN16) |
V_FW_WR_FLOWID(cst->etid));
flowc->mnemval[0].mnemonic = FW_FLOWC_MNEM_PFNVFN;
flowc->mnemval[0].val = htobe32(pfvf);
flowc->mnemval[1].mnemonic = FW_FLOWC_MNEM_CH;
flowc->mnemval[1].val = htobe32(pi->tx_chan);
flowc->mnemval[2].mnemonic = FW_FLOWC_MNEM_PORT;
flowc->mnemval[2].val = htobe32(pi->tx_chan);
flowc->mnemval[3].mnemonic = FW_FLOWC_MNEM_IQID;
flowc->mnemval[3].val = htobe32(cst->iqid);
flowc->mnemval[4].mnemonic = FW_FLOWC_MNEM_EOSTATE;
flowc->mnemval[4].val = htobe32(FW_FLOWC_MNEM_EOSTATE_ESTABLISHED);
flowc->mnemval[5].mnemonic = FW_FLOWC_MNEM_SCHEDCLASS;
flowc->mnemval[5].val = htobe32(cst->schedcl);
commit_wrq_wr(cst->eo_txq, flowc, &cookie);
cst->flags &= ~EO_FLOWC_PENDING;
cst->flags |= EO_FLOWC_RPL_PENDING;
MPASS(cst->tx_credits >= ETID_FLOWC_LEN16); /* flowc is first WR. */
cst->tx_credits -= ETID_FLOWC_LEN16;
return (0);
}
#define ETID_FLUSH_LEN16 (howmany(sizeof (struct fw_flowc_wr), 16))
void
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
send_etid_flush_wr(struct cxgbe_rate_tag *cst)
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
{
struct fw_flowc_wr *flowc;
struct wrq_cookie cookie;
mtx_assert(&cst->lock, MA_OWNED);
flowc = start_wrq_wr(cst->eo_txq, ETID_FLUSH_LEN16, &cookie);
if (__predict_false(flowc == NULL))
CXGBE_UNIMPLEMENTED(__func__);
bzero(flowc, ETID_FLUSH_LEN16 * 16);
flowc->op_to_nparams = htobe32(V_FW_WR_OP(FW_FLOWC_WR) |
V_FW_FLOWC_WR_NPARAMS(0) | F_FW_WR_COMPL);
flowc->flowid_len16 = htobe32(V_FW_WR_LEN16(ETID_FLUSH_LEN16) |
V_FW_WR_FLOWID(cst->etid));
commit_wrq_wr(cst->eo_txq, flowc, &cookie);
cst->flags |= EO_FLUSH_RPL_PENDING;
MPASS(cst->tx_credits >= ETID_FLUSH_LEN16);
cst->tx_credits -= ETID_FLUSH_LEN16;
cst->ncompl++;
}
static void
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
write_ethofld_wr(struct cxgbe_rate_tag *cst, struct fw_eth_tx_eo_wr *wr,
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
struct mbuf *m0, int compl)
{
struct cpl_tx_pkt_core *cpl;
uint64_t ctrl1;
uint32_t ctrl; /* used in many unrelated places */
int len16, pktlen, nsegs, immhdrs;
caddr_t dst;
uintptr_t p;
struct ulptx_sgl *usgl;
struct sglist sg;
struct sglist_seg segs[38]; /* XXX: find real limit. XXX: get off the stack */
mtx_assert(&cst->lock, MA_OWNED);
M_ASSERTPKTHDR(m0);
KASSERT(m0->m_pkthdr.l2hlen > 0 && m0->m_pkthdr.l3hlen > 0 &&
m0->m_pkthdr.l4hlen > 0,
("%s: ethofld mbuf %p is missing header lengths", __func__, m0));
len16 = mbuf_eo_len16(m0);
nsegs = mbuf_eo_nsegs(m0);
pktlen = m0->m_pkthdr.len;
ctrl = sizeof(struct cpl_tx_pkt_core);
if (needs_tso(m0))
ctrl += sizeof(struct cpl_tx_pkt_lso_core);
immhdrs = m0->m_pkthdr.l2hlen + m0->m_pkthdr.l3hlen + m0->m_pkthdr.l4hlen;
ctrl += immhdrs;
wr->op_immdlen = htobe32(V_FW_WR_OP(FW_ETH_TX_EO_WR) |
V_FW_ETH_TX_EO_WR_IMMDLEN(ctrl) | V_FW_WR_COMPL(!!compl));
wr->equiq_to_len16 = htobe32(V_FW_WR_LEN16(len16) |
V_FW_WR_FLOWID(cst->etid));
wr->r3 = 0;
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
if (needs_outer_udp_csum(m0)) {
cxgbe(4): Add rate limiting support for UDP. MFC after: 1 month Sponsored by: Chelsio Communications
2018-10-31 19:19:13 +00:00
wr->u.udpseg.type = FW_ETH_TX_EO_TYPE_UDPSEG;
wr->u.udpseg.ethlen = m0->m_pkthdr.l2hlen;
wr->u.udpseg.iplen = htobe16(m0->m_pkthdr.l3hlen);
wr->u.udpseg.udplen = m0->m_pkthdr.l4hlen;
wr->u.udpseg.rtplen = 0;
wr->u.udpseg.r4 = 0;
wr->u.udpseg.mss = htobe16(pktlen - immhdrs);
wr->u.udpseg.schedpktsize = wr->u.udpseg.mss;
wr->u.udpseg.plen = htobe32(pktlen - immhdrs);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
cpl = (void *)(wr + 1);
cxgbe(4): Add rate limiting support for UDP. MFC after: 1 month Sponsored by: Chelsio Communications
2018-10-31 19:19:13 +00:00
} else {
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
MPASS(needs_outer_tcp_csum(m0));
cxgbe(4): Add rate limiting support for UDP. MFC after: 1 month Sponsored by: Chelsio Communications
2018-10-31 19:19:13 +00:00
wr->u.tcpseg.type = FW_ETH_TX_EO_TYPE_TCPSEG;
wr->u.tcpseg.ethlen = m0->m_pkthdr.l2hlen;
wr->u.tcpseg.iplen = htobe16(m0->m_pkthdr.l3hlen);
wr->u.tcpseg.tcplen = m0->m_pkthdr.l4hlen;
wr->u.tcpseg.tsclk_tsoff = mbuf_eo_tsclk_tsoff(m0);
wr->u.tcpseg.r4 = 0;
wr->u.tcpseg.r5 = 0;
wr->u.tcpseg.plen = htobe32(pktlen - immhdrs);
if (needs_tso(m0)) {
struct cpl_tx_pkt_lso_core *lso = (void *)(wr + 1);
wr->u.tcpseg.mss = htobe16(m0->m_pkthdr.tso_segsz);
ctrl = V_LSO_OPCODE(CPL_TX_PKT_LSO) |
F_LSO_FIRST_SLICE | F_LSO_LAST_SLICE |
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
V_LSO_ETHHDR_LEN((m0->m_pkthdr.l2hlen -
ETHER_HDR_LEN) >> 2) |
cxgbe(4): Add rate limiting support for UDP. MFC after: 1 month Sponsored by: Chelsio Communications
2018-10-31 19:19:13 +00:00
V_LSO_IPHDR_LEN(m0->m_pkthdr.l3hlen >> 2) |
V_LSO_TCPHDR_LEN(m0->m_pkthdr.l4hlen >> 2);
if (m0->m_pkthdr.l3hlen == sizeof(struct ip6_hdr))
ctrl |= F_LSO_IPV6;
lso->lso_ctrl = htobe32(ctrl);
lso->ipid_ofst = htobe16(0);
lso->mss = htobe16(m0->m_pkthdr.tso_segsz);
lso->seqno_offset = htobe32(0);
lso->len = htobe32(pktlen);
cpl = (void *)(lso + 1);
} else {
wr->u.tcpseg.mss = htobe16(0xffff);
cpl = (void *)(wr + 1);
}
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
}
/* Checksum offload must be requested for ethofld. */
cxgbe(4): add support for stateless offloads for VXLAN traffic. Hardware assistance includes checksumming (tx and rx), TSO, and RSS on the inner traffic in a VXLAN tunnel. Relnotes: Yes Sponsored by: Chelsio Communications
2020-09-18 03:01:47 +00:00
MPASS(needs_outer_l4_csum(m0));
cxgbe(4): Use the _XT variant of the CPL used to transmit NIC traffic. CPL_TX_PKT_XT disables the internal parser on the chip and instead relies on the driver to provide the exact length of the L2 and L3 headers. This allows hw checksumming and TSO to be used with L2 and L3 encapsulations that the chip doesn't understand directly. Note that netmap tx still uses the old CPL as it never uses the hw to generate the checksum on tx. Reviewed by: jhb@ MFC after: 1 month Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22788
2019-12-13 20:38:58 +00:00
ctrl1 = csum_to_ctrl(cst->adapter, m0);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
/* VLAN tag insertion */
if (needs_vlan_insertion(m0)) {
ctrl1 |= F_TXPKT_VLAN_VLD |
V_TXPKT_VLAN(m0->m_pkthdr.ether_vtag);
}
/* CPL header */
cpl->ctrl0 = cst->ctrl0;
cpl->pack = 0;
cpl->len = htobe16(pktlen);
cpl->ctrl1 = htobe64(ctrl1);
cxgbe(4): Add rate limiting support for UDP. MFC after: 1 month Sponsored by: Chelsio Communications
2018-10-31 19:19:13 +00:00
/* Copy Ethernet, IP & TCP/UDP hdrs as immediate data */
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
p = (uintptr_t)(cpl + 1);
m_copydata(m0, 0, immhdrs, (void *)p);
/* SGL */
dst = (void *)(cpl + 1);
if (nsegs > 0) {
int i, pad;
/* zero-pad upto next 16Byte boundary, if not 16Byte aligned */
p += immhdrs;
pad = 16 - (immhdrs & 0xf);
bzero((void *)p, pad);
usgl = (void *)(p + pad);
usgl->cmd_nsge = htobe32(V_ULPTX_CMD(ULP_TX_SC_DSGL) |
V_ULPTX_NSGE(nsegs));
sglist_init(&sg, nitems(segs), segs);
for (; m0 != NULL; m0 = m0->m_next) {
if (__predict_false(m0->m_len == 0))
continue;
if (immhdrs >= m0->m_len) {
immhdrs -= m0->m_len;
continue;
}
Step 4.1: mechanically rename M_NOMAP to M_EXTPG Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:21:11 +00:00
if (m0->m_flags & M_EXTPG)
Step 2.2: o Shrink sglist(9) functions to work with multipage mbufs down from four functions to two. o Don't use 'struct mbuf_ext_pgs *' as argument, use struct mbuf. o Rename to something matching _epg. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 23:46:29 +00:00
sglist_append_mbuf_epg(&sg, m0,
mtod(m0, vm_offset_t), m0->m_len);
else
sglist_append(&sg, mtod(m0, char *) + immhdrs,
m0->m_len - immhdrs);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
immhdrs = 0;
}
MPASS(sg.sg_nseg == nsegs);
/*
* Zero pad last 8B in case the WR doesn't end on a 16B
* boundary.
*/
*(uint64_t *)((char *)wr + len16 * 16 - 8) = 0;
usgl->len0 = htobe32(segs[0].ss_len);
usgl->addr0 = htobe64(segs[0].ss_paddr);
for (i = 0; i < nsegs - 1; i++) {
usgl->sge[i / 2].len[i & 1] = htobe32(segs[i + 1].ss_len);
usgl->sge[i / 2].addr[i & 1] = htobe64(segs[i + 1].ss_paddr);
}
if (i & 1)
usgl->sge[i / 2].len[1] = htobe32(0);
}
}
static void
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
ethofld_tx(struct cxgbe_rate_tag *cst)
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
{
struct mbuf *m;
struct wrq_cookie cookie;
int next_credits, compl;
struct fw_eth_tx_eo_wr *wr;
mtx_assert(&cst->lock, MA_OWNED);
while ((m = mbufq_first(&cst->pending_tx)) != NULL) {
M_ASSERTPKTHDR(m);
/* How many len16 credits do we need to send this mbuf. */
next_credits = mbuf_eo_len16(m);
MPASS(next_credits > 0);
if (next_credits > cst->tx_credits) {
/*
* Tx will make progress eventually because there is at
* least one outstanding fw4_ack that will return
* credits and kick the tx.
*/
MPASS(cst->ncompl > 0);
return;
}
wr = start_wrq_wr(cst->eo_txq, next_credits, &cookie);
if (__predict_false(wr == NULL)) {
/* XXX: wishful thinking, not a real assertion. */
MPASS(cst->ncompl > 0);
return;
}
cst->tx_credits -= next_credits;
cst->tx_nocompl += next_credits;
compl = cst->ncompl == 0 || cst->tx_nocompl >= cst->tx_total / 2;
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
ETHER_BPF_MTAP(cst->com.ifp, m);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
write_ethofld_wr(cst, wr, m, compl);
commit_wrq_wr(cst->eo_txq, wr, &cookie);
if (compl) {
cst->ncompl++;
cst->tx_nocompl = 0;
}
(void) mbufq_dequeue(&cst->pending_tx);
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
/*
* Drop the mbuf's reference on the tag now rather
* than waiting until m_freem(). This ensures that
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
* cxgbe_rate_tag_free gets called when the inp drops
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
* its reference on the tag and there are no more
* mbufs in the pending_tx queue and can flush any
* pending requests. Otherwise if the last mbuf
* doesn't request a completion the etid will never be
* released.
*/
m->m_pkthdr.snd_tag = NULL;
m->m_pkthdr.csum_flags &= ~CSUM_SND_TAG;
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
m_snd_tag_rele(&cst->com);
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
mbufq_enqueue(&cst->pending_fwack, m);
}
}
int
ethofld_transmit(struct ifnet *ifp, struct mbuf *m0)
{
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
struct cxgbe_rate_tag *cst;
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
int rc;
MPASS(m0->m_nextpkt == NULL);
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
MPASS(m0->m_pkthdr.csum_flags & CSUM_SND_TAG);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
MPASS(m0->m_pkthdr.snd_tag != NULL);
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
cst = mst_to_crt(m0->m_pkthdr.snd_tag);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
mtx_lock(&cst->lock);
MPASS(cst->flags & EO_SND_TAG_REF);
if (__predict_false(cst->flags & EO_FLOWC_PENDING)) {
struct vi_info *vi = ifp->if_softc;
struct port_info *pi = vi->pi;
struct adapter *sc = pi->adapter;
const uint32_t rss_mask = vi->rss_size - 1;
uint32_t rss_hash;
cst->eo_txq = &sc->sge.ofld_txq[vi->first_ofld_txq];
if (M_HASHTYPE_ISHASH(m0))
rss_hash = m0->m_pkthdr.flowid;
else
rss_hash = arc4random();
/* We assume RSS hashing */
cst->iqid = vi->rss[rss_hash & rss_mask];
cst->eo_txq += rss_hash % vi->nofldtxq;
rc = send_etid_flowc_wr(cst, pi, vi);
if (rc != 0)
goto done;
}
if (__predict_false(cst->plen + m0->m_pkthdr.len > eo_max_backlog)) {
rc = ENOBUFS;
goto done;
}
mbufq_enqueue(&cst->pending_tx, m0);
cst->plen += m0->m_pkthdr.len;
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
/*
* Hold an extra reference on the tag while generating work
* requests to ensure that we don't try to free the tag during
* ethofld_tx() in case we are sending the final mbuf after
* the inp was freed.
*/
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
m_snd_tag_ref(&cst->com);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
ethofld_tx(cst);
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
mtx_unlock(&cst->lock);
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
m_snd_tag_rele(&cst->com);
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
return (0);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
done:
mtx_unlock(&cst->lock);
if (__predict_false(rc != 0))
m_freem(m0);
return (rc);
}
static int
ethofld_fw4_ack(struct sge_iq *iq, const struct rss_header *rss, struct mbuf *m0)
{
struct adapter *sc = iq->adapter;
const struct cpl_fw4_ack *cpl = (const void *)(rss + 1);
struct mbuf *m;
u_int etid = G_CPL_FW4_ACK_FLOWID(be32toh(OPCODE_TID(cpl)));
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
struct cxgbe_rate_tag *cst;
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
uint8_t credits = cpl->credits;
cst = lookup_etid(sc, etid);
mtx_lock(&cst->lock);
if (__predict_false(cst->flags & EO_FLOWC_RPL_PENDING)) {
MPASS(credits >= ETID_FLOWC_LEN16);
credits -= ETID_FLOWC_LEN16;
cst->flags &= ~EO_FLOWC_RPL_PENDING;
}
KASSERT(cst->ncompl > 0,
("%s: etid %u (%p) wasn't expecting completion.",
__func__, etid, cst));
cst->ncompl--;
while (credits > 0) {
m = mbufq_dequeue(&cst->pending_fwack);
if (__predict_false(m == NULL)) {
/*
* The remaining credits are for the final flush that
* was issued when the tag was freed by the kernel.
*/
MPASS((cst->flags &
(EO_FLUSH_RPL_PENDING | EO_SND_TAG_REF)) ==
EO_FLUSH_RPL_PENDING);
MPASS(credits == ETID_FLUSH_LEN16);
MPASS(cst->tx_credits + cpl->credits == cst->tx_total);
MPASS(cst->ncompl == 0);
cst->flags &= ~EO_FLUSH_RPL_PENDING;
cst->tx_credits += cpl->credits;
Split Chelsio send tags into a generic base tag and a ratelimit tag. NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a distinct tag from a ratelimit tag. To support this, refactor cxgbe_snd_tag to be a simple send tag with a type and convert the existing ratelimit tag to a new cxgbe_rate_tag structure. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D22072
2019-10-22 20:41:54 +00:00
cxgbe_rate_tag_free_locked(cst);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
return (0); /* cst is gone. */
}
KASSERT(m != NULL,
("%s: too many credits (%u, %u)", __func__, cpl->credits,
credits));
KASSERT(credits >= mbuf_eo_len16(m),
("%s: too few credits (%u, %u, %u)", __func__,
cpl->credits, credits, mbuf_eo_len16(m)));
credits -= mbuf_eo_len16(m);
cst->plen -= m->m_pkthdr.len;
m_freem(m);
}
cst->tx_credits += cpl->credits;
MPASS(cst->tx_credits <= cst->tx_total);
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
if (cst->flags & EO_SND_TAG_REF) {
/*
* As with ethofld_transmit(), hold an extra reference
* so that the tag is stable across ethold_tx().
*/
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
m_snd_tag_ref(&cst->com);
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
m = mbufq_first(&cst->pending_tx);
if (m != NULL && cst->tx_credits >= mbuf_eo_len16(m))
ethofld_tx(cst);
mtx_unlock(&cst->lock);
Store the send tag type in the common send tag header. Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689
2020-10-06 17:58:56 +00:00
m_snd_tag_rele(&cst->com);
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
} else {
/*
* There shouldn't be any pending packets if the tag
* was freed by the kernel since any pending packet
* should hold a reference to the tag.
*/
MPASS(mbufq_first(&cst->pending_tx) == NULL);
mtx_unlock(&cst->lock);
cxgbe(4): Data path for rate-limited tx. This is hardware support for the SO_MAX_PACING_RATE sockopt (see setsockopt(2)), which is available in kernels built with "options RATELIMIT". Relnotes: Yes Sponsored by: Chelsio Communications
2018-05-24 10:18:14 +00:00
}
return (0);
}
#endif
Reference in New Issue Copy Permalink