2011-02-18 08:00:26 +00:00
|
|
|
/*-
|
2017-11-27 14:52:40 +00:00
|
|
|
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
|
|
|
|
*
|
2011-02-18 08:00:26 +00:00
|
|
|
* Copyright (c) 2011 Chelsio Communications, Inc.
|
|
|
|
* All rights reserved.
|
|
|
|
* Written by: Navdeep Parhar <np@FreeBSD.org>
|
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
* are met:
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
|
|
|
*
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
*
|
|
|
|
* $FreeBSD$
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
|
|
|
#ifndef __T4_ADAPTER_H__
|
|
|
|
#define __T4_ADAPTER_H__
|
|
|
|
|
2011-12-16 02:09:51 +00:00
|
|
|
#include <sys/kernel.h>
|
2011-02-18 08:00:26 +00:00
|
|
|
#include <sys/bus.h>
|
NIC KTLS for Chelsio T6 adapters.
This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters.
Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE
connections.
NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment
the encrypted TLS frames output by the crypto engine. Instead, the
TOE is placed into a special setup to permit "dummy" connections to be
associated with regular sockets using KTLS. This permits using the
TOE to segment the encrypted TLS records. However, this approach does
have some limitations:
1) Regular TOE sockets cannot be used when the TOE is in this special
mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but
not both at the same time.
2) In NIC KTLS mode, the TOE is only able to accept a per-connection
timestamp offset that varies in the upper 4 bits. Put another way,
only connections whose timestamp offset has the 28 lower bits
cleared can use NIC KTLS and generate correct timestamps. The
driver will refuse to enable NIC KTLS on connections with a
timestamp offset with any of the lower 28 bits set. To use NIC
KTLS, users can either disable TCP timestamps by setting the
net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the
tcp_new_ts_offset() function to clear the lower 28 bits of the
generated offset.
3) Because the TCP segmentation relies on fields mirrored in a TCB in
the TOE, not all fields in a TCP packet can be sent in the TCP
segments generated from a TLS record. Specifically, for packets
containing TCP options other than timestamps, the driver will
inject an "empty" TCP packet holding the requested options (e.g. a
SACK scoreboard) along with the segments from the TLS record.
These empty TCP packets are counted by the
dev.cc.N.txq.M.kern_tls_options sysctls.
Unlike TOE TLS which is able to buffer encrypted TLS records in
on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS
records for retransmit requests as well as non-retransmit requests
that do not include the start of a TLS record but do include the
trailer. The T6 NIC KTLS code tries to optimize some of the cases for
requests to transmit partial TLS records. In particular it attempts
to minimize sending "waste" bytes that have to be given as input to
the crypto engine but are not needed on the wire to satisfy mbufs sent
from the TCP stack down to the driver.
TCP packets for TLS requests are broken down into the following
classes (with associated counters):
- Mbufs that send an entire TLS record in full do not have any waste
bytes (dev.cc.N.txq.M.kern_tls_full).
- Mbufs that send a short TLS record that ends before the end of the
trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC,
the encryption must always start at the beginning, so if the mbuf
starts at an offset into the TLS record, the offset bytes will be
"waste" bytes. For sockets using AES-GCM, the encryption can start
at the 16 byte block before the starting offset capping the waste at
15 bytes.
- Mbufs that send a partial TLS record that has a non-zero starting
offset but ends at the end of the trailer
(dev.cc.N.txq.M.kern_tls_partial). In order to compute the
authentication hash stored in the trailer, the entire TLS record
must be sent as input to the crypto engine, so the bytes before the
offset are always "waste" bytes.
In addition, other per-txq sysctls are provided:
- dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq
using AES-CBC.
- dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq
using AES-GCM.
- dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to
compensate for the TOE engine not being able to set FIN on the last
segment of a TLS record if the TLS record mbuf had FIN set.
- dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this
txq including full, short, and partial records.
- dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header
and payload) sent for TLS record requests.
- dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS
record requests.
To enable NIC KTLS with T6, set the following tunables prior to
loading the cxgbe(4) driver:
hw.cxgbe.config_file=kern_tls
hw.cxgbe.kern_tls=1
Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
|
|
|
#include <sys/counter.h>
|
2011-02-18 08:00:26 +00:00
|
|
|
#include <sys/rman.h>
|
|
|
|
#include <sys/types.h>
|
2013-10-26 17:58:36 +00:00
|
|
|
#include <sys/lock.h>
|
2011-02-18 08:00:26 +00:00
|
|
|
#include <sys/malloc.h>
|
2013-10-26 17:58:36 +00:00
|
|
|
#include <sys/rwlock.h>
|
|
|
|
#include <sys/sx.h>
|
2018-11-15 23:00:30 +00:00
|
|
|
#include <sys/vmem.h>
|
2013-10-26 17:58:36 +00:00
|
|
|
#include <vm/uma.h>
|
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
#include <dev/pci/pcivar.h>
|
|
|
|
#include <dev/pci/pcireg.h>
|
|
|
|
#include <machine/bus.h>
|
|
|
|
#include <sys/socket.h>
|
|
|
|
#include <sys/sysctl.h>
|
2021-05-23 21:58:29 +00:00
|
|
|
#include <sys/taskqueue.h>
|
2011-02-18 08:00:26 +00:00
|
|
|
#include <net/ethernet.h>
|
|
|
|
#include <net/if.h>
|
2013-10-26 17:58:36 +00:00
|
|
|
#include <net/if_var.h>
|
2011-02-18 08:00:26 +00:00
|
|
|
#include <net/if_media.h>
|
2020-02-04 01:09:02 +00:00
|
|
|
#include <net/pfil.h>
|
2012-05-24 23:03:23 +00:00
|
|
|
#include <netinet/in.h>
|
2011-02-18 08:00:26 +00:00
|
|
|
#include <netinet/tcp_lro.h>
|
|
|
|
|
|
|
|
#include "offload.h"
|
2016-06-07 00:27:55 +00:00
|
|
|
#include "t4_ioctl.h"
|
2014-05-27 18:18:41 +00:00
|
|
|
#include "common/t4_msg.h"
|
2011-12-16 02:09:51 +00:00
|
|
|
#include "firmware/t4fw_interface.h"
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2014-12-12 21:54:59 +00:00
|
|
|
#define KTR_CXGBE KTR_SPARE3
|
2011-02-18 08:00:26 +00:00
|
|
|
MALLOC_DECLARE(M_CXGBE);
|
|
|
|
#define CXGBE_UNIMPLEMENTED(s) \
|
|
|
|
panic("%s (%s, line %d) not implemented yet.", s, __FILE__, __LINE__)
|
|
|
|
|
2021-05-23 21:58:29 +00:00
|
|
|
/*
|
|
|
|
* Same as LIST_HEAD from queue.h. This is to avoid conflict with LinuxKPI's
|
|
|
|
* LIST_HEAD when building iw_cxgbe.
|
|
|
|
*/
|
|
|
|
#define CXGBE_LIST_HEAD(name, type) \
|
|
|
|
struct name { \
|
|
|
|
struct type *lh_first; /* first element */ \
|
|
|
|
}
|
|
|
|
|
2012-02-07 06:21:59 +00:00
|
|
|
#ifndef SYSCTL_ADD_UQUAD
|
|
|
|
#define SYSCTL_ADD_UQUAD SYSCTL_ADD_QUAD
|
|
|
|
#define sysctl_handle_64 sysctl_handle_quad
|
|
|
|
#define CTLTYPE_U64 CTLTYPE_QUAD
|
|
|
|
#endif
|
|
|
|
|
2018-11-27 22:02:54 +00:00
|
|
|
SYSCTL_DECL(_hw_cxgbe);
|
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
struct adapter;
|
|
|
|
typedef struct adapter adapter_t;
|
|
|
|
|
|
|
|
enum {
|
2014-08-02 00:56:34 +00:00
|
|
|
/*
|
|
|
|
* All ingress queues use this entry size. Note that the firmware event
|
|
|
|
* queue and any iq expecting CPL_RX_PKT in the descriptor needs this to
|
|
|
|
* be at least 64.
|
|
|
|
*/
|
|
|
|
IQ_ESIZE = 64,
|
|
|
|
|
|
|
|
/* Default queue sizes for all kinds of ingress queues */
|
2011-02-18 08:00:26 +00:00
|
|
|
FW_IQ_QSIZE = 256,
|
|
|
|
RX_IQ_QSIZE = 1024,
|
|
|
|
|
2014-08-02 00:56:34 +00:00
|
|
|
/* All egress queues use this entry size */
|
|
|
|
EQ_ESIZE = 64,
|
|
|
|
|
|
|
|
/* Default queue sizes for all kinds of egress queues */
|
2018-08-11 21:10:08 +00:00
|
|
|
CTRL_EQ_QSIZE = 1024,
|
2014-08-02 00:56:34 +00:00
|
|
|
TX_EQ_QSIZE = 1024,
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2011-03-08 03:04:07 +00:00
|
|
|
#if MJUMPAGESIZE != MCLBYTES
|
2014-03-18 20:14:13 +00:00
|
|
|
SW_ZONE_SIZES = 4, /* cluster, jumbop, jumbo9k, jumbo16k */
|
2011-03-08 03:04:07 +00:00
|
|
|
#else
|
2014-03-18 20:14:13 +00:00
|
|
|
SW_ZONE_SIZES = 3, /* cluster, jumbo9k, jumbo16k */
|
2011-03-08 03:04:07 +00:00
|
|
|
#endif
|
2014-12-06 01:47:38 +00:00
|
|
|
CL_METADATA_SIZE = CACHE_LINE_SIZE,
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2014-08-02 00:56:34 +00:00
|
|
|
SGE_MAX_WR_NDESC = SGE_MAX_WR_LEN / EQ_ESIZE, /* max WR size in desc */
|
2014-12-31 23:19:16 +00:00
|
|
|
TX_SGL_SEGS = 39,
|
|
|
|
TX_SGL_SEGS_TSO = 38,
|
2020-09-22 04:16:40 +00:00
|
|
|
TX_SGL_SEGS_VM = 38,
|
|
|
|
TX_SGL_SEGS_VM_TSO = 37,
|
2018-10-22 23:57:59 +00:00
|
|
|
TX_SGL_SEGS_EO_TSO = 30, /* XXX: lower for IPv6. */
|
2020-09-18 03:01:47 +00:00
|
|
|
TX_SGL_SEGS_VXLAN_TSO = 37,
|
2011-02-18 08:00:26 +00:00
|
|
|
TX_WR_FLITS = SGE_MAX_WR_LEN / 8
|
|
|
|
};
|
|
|
|
|
2011-03-24 01:03:01 +00:00
|
|
|
enum {
|
|
|
|
/* adapter intr_type */
|
|
|
|
INTR_INTX = (1 << 0),
|
|
|
|
INTR_MSI = (1 << 1),
|
|
|
|
INTR_MSIX = (1 << 2)
|
|
|
|
};
|
|
|
|
|
2014-05-27 18:18:41 +00:00
|
|
|
enum {
|
|
|
|
XGMAC_MTU = (1 << 0),
|
|
|
|
XGMAC_PROMISC = (1 << 1),
|
|
|
|
XGMAC_ALLMULTI = (1 << 2),
|
|
|
|
XGMAC_VLANEX = (1 << 3),
|
|
|
|
XGMAC_UCADDR = (1 << 4),
|
|
|
|
XGMAC_MCADDRS = (1 << 5),
|
|
|
|
|
|
|
|
XGMAC_ALL = 0xffff
|
|
|
|
};
|
|
|
|
|
2013-01-10 23:56:50 +00:00
|
|
|
enum {
|
|
|
|
/* flags understood by begin_synchronized_op */
|
|
|
|
HOLD_LOCK = (1 << 0),
|
|
|
|
SLEEP_OK = (1 << 1),
|
|
|
|
INTR_OK = (1 << 2),
|
|
|
|
|
|
|
|
/* flags understood by end_synchronized_op */
|
|
|
|
LOCK_HELD = HOLD_LOCK,
|
|
|
|
};
|
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
enum {
|
2022-02-04 21:16:35 +00:00
|
|
|
/* adapter flags. synch_op or adapter_lock. */
|
2011-02-18 08:00:26 +00:00
|
|
|
FULL_INIT_DONE = (1 << 0),
|
|
|
|
FW_OK = (1 << 1),
|
2017-08-28 03:13:16 +00:00
|
|
|
CHK_MBOX_ACCESS = (1 << 2),
|
2011-12-16 02:09:51 +00:00
|
|
|
MASTER_PF = (1 << 3),
|
2013-08-30 01:45:36 +00:00
|
|
|
BUF_PACKING_OK = (1 << 6),
|
2016-08-08 21:45:39 +00:00
|
|
|
IS_VF = (1 << 7),
|
2021-03-24 01:01:01 +00:00
|
|
|
KERN_TLS_ON = (1 << 8), /* HW is configured for KERN_TLS */
|
2011-02-18 08:00:26 +00:00
|
|
|
CXGBE_BUSY = (1 << 9),
|
2022-02-04 21:16:35 +00:00
|
|
|
|
|
|
|
/* adapter error_flags. reg_lock for HW_OFF_LIMITS, atomics for the rest. */
|
|
|
|
ADAP_STOPPED = (1 << 0), /* Adapter has been stopped. */
|
|
|
|
ADAP_FATAL_ERR = (1 << 1), /* Encountered a fatal error. */
|
|
|
|
HW_OFF_LIMITS = (1 << 2), /* off limits to all except reset_thread */
|
|
|
|
ADAP_CIM_ERR = (1 << 3), /* Error was related to FW/CIM. */
|
2011-02-18 08:00:26 +00:00
|
|
|
|
|
|
|
/* port flags */
|
Add support for packet-sniffing tracers to cxgbe(4). This works with
all T4 and T5 based cards and is useful for analyzing TSO, LRO, TOE, and
for general purpose monitoring without tapping any cxgbe or cxl ifnet
directly.
Tracers on the T4/T5 chips provide access to Ethernet frames exactly as
they were received from or transmitted on the wire. On transmit, a
tracer will capture a frame after TSO segmentation, hw VLAN tag
insertion, hw L3 & L4 checksum insertion, etc. It will also capture
frames generated by the TCP offload engine (TOE traffic is normally
invisible to the kernel). On receive, a tracer will capture a frame
before hw VLAN extraction, runt filtering, other badness filtering,
before the steering/drop/L2-rewrite filters or the TOE have had a go at
it, and of course before sw LRO in the driver.
There are 4 tracers on a chip. A tracer can trace only in one direction
(tx or rx). For now cxgbetool will set up tracers to capture the first
128B of every transmitted or received frame on a given port. This is a
small subset of what the hardware can do. A pseudo ifnet with the same
name as the nexus driver (t4nex0 or t5nex0) will be created for tracing.
The data delivered to this ifnet is an additional copy made inside the
chip. Normal delivery to cxgbe<n> or cxl<n> will be made as usual.
/* watch cxl0, which is the first port hanging off t5nex0. */
# cxgbetool t5nex0 tracer 0 tx0 (watch what cxl0 is transmitting)
# cxgbetool t5nex0 tracer 1 rx0 (watch what cxl0 is receiving)
# cxgbetool t5nex0 tracer list
# tcpdump -i t5nex0 <== all that cxl0 sees and puts on the wire
If you were doing TSO, a tcpdump on cxl0 may have shown you ~64K
"frames" with no L3/L4 checksum but this will show you the frames that
were actually transmitted.
/* all done */
# cxgbetool t5nex0 tracer 0 disable
# cxgbetool t5nex0 tracer 1 disable
# cxgbetool t5nex0 tracer list
# ifconfig t5nex0 destroy
2013-07-26 22:04:11 +00:00
|
|
|
HAS_TRACEQ = (1 << 3),
|
2018-05-30 22:36:09 +00:00
|
|
|
FIXED_IFMEDIA = (1 << 4), /* ifmedia list doesn't change. */
|
2015-12-03 00:02:01 +00:00
|
|
|
|
|
|
|
/* VI flags */
|
|
|
|
DOOMED = (1 << 0),
|
|
|
|
VI_INIT_DONE = (1 << 1),
|
2022-01-13 22:21:49 +00:00
|
|
|
/* 1 << 2 is unused, was VI_SYSCTL_CTX */
|
2020-09-22 04:16:40 +00:00
|
|
|
TX_USES_VM_WR = (1 << 3),
|
2021-04-23 00:45:52 +00:00
|
|
|
VI_SKIP_STATS = (1 << 4),
|
2015-06-16 12:36:29 +00:00
|
|
|
|
|
|
|
/* adapter debug_flags */
|
2017-08-30 23:41:04 +00:00
|
|
|
DF_DUMP_MBOX = (1 << 0), /* Log all mbox cmd/rpl. */
|
|
|
|
DF_LOAD_FW_ANYTIME = (1 << 1), /* Allow LOAD_FW after init */
|
|
|
|
DF_DISABLE_TCB_CACHE = (1 << 2), /* Disable TCB cache (T6+) */
|
2018-12-06 06:18:21 +00:00
|
|
|
DF_DISABLE_CFG_RETRY = (1 << 3), /* Disable fallback config */
|
2019-02-01 20:42:49 +00:00
|
|
|
DF_VERBOSE_SLOWINTR = (1 << 4), /* Chatty slow intr handler */
|
2011-02-18 08:00:26 +00:00
|
|
|
};
|
|
|
|
|
2015-12-03 00:02:01 +00:00
|
|
|
#define IS_DOOMED(vi) ((vi)->flags & DOOMED)
|
|
|
|
#define SET_DOOMED(vi) do {(vi)->flags |= DOOMED;} while (0)
|
2013-01-10 23:56:50 +00:00
|
|
|
#define IS_BUSY(sc) ((sc)->flags & CXGBE_BUSY)
|
|
|
|
#define SET_BUSY(sc) do {(sc)->flags |= CXGBE_BUSY;} while (0)
|
|
|
|
#define CLR_BUSY(sc) do {(sc)->flags &= ~CXGBE_BUSY;} while (0)
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2015-12-03 00:02:01 +00:00
|
|
|
struct vi_info {
|
2011-02-18 08:00:26 +00:00
|
|
|
device_t dev;
|
2015-12-03 00:02:01 +00:00
|
|
|
struct port_info *pi;
|
2020-06-25 17:04:22 +00:00
|
|
|
struct adapter *adapter;
|
2011-02-18 08:00:26 +00:00
|
|
|
|
|
|
|
struct ifnet *ifp;
|
2020-02-04 01:09:02 +00:00
|
|
|
struct pfil_head *pfil;
|
2011-02-18 08:00:26 +00:00
|
|
|
|
|
|
|
unsigned long flags;
|
|
|
|
int if_flags;
|
|
|
|
|
2016-06-23 02:53:00 +00:00
|
|
|
uint16_t *rss, *nm_rss;
|
2019-03-20 17:27:11 +00:00
|
|
|
uint16_t viid; /* opaque VI identifier */
|
|
|
|
uint16_t smt_idx;
|
|
|
|
uint16_t vin;
|
|
|
|
uint8_t vfvld;
|
2011-02-18 08:00:26 +00:00
|
|
|
int16_t xact_addr_filt;/* index of exact MAC address filter */
|
|
|
|
uint16_t rss_size; /* size of VI's RSS table slice */
|
2015-07-17 06:46:18 +00:00
|
|
|
uint16_t rss_base; /* start of VI's RSS table slice */
|
2018-10-25 06:24:42 +00:00
|
|
|
int hashen;
|
2015-12-03 00:02:01 +00:00
|
|
|
|
|
|
|
int nintr;
|
|
|
|
int first_intr;
|
2011-02-18 08:00:26 +00:00
|
|
|
|
|
|
|
/* These need to be int as they are used in sysctl */
|
2017-05-10 00:42:28 +00:00
|
|
|
int ntxq; /* # of tx queues */
|
|
|
|
int first_txq; /* index of first tx queue */
|
|
|
|
int rsrv_noflowq; /* Reserve queue 0 for non-flowid packets */
|
|
|
|
int nrxq; /* # of rx queues */
|
|
|
|
int first_rxq; /* index of first rx queue */
|
2011-12-16 02:09:51 +00:00
|
|
|
int nofldtxq; /* # of offload tx queues */
|
|
|
|
int first_ofld_txq; /* index of first offload tx queue */
|
|
|
|
int nofldrxq; /* # of offload rx queues */
|
|
|
|
int first_ofld_rxq; /* index of first offload rx queue */
|
2016-06-23 02:53:00 +00:00
|
|
|
int nnmtxq;
|
|
|
|
int first_nm_txq;
|
|
|
|
int nnmrxq;
|
|
|
|
int first_nm_rxq;
|
2011-02-18 08:00:26 +00:00
|
|
|
int tmr_idx;
|
2017-10-05 07:18:16 +00:00
|
|
|
int ofld_tmr_idx;
|
2011-02-18 08:00:26 +00:00
|
|
|
int pktc_idx;
|
2017-10-05 07:18:16 +00:00
|
|
|
int ofld_pktc_idx;
|
2011-02-18 08:00:26 +00:00
|
|
|
int qsize_rxq;
|
|
|
|
int qsize_txq;
|
|
|
|
|
2015-12-03 00:02:01 +00:00
|
|
|
struct timeval last_refreshed;
|
|
|
|
struct fw_vi_stats_vf stats;
|
2021-04-01 07:57:19 +00:00
|
|
|
struct mtx tick_mtx;
|
2015-12-03 00:02:01 +00:00
|
|
|
struct callout tick;
|
2021-04-01 07:57:19 +00:00
|
|
|
|
2021-04-26 21:03:06 +00:00
|
|
|
struct sysctl_ctx_list ctx;
|
|
|
|
struct sysctl_oid *rxq_oid;
|
|
|
|
struct sysctl_oid *txq_oid;
|
|
|
|
struct sysctl_oid *nm_rxq_oid;
|
|
|
|
struct sysctl_oid *nm_txq_oid;
|
|
|
|
struct sysctl_oid *ofld_rxq_oid;
|
|
|
|
struct sysctl_oid *ofld_txq_oid;
|
2015-12-03 00:02:01 +00:00
|
|
|
|
|
|
|
uint8_t hw_addr[ETHER_ADDR_LEN]; /* factory MAC address, won't change */
|
|
|
|
};
|
|
|
|
|
2017-05-02 20:38:10 +00:00
|
|
|
struct tx_ch_rl_params {
|
|
|
|
enum fw_sched_params_rate ratemode; /* %port (REL) or kbps (ABS) */
|
|
|
|
uint32_t maxrate;
|
|
|
|
};
|
|
|
|
|
2021-06-24 20:05:57 +00:00
|
|
|
/* CLRL state */
|
|
|
|
enum clrl_state {
|
|
|
|
CS_UNINITIALIZED = 0,
|
|
|
|
CS_PARAMS_SET, /* sw parameters have been set. */
|
|
|
|
CS_HW_UPDATE_REQUESTED, /* async HW update requested. */
|
|
|
|
CS_HW_UPDATE_IN_PROGRESS, /* sync hw update in progress. */
|
|
|
|
CS_HW_CONFIGURED /* configured in the hardware. */
|
|
|
|
};
|
|
|
|
|
|
|
|
/* CLRL flags */
|
2016-06-07 00:27:55 +00:00
|
|
|
enum {
|
2021-06-24 20:05:57 +00:00
|
|
|
CF_USER = (1 << 0), /* was configured by driver ioctl. */
|
2016-06-07 00:27:55 +00:00
|
|
|
};
|
|
|
|
|
2017-05-02 20:38:10 +00:00
|
|
|
struct tx_cl_rl_params {
|
2021-06-24 20:05:57 +00:00
|
|
|
enum clrl_state state;
|
2016-06-07 00:27:55 +00:00
|
|
|
int refcount;
|
2018-08-06 23:21:13 +00:00
|
|
|
uint8_t flags;
|
2017-05-02 20:38:10 +00:00
|
|
|
enum fw_sched_params_rate ratemode; /* %port REL or ABS value */
|
|
|
|
enum fw_sched_params_unit rateunit; /* kbps or pps (when ABS) */
|
|
|
|
enum fw_sched_params_mode mode; /* aggr or per-flow */
|
|
|
|
uint32_t maxrate;
|
|
|
|
uint16_t pktsize;
|
2018-08-07 22:13:03 +00:00
|
|
|
uint16_t burstsize;
|
2017-05-02 20:38:10 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
/* Tx scheduler parameters for a channel/port */
|
|
|
|
struct tx_sched_params {
|
|
|
|
/* Channel Rate Limiter */
|
|
|
|
struct tx_ch_rl_params ch_rl;
|
|
|
|
|
|
|
|
/* Class WRR */
|
|
|
|
/* XXX */
|
|
|
|
|
2018-08-07 22:13:03 +00:00
|
|
|
/* Class Rate Limiter (including the default pktsize and burstsize). */
|
|
|
|
int pktsize;
|
|
|
|
int burstsize;
|
2017-05-02 20:38:10 +00:00
|
|
|
struct tx_cl_rl_params cl_rl[];
|
2016-06-07 00:27:55 +00:00
|
|
|
};
|
|
|
|
|
2015-12-03 00:02:01 +00:00
|
|
|
struct port_info {
|
|
|
|
device_t dev;
|
|
|
|
struct adapter *adapter;
|
|
|
|
|
|
|
|
struct vi_info *vi;
|
|
|
|
int nvi;
|
|
|
|
int up_vis;
|
|
|
|
int uld_vis;
|
2020-09-18 03:01:47 +00:00
|
|
|
bool vxlan_tcam_entry;
|
2015-12-03 00:02:01 +00:00
|
|
|
|
2017-05-02 20:38:10 +00:00
|
|
|
struct tx_sched_params *sched_params;
|
2016-06-07 00:27:55 +00:00
|
|
|
|
2015-12-03 00:02:01 +00:00
|
|
|
struct mtx pi_lock;
|
|
|
|
char lockname[16];
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
uint8_t lport; /* associated offload logical port */
|
|
|
|
int8_t mdio_addr;
|
|
|
|
uint8_t port_type;
|
|
|
|
uint8_t mod_type;
|
|
|
|
uint8_t port_id;
|
|
|
|
uint8_t tx_chan;
|
2017-10-24 05:41:48 +00:00
|
|
|
uint8_t mps_bg_map; /* rx MPS buffer group bitmap */
|
|
|
|
uint8_t rx_e_chan_map; /* rx TP e-channel bitmap */
|
2021-02-26 02:10:52 +00:00
|
|
|
uint8_t rx_c_chan; /* rx TP c-channel */
|
2015-12-03 00:02:01 +00:00
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
struct link_config link_cfg;
|
2017-08-28 21:44:25 +00:00
|
|
|
struct ifmedia media;
|
2014-09-27 05:50:31 +00:00
|
|
|
|
|
|
|
struct port_stats stats;
|
|
|
|
u_int tnl_cong_drops;
|
2014-12-31 23:19:16 +00:00
|
|
|
u_int tx_parse_error;
|
2020-10-09 22:23:39 +00:00
|
|
|
int fcs_reg;
|
|
|
|
uint64_t fcs_base;
|
2022-01-13 22:21:49 +00:00
|
|
|
|
|
|
|
struct sysctl_ctx_list ctx;
|
2011-02-18 08:00:26 +00:00
|
|
|
};
|
|
|
|
|
2015-12-03 00:02:01 +00:00
|
|
|
#define IS_MAIN_VI(vi) ((vi) == &((vi)->pi->vi[0]))
|
|
|
|
|
2014-03-18 20:14:13 +00:00
|
|
|
struct cluster_metadata {
|
2020-02-03 23:50:29 +00:00
|
|
|
uma_zone_t zone;
|
|
|
|
caddr_t cl;
|
2014-03-18 20:14:13 +00:00
|
|
|
u_int refcount;
|
2011-02-18 08:00:26 +00:00
|
|
|
};
|
|
|
|
|
2014-03-18 20:14:13 +00:00
|
|
|
struct fl_sdesc {
|
|
|
|
caddr_t cl;
|
2014-07-22 02:02:39 +00:00
|
|
|
uint16_t nmbuf; /* # of driver originated mbufs with ref on cluster */
|
2020-02-04 00:51:10 +00:00
|
|
|
int16_t moff; /* offset of metadata from cl */
|
|
|
|
uint8_t zidx;
|
2014-03-18 20:14:13 +00:00
|
|
|
};
|
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
struct tx_desc {
|
|
|
|
__be64 flit[8];
|
|
|
|
};
|
|
|
|
|
|
|
|
struct tx_sdesc {
|
2014-12-31 23:19:16 +00:00
|
|
|
struct mbuf *m; /* m_nextpkt linked chain of frames */
|
2011-02-18 08:00:26 +00:00
|
|
|
uint8_t desc_used; /* # of hardware descriptors used by the WR */
|
|
|
|
};
|
|
|
|
|
2014-08-02 00:56:34 +00:00
|
|
|
|
|
|
|
#define IQ_PAD (IQ_ESIZE - sizeof(struct rsp_ctrl) - sizeof(struct rss_header))
|
|
|
|
struct iq_desc {
|
|
|
|
struct rss_header rss;
|
|
|
|
uint8_t cpl[IQ_PAD];
|
|
|
|
struct rsp_ctrl rsp;
|
|
|
|
};
|
|
|
|
#undef IQ_PAD
|
|
|
|
CTASSERT(sizeof(struct iq_desc) == IQ_ESIZE);
|
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
enum {
|
|
|
|
/* iq flags */
|
2021-04-26 21:03:06 +00:00
|
|
|
IQ_SW_ALLOCATED = (1 << 0), /* sw resources allocated */
|
2011-12-16 02:09:51 +00:00
|
|
|
IQ_HAS_FL = (1 << 1), /* iq associated with a freelist */
|
2018-08-18 04:23:51 +00:00
|
|
|
IQ_RX_TIMESTAMP = (1 << 2), /* provide the SGE rx timestamp */
|
2011-12-16 02:09:51 +00:00
|
|
|
IQ_LRO_ENABLED = (1 << 3), /* iq is an eth rxq with LRO enabled */
|
2017-04-17 09:00:20 +00:00
|
|
|
IQ_ADJ_CREDIT = (1 << 4), /* hw is off by 1 credit for this iq */
|
2021-04-26 21:03:06 +00:00
|
|
|
IQ_HW_ALLOCATED = (1 << 5), /* fw/hw resources allocated */
|
2011-04-15 03:09:27 +00:00
|
|
|
|
|
|
|
/* iq state */
|
|
|
|
IQS_DISABLED = 0,
|
|
|
|
IQS_BUSY = 1,
|
|
|
|
IQS_IDLE = 2,
|
2016-06-23 02:53:00 +00:00
|
|
|
|
|
|
|
/* netmap related flags */
|
|
|
|
NM_OFF = 0,
|
|
|
|
NM_ON = 1,
|
|
|
|
NM_BUSY = 2,
|
2011-02-18 08:00:26 +00:00
|
|
|
};
|
|
|
|
|
2018-04-30 15:18:38 +00:00
|
|
|
enum {
|
|
|
|
CPL_COOKIE_RESERVED = 0,
|
|
|
|
CPL_COOKIE_FILTER,
|
|
|
|
CPL_COOKIE_DDP0,
|
|
|
|
CPL_COOKIE_DDP1,
|
|
|
|
CPL_COOKIE_TOM,
|
cxgbe(4): Add support for hash filters.
These filters reside in the card's memory instead of its TCAM and can be
configured via a new "hashfilter" subcommand in cxgbetool. Hash and
normal TCAM filters can be used together. The hardware does an
exact-match of packet fields for hash filters, unlike the masked match
performed for TCAM filters. Any T5/T6 card with memory can support at
least half a million hash filters. The sample config file with the
driver configures 512K of these, it is possible to double this to 1
million+ in some cases.
The chip does an exact-match of fields of incoming datagrams with hash
filters and performs the action configured for the filter if it matches.
The fields to match are specified in a "filter mask" in the firmware
config file. The filter mask always includes the 5-tuple (sip, dip,
sport, dport, ipproto). It can, optionally, also include any subset of
the filter mode (see filterMode and filterMask in the firmware config
file).
For example:
filterMode = fragmentation, mpshittype, protocol, vlan, port, fcoe
filterMask = protocol, port, vlan
Exact values of the 5-tuple, the physical port, and VLAN tag would have
to be provided while setting up a hash filter with the chip
configuration above.
Hash filters support all actions supported by TCAM filters. A packet
that hits a hash filter can be dropped, let through (with optional
steering to a specific queue or RSS region), switched out of another
port (with optional L2 rewrite of DMAC, SMAC, VLAN tag), or get NAT'ed.
(Support for some of these will show up in the driver in a follow-up
commit very shortly).
Sponsored by: Chelsio Communications
2018-05-09 04:09:49 +00:00
|
|
|
CPL_COOKIE_HASHFILTER,
|
2018-05-24 08:21:43 +00:00
|
|
|
CPL_COOKIE_ETHOFLD,
|
NIC KTLS for Chelsio T6 adapters.
This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters.
Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE
connections.
NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment
the encrypted TLS frames output by the crypto engine. Instead, the
TOE is placed into a special setup to permit "dummy" connections to be
associated with regular sockets using KTLS. This permits using the
TOE to segment the encrypted TLS records. However, this approach does
have some limitations:
1) Regular TOE sockets cannot be used when the TOE is in this special
mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but
not both at the same time.
2) In NIC KTLS mode, the TOE is only able to accept a per-connection
timestamp offset that varies in the upper 4 bits. Put another way,
only connections whose timestamp offset has the 28 lower bits
cleared can use NIC KTLS and generate correct timestamps. The
driver will refuse to enable NIC KTLS on connections with a
timestamp offset with any of the lower 28 bits set. To use NIC
KTLS, users can either disable TCP timestamps by setting the
net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the
tcp_new_ts_offset() function to clear the lower 28 bits of the
generated offset.
3) Because the TCP segmentation relies on fields mirrored in a TCB in
the TOE, not all fields in a TCP packet can be sent in the TCP
segments generated from a TLS record. Specifically, for packets
containing TCP options other than timestamps, the driver will
inject an "empty" TCP packet holding the requested options (e.g. a
SACK scoreboard) along with the segments from the TLS record.
These empty TCP packets are counted by the
dev.cc.N.txq.M.kern_tls_options sysctls.
Unlike TOE TLS which is able to buffer encrypted TLS records in
on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS
records for retransmit requests as well as non-retransmit requests
that do not include the start of a TLS record but do include the
trailer. The T6 NIC KTLS code tries to optimize some of the cases for
requests to transmit partial TLS records. In particular it attempts
to minimize sending "waste" bytes that have to be given as input to
the crypto engine but are not needed on the wire to satisfy mbufs sent
from the TCP stack down to the driver.
TCP packets for TLS requests are broken down into the following
classes (with associated counters):
- Mbufs that send an entire TLS record in full do not have any waste
bytes (dev.cc.N.txq.M.kern_tls_full).
- Mbufs that send a short TLS record that ends before the end of the
trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC,
the encryption must always start at the beginning, so if the mbuf
starts at an offset into the TLS record, the offset bytes will be
"waste" bytes. For sockets using AES-GCM, the encryption can start
at the 16 byte block before the starting offset capping the waste at
15 bytes.
- Mbufs that send a partial TLS record that has a non-zero starting
offset but ends at the end of the trailer
(dev.cc.N.txq.M.kern_tls_partial). In order to compute the
authentication hash stored in the trailer, the entire TLS record
must be sent as input to the crypto engine, so the bytes before the
offset are always "waste" bytes.
In addition, other per-txq sysctls are provided:
- dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq
using AES-CBC.
- dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq
using AES-GCM.
- dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to
compensate for the TOE engine not being able to set FIN on the last
segment of a TLS record if the TLS record mbuf had FIN set.
- dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this
txq including full, short, and partial records.
- dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header
and payload) sent for TLS record requests.
- dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS
record requests.
To enable NIC KTLS with T6, set the following tunables prior to
loading the cxgbe(4) driver:
hw.cxgbe.config_file=kern_tls
hw.cxgbe.kern_tls=1
Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
|
|
|
CPL_COOKIE_KERN_TLS,
|
2018-04-30 15:18:38 +00:00
|
|
|
|
|
|
|
NUM_CPL_COOKIES = 8 /* Limited by M_COOKIE. Do not increase. */
|
|
|
|
};
|
|
|
|
|
2016-07-05 01:29:24 +00:00
|
|
|
struct sge_iq;
|
|
|
|
struct rss_header;
|
|
|
|
typedef int (*cpl_handler_t)(struct sge_iq *, const struct rss_header *,
|
|
|
|
struct mbuf *);
|
|
|
|
typedef int (*an_handler_t)(struct sge_iq *, const struct rsp_ctrl *);
|
|
|
|
typedef int (*fw_msg_handler_t)(struct adapter *, const __be64 *);
|
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
/*
|
|
|
|
* Ingress Queue: T4 is producer, driver is consumer.
|
|
|
|
*/
|
|
|
|
struct sge_iq {
|
2011-03-05 03:42:03 +00:00
|
|
|
uint32_t flags;
|
2011-12-16 02:09:51 +00:00
|
|
|
volatile int state;
|
2011-03-05 03:42:03 +00:00
|
|
|
struct adapter *adapter;
|
2014-08-02 00:56:34 +00:00
|
|
|
struct iq_desc *desc; /* KVA of descriptor ring */
|
|
|
|
int8_t intr_pktc_idx; /* packet count threshold index */
|
2011-02-18 08:00:26 +00:00
|
|
|
uint8_t gen; /* generation bit */
|
|
|
|
uint8_t intr_params; /* interrupt holdoff parameters */
|
2021-04-26 21:03:06 +00:00
|
|
|
int8_t cong; /* congestion settings */
|
2011-02-18 08:00:26 +00:00
|
|
|
uint16_t qsize; /* size (# of entries) of the queue */
|
2014-08-02 00:56:34 +00:00
|
|
|
uint16_t sidx; /* index of the entry with the status page */
|
2011-02-18 08:00:26 +00:00
|
|
|
uint16_t cidx; /* consumer index */
|
2011-12-16 02:09:51 +00:00
|
|
|
uint16_t cntxt_id; /* SGE context id for the iq */
|
2014-08-02 00:56:34 +00:00
|
|
|
uint16_t abs_id; /* absolute SGE id for the iq */
|
2021-04-26 21:03:06 +00:00
|
|
|
int16_t intr_idx; /* interrupt used by the queue */
|
2011-12-16 02:09:51 +00:00
|
|
|
|
|
|
|
STAILQ_ENTRY(sge_iq) link;
|
2014-08-02 00:56:34 +00:00
|
|
|
|
|
|
|
bus_dma_tag_t desc_tag;
|
|
|
|
bus_dmamap_t desc_map;
|
|
|
|
bus_addr_t ba; /* bus address of descriptor ring */
|
2011-02-18 08:00:26 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
enum {
|
2021-04-26 21:03:06 +00:00
|
|
|
/* eq type */
|
2011-12-16 02:09:51 +00:00
|
|
|
EQ_CTRL = 1,
|
|
|
|
EQ_ETH = 2,
|
|
|
|
EQ_OFLD = 3,
|
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
/* eq flags */
|
2021-04-26 21:03:06 +00:00
|
|
|
EQ_SW_ALLOCATED = (1 << 0), /* sw resources allocated */
|
|
|
|
EQ_HW_ALLOCATED = (1 << 1), /* hw/fw resources allocated */
|
2014-12-31 23:19:16 +00:00
|
|
|
EQ_ENABLED = (1 << 3), /* open for business */
|
2017-05-09 18:33:41 +00:00
|
|
|
EQ_QFLUSH = (1 << 4), /* if_qflush in progress */
|
2011-02-18 08:00:26 +00:00
|
|
|
};
|
|
|
|
|
2013-03-30 02:26:20 +00:00
|
|
|
/* Listed in order of preference. Update t4_sysctls too if you change these */
|
2013-04-11 22:49:29 +00:00
|
|
|
enum {DOORBELL_UDB, DOORBELL_WCWR, DOORBELL_UDBWC, DOORBELL_KDB};
|
2013-03-30 02:26:20 +00:00
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
/*
|
|
|
|
* Egress Queue: driver is producer, T4 is consumer.
|
|
|
|
*
|
|
|
|
* Note: A free list is an egress queue (driver produces the buffers and T4
|
|
|
|
* consumes them) but it's special enough to have its own struct (see sge_fl).
|
|
|
|
*/
|
|
|
|
struct sge_eq {
|
2011-12-16 02:09:51 +00:00
|
|
|
unsigned int flags; /* MUST be first */
|
|
|
|
unsigned int cntxt_id; /* SGE context id for the eq */
|
2016-08-09 17:49:42 +00:00
|
|
|
unsigned int abs_id; /* absolute SGE id for the eq */
|
2021-04-26 21:03:06 +00:00
|
|
|
uint8_t type; /* EQ_CTRL/EQ_ETH/EQ_OFLD */
|
|
|
|
uint8_t doorbells;
|
|
|
|
uint8_t tx_chan; /* tx channel used by the eq */
|
2011-02-18 08:00:26 +00:00
|
|
|
struct mtx eq_lock;
|
|
|
|
|
|
|
|
struct tx_desc *desc; /* KVA of descriptor ring */
|
2013-03-30 02:26:20 +00:00
|
|
|
volatile uint32_t *udb; /* KVA of doorbell (lies within BAR2) */
|
|
|
|
u_int udb_qid; /* relative qid within the doorbell page */
|
2014-12-31 23:19:16 +00:00
|
|
|
uint16_t sidx; /* index of the entry with the status page */
|
2011-02-18 08:00:26 +00:00
|
|
|
uint16_t cidx; /* consumer idx (desc idx) */
|
|
|
|
uint16_t pidx; /* producer idx (desc idx) */
|
2014-12-31 23:19:16 +00:00
|
|
|
uint16_t equeqidx; /* EQUEQ last requested at this pidx */
|
|
|
|
uint16_t dbidx; /* pidx of the most recent doorbell */
|
2021-04-26 21:03:06 +00:00
|
|
|
uint16_t iqid; /* cached iq->cntxt_id (see iq below) */
|
2014-12-31 23:19:16 +00:00
|
|
|
volatile u_int equiq; /* EQUIQ outstanding */
|
2021-04-26 21:03:06 +00:00
|
|
|
struct sge_iq *iq; /* iq that receives egr_update for the eq */
|
2011-12-16 02:09:51 +00:00
|
|
|
|
2014-12-31 23:19:16 +00:00
|
|
|
bus_dma_tag_t desc_tag;
|
|
|
|
bus_dmamap_t desc_map;
|
|
|
|
bus_addr_t ba; /* bus address of descriptor ring */
|
|
|
|
char lockname[16];
|
2011-12-16 02:09:51 +00:00
|
|
|
};
|
|
|
|
|
2020-02-04 00:51:10 +00:00
|
|
|
struct rx_buf_info {
|
2014-03-18 20:14:13 +00:00
|
|
|
uma_zone_t zone; /* zone that this cluster comes from */
|
2020-02-04 00:51:10 +00:00
|
|
|
uint16_t size1; /* same as size of cluster: 2K/4K/9K/16K.
|
|
|
|
* hwsize[hwidx1] = size1. No spare. */
|
|
|
|
uint16_t size2; /* hwsize[hwidx2] = size2.
|
|
|
|
* spare in cluster = size1 - size2. */
|
|
|
|
int8_t hwidx1; /* SGE bufsize idx for size1 */
|
|
|
|
int8_t hwidx2; /* SGE bufsize idx for size2 */
|
|
|
|
uint8_t type; /* EXT_xxx type of the cluster */
|
2013-08-30 01:45:36 +00:00
|
|
|
};
|
|
|
|
|
2016-03-10 06:15:31 +00:00
|
|
|
enum {
|
|
|
|
NUM_MEMWIN = 3,
|
|
|
|
|
|
|
|
MEMWIN0_APERTURE = 2048,
|
|
|
|
MEMWIN0_BASE = 0x1b800,
|
|
|
|
|
|
|
|
MEMWIN1_APERTURE = 32768,
|
|
|
|
MEMWIN1_BASE = 0x28000,
|
|
|
|
|
|
|
|
MEMWIN2_APERTURE_T4 = 65536,
|
|
|
|
MEMWIN2_BASE_T4 = 0x30000,
|
|
|
|
|
|
|
|
MEMWIN2_APERTURE_T5 = 128 * 1024,
|
|
|
|
MEMWIN2_BASE_T5 = 0x60000,
|
|
|
|
};
|
|
|
|
|
|
|
|
struct memwin {
|
|
|
|
struct rwlock mw_lock __aligned(CACHE_LINE_SIZE);
|
|
|
|
uint32_t mw_base; /* constant after setup_memwin */
|
|
|
|
uint32_t mw_aperture; /* ditto */
|
|
|
|
uint32_t mw_curpos; /* protected by mw_lock */
|
|
|
|
};
|
|
|
|
|
2011-12-16 02:09:51 +00:00
|
|
|
enum {
|
|
|
|
FL_STARVING = (1 << 0), /* on the adapter's list of starving fl's */
|
|
|
|
FL_DOOMED = (1 << 1), /* about to be destroyed */
|
2013-08-30 01:45:36 +00:00
|
|
|
FL_BUF_PACKING = (1 << 2), /* buffer packing enabled */
|
2014-08-02 06:55:36 +00:00
|
|
|
FL_BUF_RESUME = (1 << 3), /* resume from the middle of the frame */
|
2011-04-19 22:08:28 +00:00
|
|
|
};
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2014-08-02 06:55:36 +00:00
|
|
|
#define FL_RUNNING_LOW(fl) \
|
|
|
|
(IDXDIFF(fl->dbidx * 8, fl->cidx, fl->sidx * 8) <= fl->lowat)
|
|
|
|
#define FL_NOT_RUNNING_LOW(fl) \
|
|
|
|
(IDXDIFF(fl->dbidx * 8, fl->cidx, fl->sidx * 8) >= 2 * fl->lowat)
|
2011-12-16 02:09:51 +00:00
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
struct sge_fl {
|
|
|
|
struct mtx fl_lock;
|
|
|
|
__be64 *desc; /* KVA of descriptor ring, ptr to addresses */
|
|
|
|
struct fl_sdesc *sdesc; /* KVA of software descriptor ring */
|
2020-02-04 00:51:10 +00:00
|
|
|
uint16_t zidx; /* refill zone idx */
|
|
|
|
uint16_t safe_zidx;
|
2014-08-02 06:55:36 +00:00
|
|
|
uint16_t lowat; /* # of buffers <= this means fl needs help */
|
|
|
|
int flags;
|
|
|
|
uint16_t buf_boundary;
|
2014-03-18 20:14:13 +00:00
|
|
|
|
2014-08-02 06:55:36 +00:00
|
|
|
/* The 16b idx all deal with hw descriptors */
|
|
|
|
uint16_t dbidx; /* hw pidx after last doorbell */
|
|
|
|
uint16_t sidx; /* index of status page */
|
|
|
|
volatile uint16_t hw_cidx;
|
|
|
|
|
|
|
|
/* The 32b idx are all buffer idx, not hardware descriptor idx */
|
|
|
|
uint32_t cidx; /* consumer index */
|
|
|
|
uint32_t pidx; /* producer index */
|
|
|
|
|
|
|
|
uint32_t dbval;
|
|
|
|
u_int rx_offset; /* offset in fl buf (when buffer packing) */
|
|
|
|
volatile uint32_t *udb;
|
2014-03-18 20:14:13 +00:00
|
|
|
|
|
|
|
uint64_t cl_allocated; /* # of clusters allocated */
|
|
|
|
uint64_t cl_recycled; /* # of clusters recycled */
|
|
|
|
uint64_t cl_fast_recycled; /* # of clusters recycled (fast) */
|
2014-08-02 06:55:36 +00:00
|
|
|
|
|
|
|
/* These 3 are valid when FL_BUF_RESUME is set, stale otherwise. */
|
|
|
|
struct mbuf *m0;
|
|
|
|
struct mbuf **pnext;
|
|
|
|
u_int remaining;
|
|
|
|
|
|
|
|
uint16_t qsize; /* # of hw descriptors (status page included) */
|
|
|
|
uint16_t cntxt_id; /* SGE context id for the freelist */
|
|
|
|
TAILQ_ENTRY(sge_fl) link; /* All starving freelists */
|
|
|
|
bus_dma_tag_t desc_tag;
|
|
|
|
bus_dmamap_t desc_map;
|
|
|
|
char lockname[16];
|
|
|
|
bus_addr_t ba; /* bus address of descriptor ring */
|
2011-02-18 08:00:26 +00:00
|
|
|
};
|
|
|
|
|
2014-12-31 23:19:16 +00:00
|
|
|
struct mp_ring;
|
|
|
|
|
2020-07-03 04:44:23 +00:00
|
|
|
struct txpkts {
|
|
|
|
uint8_t wr_type; /* type 0 or type 1 */
|
|
|
|
uint8_t npkt; /* # of packets in this work request */
|
|
|
|
uint8_t len16; /* # of 16B pieces used by this work request */
|
2021-02-01 11:00:09 +00:00
|
|
|
uint8_t score;
|
2020-07-03 04:44:23 +00:00
|
|
|
uint8_t max_npkt; /* maximum number of packets allowed */
|
|
|
|
uint16_t plen; /* total payload (sum of all packets) */
|
|
|
|
|
|
|
|
/* straight from fw_eth_tx_pkts_vm_wr. */
|
|
|
|
__u8 ethmacdst[6];
|
|
|
|
__u8 ethmacsrc[6];
|
|
|
|
__be16 ethtype;
|
|
|
|
__be16 vlantci;
|
|
|
|
|
|
|
|
struct mbuf *mb[15];
|
|
|
|
};
|
|
|
|
|
2011-04-19 22:08:28 +00:00
|
|
|
/* txq: SGE egress queue + what's needed for Ethernet NIC */
|
2011-02-18 08:00:26 +00:00
|
|
|
struct sge_txq {
|
|
|
|
struct sge_eq eq; /* MUST be first */
|
2011-04-19 22:08:28 +00:00
|
|
|
|
|
|
|
struct ifnet *ifp; /* the interface this txq belongs to */
|
2014-12-31 23:19:16 +00:00
|
|
|
struct mp_ring *r; /* tx software ring */
|
2011-04-19 22:08:28 +00:00
|
|
|
struct tx_sdesc *sdesc; /* KVA of software descriptor ring */
|
2014-12-31 23:19:16 +00:00
|
|
|
struct sglist *gl;
|
|
|
|
__be32 cpl_ctrl0; /* for convenience */
|
2016-06-08 14:15:29 +00:00
|
|
|
int tc_idx; /* traffic class */
|
2021-02-01 11:00:09 +00:00
|
|
|
uint64_t last_tx; /* cycle count when eth_tx was last called */
|
2020-07-03 04:44:23 +00:00
|
|
|
struct txpkts txp;
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2014-12-31 23:19:16 +00:00
|
|
|
struct task tx_reclaim_task;
|
2011-02-18 08:00:26 +00:00
|
|
|
/* stats for common events first */
|
|
|
|
|
|
|
|
uint64_t txcsum; /* # of times hardware assisted with checksum */
|
2012-06-29 19:51:06 +00:00
|
|
|
uint64_t tso_wrs; /* # of TSO work requests */
|
2011-02-18 08:00:26 +00:00
|
|
|
uint64_t vlan_insertion;/* # of times VLAN tag was inserted */
|
|
|
|
uint64_t imm_wrs; /* # of work requests with immediate data */
|
|
|
|
uint64_t sgl_wrs; /* # of work requests with direct SGL */
|
|
|
|
uint64_t txpkt_wrs; /* # of txpkt work requests (not coalesced) */
|
2014-12-31 23:19:16 +00:00
|
|
|
uint64_t txpkts0_wrs; /* # of type0 coalesced tx work requests */
|
|
|
|
uint64_t txpkts1_wrs; /* # of type1 coalesced tx work requests */
|
|
|
|
uint64_t txpkts0_pkts; /* # of frames in type0 coalesced tx WRs */
|
|
|
|
uint64_t txpkts1_pkts; /* # of frames in type1 coalesced tx WRs */
|
2021-02-01 11:00:09 +00:00
|
|
|
uint64_t txpkts_flush; /* # of times txp had to be sent by tx_update */
|
2018-11-06 00:11:36 +00:00
|
|
|
uint64_t raw_wrs; /* # of raw work requests (alloc_wr_mbuf) */
|
2020-09-18 03:01:47 +00:00
|
|
|
uint64_t vxlan_tso_wrs; /* # of VXLAN TSO work requests */
|
|
|
|
uint64_t vxlan_txcsum;
|
NIC KTLS for Chelsio T6 adapters.
This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters.
Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE
connections.
NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment
the encrypted TLS frames output by the crypto engine. Instead, the
TOE is placed into a special setup to permit "dummy" connections to be
associated with regular sockets using KTLS. This permits using the
TOE to segment the encrypted TLS records. However, this approach does
have some limitations:
1) Regular TOE sockets cannot be used when the TOE is in this special
mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but
not both at the same time.
2) In NIC KTLS mode, the TOE is only able to accept a per-connection
timestamp offset that varies in the upper 4 bits. Put another way,
only connections whose timestamp offset has the 28 lower bits
cleared can use NIC KTLS and generate correct timestamps. The
driver will refuse to enable NIC KTLS on connections with a
timestamp offset with any of the lower 28 bits set. To use NIC
KTLS, users can either disable TCP timestamps by setting the
net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the
tcp_new_ts_offset() function to clear the lower 28 bits of the
generated offset.
3) Because the TCP segmentation relies on fields mirrored in a TCB in
the TOE, not all fields in a TCP packet can be sent in the TCP
segments generated from a TLS record. Specifically, for packets
containing TCP options other than timestamps, the driver will
inject an "empty" TCP packet holding the requested options (e.g. a
SACK scoreboard) along with the segments from the TLS record.
These empty TCP packets are counted by the
dev.cc.N.txq.M.kern_tls_options sysctls.
Unlike TOE TLS which is able to buffer encrypted TLS records in
on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS
records for retransmit requests as well as non-retransmit requests
that do not include the start of a TLS record but do include the
trailer. The T6 NIC KTLS code tries to optimize some of the cases for
requests to transmit partial TLS records. In particular it attempts
to minimize sending "waste" bytes that have to be given as input to
the crypto engine but are not needed on the wire to satisfy mbufs sent
from the TCP stack down to the driver.
TCP packets for TLS requests are broken down into the following
classes (with associated counters):
- Mbufs that send an entire TLS record in full do not have any waste
bytes (dev.cc.N.txq.M.kern_tls_full).
- Mbufs that send a short TLS record that ends before the end of the
trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC,
the encryption must always start at the beginning, so if the mbuf
starts at an offset into the TLS record, the offset bytes will be
"waste" bytes. For sockets using AES-GCM, the encryption can start
at the 16 byte block before the starting offset capping the waste at
15 bytes.
- Mbufs that send a partial TLS record that has a non-zero starting
offset but ends at the end of the trailer
(dev.cc.N.txq.M.kern_tls_partial). In order to compute the
authentication hash stored in the trailer, the entire TLS record
must be sent as input to the crypto engine, so the bytes before the
offset are always "waste" bytes.
In addition, other per-txq sysctls are provided:
- dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq
using AES-CBC.
- dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq
using AES-GCM.
- dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to
compensate for the TOE engine not being able to set FIN on the last
segment of a TLS record if the TLS record mbuf had FIN set.
- dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this
txq including full, short, and partial records.
- dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header
and payload) sent for TLS record requests.
- dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS
record requests.
To enable NIC KTLS with T6, set the following tunables prior to
loading the cxgbe(4) driver:
hw.cxgbe.config_file=kern_tls
hw.cxgbe.kern_tls=1
Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
|
|
|
|
|
|
|
uint64_t kern_tls_records;
|
|
|
|
uint64_t kern_tls_short;
|
|
|
|
uint64_t kern_tls_partial;
|
|
|
|
uint64_t kern_tls_full;
|
|
|
|
uint64_t kern_tls_octets;
|
|
|
|
uint64_t kern_tls_waste;
|
|
|
|
uint64_t kern_tls_options;
|
|
|
|
uint64_t kern_tls_header;
|
|
|
|
uint64_t kern_tls_fin;
|
|
|
|
uint64_t kern_tls_fin_short;
|
|
|
|
uint64_t kern_tls_cbc;
|
|
|
|
uint64_t kern_tls_gcm;
|
2011-02-18 08:00:26 +00:00
|
|
|
|
|
|
|
/* stats for not-that-common events */
|
NIC KTLS for Chelsio T6 adapters.
This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters.
Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE
connections.
NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment
the encrypted TLS frames output by the crypto engine. Instead, the
TOE is placed into a special setup to permit "dummy" connections to be
associated with regular sockets using KTLS. This permits using the
TOE to segment the encrypted TLS records. However, this approach does
have some limitations:
1) Regular TOE sockets cannot be used when the TOE is in this special
mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but
not both at the same time.
2) In NIC KTLS mode, the TOE is only able to accept a per-connection
timestamp offset that varies in the upper 4 bits. Put another way,
only connections whose timestamp offset has the 28 lower bits
cleared can use NIC KTLS and generate correct timestamps. The
driver will refuse to enable NIC KTLS on connections with a
timestamp offset with any of the lower 28 bits set. To use NIC
KTLS, users can either disable TCP timestamps by setting the
net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the
tcp_new_ts_offset() function to clear the lower 28 bits of the
generated offset.
3) Because the TCP segmentation relies on fields mirrored in a TCB in
the TOE, not all fields in a TCP packet can be sent in the TCP
segments generated from a TLS record. Specifically, for packets
containing TCP options other than timestamps, the driver will
inject an "empty" TCP packet holding the requested options (e.g. a
SACK scoreboard) along with the segments from the TLS record.
These empty TCP packets are counted by the
dev.cc.N.txq.M.kern_tls_options sysctls.
Unlike TOE TLS which is able to buffer encrypted TLS records in
on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS
records for retransmit requests as well as non-retransmit requests
that do not include the start of a TLS record but do include the
trailer. The T6 NIC KTLS code tries to optimize some of the cases for
requests to transmit partial TLS records. In particular it attempts
to minimize sending "waste" bytes that have to be given as input to
the crypto engine but are not needed on the wire to satisfy mbufs sent
from the TCP stack down to the driver.
TCP packets for TLS requests are broken down into the following
classes (with associated counters):
- Mbufs that send an entire TLS record in full do not have any waste
bytes (dev.cc.N.txq.M.kern_tls_full).
- Mbufs that send a short TLS record that ends before the end of the
trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC,
the encryption must always start at the beginning, so if the mbuf
starts at an offset into the TLS record, the offset bytes will be
"waste" bytes. For sockets using AES-GCM, the encryption can start
at the 16 byte block before the starting offset capping the waste at
15 bytes.
- Mbufs that send a partial TLS record that has a non-zero starting
offset but ends at the end of the trailer
(dev.cc.N.txq.M.kern_tls_partial). In order to compute the
authentication hash stored in the trailer, the entire TLS record
must be sent as input to the crypto engine, so the bytes before the
offset are always "waste" bytes.
In addition, other per-txq sysctls are provided:
- dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq
using AES-CBC.
- dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq
using AES-GCM.
- dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to
compensate for the TOE engine not being able to set FIN on the last
segment of a TLS record if the TLS record mbuf had FIN set.
- dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this
txq including full, short, and partial records.
- dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header
and payload) sent for TLS record requests.
- dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS
record requests.
To enable NIC KTLS with T6, set the following tunables prior to
loading the cxgbe(4) driver:
hw.cxgbe.config_file=kern_tls
hw.cxgbe.kern_tls=1
Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
|
|
|
|
|
|
|
/* Optional scratch space for constructing work requests. */
|
|
|
|
uint8_t ss[SGE_MAX_WR_LEN] __aligned(16);
|
2011-04-19 22:08:28 +00:00
|
|
|
} __aligned(CACHE_LINE_SIZE);
|
2011-02-18 08:00:26 +00:00
|
|
|
|
|
|
|
/* rxq: SGE ingress queue + SGE free list + miscellaneous items */
|
|
|
|
struct sge_rxq {
|
|
|
|
struct sge_iq iq; /* MUST be first */
|
2011-12-16 02:09:51 +00:00
|
|
|
struct sge_fl fl; /* MUST follow iq */
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2011-03-05 03:27:14 +00:00
|
|
|
struct ifnet *ifp; /* the interface this rxq belongs to */
|
2011-02-18 08:00:26 +00:00
|
|
|
struct lro_ctrl lro; /* LRO state */
|
|
|
|
|
|
|
|
/* stats for common events first */
|
|
|
|
|
|
|
|
uint64_t rxcsum; /* # of times hardware assisted with checksum */
|
|
|
|
uint64_t vlan_extraction;/* # of times VLAN tag was extracted */
|
2020-09-18 03:01:47 +00:00
|
|
|
uint64_t vxlan_rxcsum;
|
2011-02-18 08:00:26 +00:00
|
|
|
|
|
|
|
/* stats for not-that-common events */
|
|
|
|
|
|
|
|
} __aligned(CACHE_LINE_SIZE);
|
|
|
|
|
2012-06-19 07:34:13 +00:00
|
|
|
static inline struct sge_rxq *
|
|
|
|
iq_to_rxq(struct sge_iq *iq)
|
|
|
|
{
|
|
|
|
|
2012-10-19 13:26:40 +00:00
|
|
|
return (__containerof(iq, struct sge_rxq, iq));
|
2012-06-19 07:34:13 +00:00
|
|
|
}
|
|
|
|
|
2011-12-16 02:09:51 +00:00
|
|
|
/* ofld_rxq: SGE ingress queue + SGE free list + miscellaneous items */
|
|
|
|
struct sge_ofld_rxq {
|
|
|
|
struct sge_iq iq; /* MUST be first */
|
|
|
|
struct sge_fl fl; /* MUST follow iq */
|
2021-05-14 19:16:33 +00:00
|
|
|
counter_u64_t rx_iscsi_ddp_setup_ok;
|
|
|
|
counter_u64_t rx_iscsi_ddp_setup_error;
|
|
|
|
uint64_t rx_iscsi_ddp_pdus;
|
|
|
|
uint64_t rx_iscsi_ddp_octets;
|
|
|
|
uint64_t rx_iscsi_fl_pdus;
|
|
|
|
uint64_t rx_iscsi_fl_octets;
|
2021-09-10 22:10:00 +00:00
|
|
|
uint64_t rx_iscsi_padding_errors;
|
|
|
|
uint64_t rx_iscsi_header_digest_errors;
|
|
|
|
uint64_t rx_iscsi_data_digest_errors;
|
2021-03-26 22:05:44 +00:00
|
|
|
u_long rx_toe_tls_records;
|
|
|
|
u_long rx_toe_tls_octets;
|
2011-12-16 02:09:51 +00:00
|
|
|
} __aligned(CACHE_LINE_SIZE);
|
2012-06-19 07:34:13 +00:00
|
|
|
|
|
|
|
static inline struct sge_ofld_rxq *
|
|
|
|
iq_to_ofld_rxq(struct sge_iq *iq)
|
|
|
|
{
|
|
|
|
|
2012-10-19 13:26:40 +00:00
|
|
|
return (__containerof(iq, struct sge_ofld_rxq, iq));
|
2012-06-19 07:34:13 +00:00
|
|
|
}
|
2011-12-16 02:09:51 +00:00
|
|
|
|
2012-06-19 07:34:13 +00:00
|
|
|
struct wrqe {
|
|
|
|
STAILQ_ENTRY(wrqe) link;
|
|
|
|
struct sge_wrq *wrq;
|
|
|
|
int wr_len;
|
2014-12-31 23:19:16 +00:00
|
|
|
char wr[] __aligned(16);
|
|
|
|
};
|
|
|
|
|
|
|
|
struct wrq_cookie {
|
|
|
|
TAILQ_ENTRY(wrq_cookie) link;
|
|
|
|
int ndesc;
|
|
|
|
int pidx;
|
2012-06-19 07:34:13 +00:00
|
|
|
};
|
|
|
|
|
2011-12-16 02:09:51 +00:00
|
|
|
/*
|
2021-03-26 22:05:31 +00:00
|
|
|
* wrq: SGE egress queue that is given prebuilt work requests. Control queues
|
|
|
|
* are of this type.
|
2011-12-16 02:09:51 +00:00
|
|
|
*/
|
|
|
|
struct sge_wrq {
|
2011-04-19 22:08:28 +00:00
|
|
|
struct sge_eq eq; /* MUST be first */
|
|
|
|
|
2011-12-16 02:09:51 +00:00
|
|
|
struct adapter *adapter;
|
2014-12-31 23:19:16 +00:00
|
|
|
struct task wrq_tx_task;
|
|
|
|
|
|
|
|
/* Tx desc reserved but WR not "committed" yet. */
|
|
|
|
TAILQ_HEAD(wrq_incomplete_wrs , wrq_cookie) incomplete_wrs;
|
2012-06-19 07:34:13 +00:00
|
|
|
|
2014-12-31 23:19:16 +00:00
|
|
|
/* List of WRs ready to go out as soon as descriptors are available. */
|
2012-06-19 07:34:13 +00:00
|
|
|
STAILQ_HEAD(, wrqe) wr_list;
|
2014-12-31 23:19:16 +00:00
|
|
|
u_int nwr_pending;
|
|
|
|
u_int ndesc_needed;
|
2011-12-16 02:09:51 +00:00
|
|
|
|
2011-04-19 22:08:28 +00:00
|
|
|
/* stats for common events first */
|
|
|
|
|
2014-12-31 23:19:16 +00:00
|
|
|
uint64_t tx_wrs_direct; /* # of WRs written directly to desc ring. */
|
|
|
|
uint64_t tx_wrs_ss; /* # of WRs copied from scratch space. */
|
|
|
|
uint64_t tx_wrs_copied; /* # of WRs queued and copied to desc ring. */
|
2011-04-19 22:08:28 +00:00
|
|
|
|
|
|
|
/* stats for not-that-common events */
|
|
|
|
|
2014-12-31 23:19:16 +00:00
|
|
|
/*
|
|
|
|
* Scratch space for work requests that wrap around after reaching the
|
2016-05-03 03:41:25 +00:00
|
|
|
* status page, and some information about the last WR that used it.
|
2014-12-31 23:19:16 +00:00
|
|
|
*/
|
|
|
|
uint16_t ss_pidx;
|
|
|
|
uint16_t ss_len;
|
|
|
|
uint8_t ss[SGE_MAX_WR_LEN];
|
|
|
|
|
2011-04-19 22:08:28 +00:00
|
|
|
} __aligned(CACHE_LINE_SIZE);
|
|
|
|
|
2021-03-26 22:05:31 +00:00
|
|
|
/* ofld_txq: SGE egress queue + miscellaneous items */
|
|
|
|
struct sge_ofld_txq {
|
|
|
|
struct sge_wrq wrq;
|
2021-04-12 20:56:04 +00:00
|
|
|
counter_u64_t tx_iscsi_pdus;
|
|
|
|
counter_u64_t tx_iscsi_octets;
|
2021-08-06 21:21:37 +00:00
|
|
|
counter_u64_t tx_iscsi_iso_wrs;
|
2021-03-26 22:05:44 +00:00
|
|
|
counter_u64_t tx_toe_tls_records;
|
|
|
|
counter_u64_t tx_toe_tls_octets;
|
2021-03-26 22:05:31 +00:00
|
|
|
} __aligned(CACHE_LINE_SIZE);
|
|
|
|
|
2017-06-15 19:56:59 +00:00
|
|
|
#define INVALID_NM_RXQ_CNTXT_ID ((uint16_t)(-1))
|
2014-05-27 18:18:41 +00:00
|
|
|
struct sge_nm_rxq {
|
2020-03-20 05:12:16 +00:00
|
|
|
/* Items used by the driver rx ithread are in this cacheline. */
|
|
|
|
volatile int nm_state __aligned(CACHE_LINE_SIZE); /* NM_OFF, NM_ON, or NM_BUSY */
|
|
|
|
u_int nid; /* netmap ring # for this queue */
|
2015-12-03 00:02:01 +00:00
|
|
|
struct vi_info *vi;
|
2014-05-27 18:18:41 +00:00
|
|
|
|
2014-08-02 00:56:34 +00:00
|
|
|
struct iq_desc *iq_desc;
|
2014-05-27 18:18:41 +00:00
|
|
|
uint16_t iq_abs_id;
|
|
|
|
uint16_t iq_cntxt_id;
|
|
|
|
uint16_t iq_cidx;
|
|
|
|
uint16_t iq_sidx;
|
|
|
|
uint8_t iq_gen;
|
2020-03-20 05:12:16 +00:00
|
|
|
uint32_t fl_sidx;
|
2014-05-27 18:18:41 +00:00
|
|
|
|
2020-03-20 05:12:16 +00:00
|
|
|
/* Items used by netmap rxsync are in this cacheline. */
|
|
|
|
__be64 *fl_desc __aligned(CACHE_LINE_SIZE);
|
2014-05-27 18:18:41 +00:00
|
|
|
uint16_t fl_cntxt_id;
|
|
|
|
uint32_t fl_pidx;
|
2020-03-20 05:12:16 +00:00
|
|
|
uint32_t fl_sidx2; /* copy of fl_sidx */
|
2014-05-27 18:18:41 +00:00
|
|
|
uint32_t fl_db_val;
|
2018-06-15 23:42:22 +00:00
|
|
|
u_int fl_db_saved;
|
2020-09-29 07:51:06 +00:00
|
|
|
u_int fl_db_threshold; /* in descriptors */
|
2020-03-20 05:12:16 +00:00
|
|
|
u_int fl_hwidx:4;
|
2014-05-27 18:18:41 +00:00
|
|
|
|
2020-03-20 05:12:16 +00:00
|
|
|
/*
|
|
|
|
* fl_cidx is used by both the ithread and rxsync, the rest are not used
|
|
|
|
* in the rx fast path.
|
|
|
|
*/
|
|
|
|
uint32_t fl_cidx __aligned(CACHE_LINE_SIZE);
|
2014-05-27 18:18:41 +00:00
|
|
|
|
|
|
|
bus_dma_tag_t iq_desc_tag;
|
|
|
|
bus_dmamap_t iq_desc_map;
|
|
|
|
bus_addr_t iq_ba;
|
|
|
|
int intr_idx;
|
|
|
|
|
|
|
|
bus_dma_tag_t fl_desc_tag;
|
|
|
|
bus_dmamap_t fl_desc_map;
|
|
|
|
bus_addr_t fl_ba;
|
2020-03-20 05:12:16 +00:00
|
|
|
};
|
2014-05-27 18:18:41 +00:00
|
|
|
|
2017-06-15 19:56:59 +00:00
|
|
|
#define INVALID_NM_TXQ_CNTXT_ID ((u_int)(-1))
|
2014-05-27 18:18:41 +00:00
|
|
|
struct sge_nm_txq {
|
|
|
|
struct tx_desc *desc;
|
|
|
|
uint16_t cidx;
|
|
|
|
uint16_t pidx;
|
|
|
|
uint16_t sidx;
|
|
|
|
uint16_t equiqidx; /* EQUIQ last requested at this pidx */
|
|
|
|
uint16_t equeqidx; /* EQUEQ last requested at this pidx */
|
|
|
|
uint16_t dbidx; /* pidx of the most recent doorbell */
|
2017-11-15 06:45:33 +00:00
|
|
|
uint8_t doorbells;
|
2014-05-27 18:18:41 +00:00
|
|
|
volatile uint32_t *udb;
|
|
|
|
u_int udb_qid;
|
|
|
|
u_int cntxt_id;
|
|
|
|
__be32 cpl_ctrl0; /* for convenience */
|
2019-12-10 08:16:19 +00:00
|
|
|
__be32 op_pkd; /* ditto */
|
2014-05-27 18:18:41 +00:00
|
|
|
u_int nid; /* netmap ring # for this queue */
|
|
|
|
|
|
|
|
/* infrequently used items after this */
|
|
|
|
|
|
|
|
bus_dma_tag_t desc_tag;
|
|
|
|
bus_dmamap_t desc_map;
|
|
|
|
bus_addr_t ba;
|
|
|
|
int iqidx;
|
|
|
|
} __aligned(CACHE_LINE_SIZE);
|
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
struct sge {
|
2011-12-16 02:09:51 +00:00
|
|
|
int nrxq; /* total # of Ethernet rx queues */
|
2017-05-10 00:42:28 +00:00
|
|
|
int ntxq; /* total # of Ethernet tx queues */
|
2011-12-16 02:09:51 +00:00
|
|
|
int nofldrxq; /* total # of TOE rx queues */
|
|
|
|
int nofldtxq; /* total # of TOE tx queues */
|
2014-05-27 18:18:41 +00:00
|
|
|
int nnmrxq; /* total # of netmap rx queues */
|
|
|
|
int nnmtxq; /* total # of netmap tx queues */
|
2011-12-16 02:09:51 +00:00
|
|
|
int niq; /* total # of ingress queues */
|
|
|
|
int neq; /* total # of egress queues */
|
2011-02-18 08:00:26 +00:00
|
|
|
|
|
|
|
struct sge_iq fwq; /* Firmware event queue */
|
2011-12-16 02:09:51 +00:00
|
|
|
struct sge_wrq *ctrlq; /* Control queues */
|
2011-02-18 08:00:26 +00:00
|
|
|
struct sge_txq *txq; /* NIC tx queues */
|
|
|
|
struct sge_rxq *rxq; /* NIC rx queues */
|
2021-03-26 22:05:31 +00:00
|
|
|
struct sge_ofld_txq *ofld_txq; /* TOE tx queues */
|
2011-12-16 02:09:51 +00:00
|
|
|
struct sge_ofld_rxq *ofld_rxq; /* TOE rx queues */
|
2014-05-27 18:18:41 +00:00
|
|
|
struct sge_nm_txq *nm_txq; /* netmap tx queues */
|
|
|
|
struct sge_nm_rxq *nm_rxq; /* netmap rx queues */
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2016-08-09 17:49:42 +00:00
|
|
|
uint16_t iq_start; /* first cntxt_id */
|
|
|
|
uint16_t iq_base; /* first abs_id */
|
|
|
|
int eq_start; /* first cntxt_id */
|
|
|
|
int eq_base; /* first abs_id */
|
2020-10-22 08:40:25 +00:00
|
|
|
int iqmap_sz;
|
|
|
|
int eqmap_sz;
|
2011-02-18 08:00:26 +00:00
|
|
|
struct sge_iq **iqmap; /* iq->cntxt_id to iq mapping */
|
|
|
|
struct sge_eq **eqmap; /* eq->cntxt_id to eq mapping */
|
2013-08-30 01:45:36 +00:00
|
|
|
|
2020-02-04 00:51:10 +00:00
|
|
|
int8_t safe_zidx;
|
|
|
|
struct rx_buf_info rx_buf_info[SW_ZONE_SIZES];
|
2011-02-18 08:00:26 +00:00
|
|
|
};
|
|
|
|
|
2016-09-16 00:08:37 +00:00
|
|
|
struct devnames {
|
|
|
|
const char *nexus_name;
|
|
|
|
const char *ifnet_name;
|
|
|
|
const char *vi_ifnet_name;
|
|
|
|
const char *pf03_drv_name;
|
|
|
|
const char *vf_nexus_name;
|
|
|
|
const char *vf_ifnet_name;
|
|
|
|
};
|
|
|
|
|
2018-11-29 01:15:53 +00:00
|
|
|
struct clip_entry;
|
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
struct adapter {
|
2011-12-16 02:09:51 +00:00
|
|
|
SLIST_ENTRY(adapter) link;
|
2011-02-18 08:00:26 +00:00
|
|
|
device_t dev;
|
|
|
|
struct cdev *cdev;
|
2016-09-16 00:08:37 +00:00
|
|
|
const struct devnames *names;
|
2011-02-18 08:00:26 +00:00
|
|
|
|
|
|
|
/* PCIe register resources */
|
|
|
|
int regs_rid;
|
|
|
|
struct resource *regs_res;
|
|
|
|
int msix_rid;
|
|
|
|
struct resource *msix_res;
|
|
|
|
bus_space_handle_t bh;
|
|
|
|
bus_space_tag_t bt;
|
|
|
|
bus_size_t mmio_len;
|
2013-03-30 02:26:20 +00:00
|
|
|
int udbs_rid;
|
|
|
|
struct resource *udbs_res;
|
|
|
|
volatile uint8_t *udbs_base;
|
2011-02-18 08:00:26 +00:00
|
|
|
|
|
|
|
unsigned int pf;
|
|
|
|
unsigned int mbox;
|
2016-03-08 07:48:55 +00:00
|
|
|
unsigned int vpd_busy;
|
|
|
|
unsigned int vpd_flag;
|
2011-02-18 08:00:26 +00:00
|
|
|
|
|
|
|
/* Interrupt information */
|
|
|
|
int intr_type;
|
|
|
|
int intr_count;
|
|
|
|
struct irq {
|
|
|
|
struct resource *res;
|
|
|
|
int rid;
|
|
|
|
void *tag;
|
2016-06-23 02:53:00 +00:00
|
|
|
struct sge_rxq *rxq;
|
|
|
|
struct sge_nm_rxq *nm_rxq;
|
|
|
|
} __aligned(CACHE_LINE_SIZE) *irq;
|
2016-08-01 22:39:51 +00:00
|
|
|
int sge_gts_reg;
|
|
|
|
int sge_kdoorbell_reg;
|
2011-02-18 08:00:26 +00:00
|
|
|
|
|
|
|
bus_dma_tag_t dmat; /* Parent DMA tag */
|
|
|
|
|
|
|
|
struct sge sge;
|
2013-08-29 06:26:22 +00:00
|
|
|
int lro_timeout;
|
2016-07-05 01:29:24 +00:00
|
|
|
int sc_do_rxcopy;
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2020-09-18 03:01:47 +00:00
|
|
|
int vxlan_port;
|
|
|
|
u_int vxlan_refcount;
|
|
|
|
int rawf_base;
|
|
|
|
int nrawf;
|
|
|
|
|
2016-03-04 13:11:13 +00:00
|
|
|
struct taskqueue *tq[MAX_NCHAN]; /* General purpose taskqueues */
|
2011-02-18 08:00:26 +00:00
|
|
|
struct port_info *port[MAX_NPORTS];
|
2017-10-24 05:41:48 +00:00
|
|
|
uint8_t chan_map[MAX_NCHAN]; /* channel -> port */
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2021-05-23 21:58:29 +00:00
|
|
|
CXGBE_LIST_HEAD(, clip_entry) *clip_table;
|
|
|
|
TAILQ_HEAD(, clip_entry) clip_pending; /* these need hw update. */
|
|
|
|
u_long clip_mask;
|
2018-11-29 01:15:53 +00:00
|
|
|
int clip_gen;
|
2021-05-23 21:58:29 +00:00
|
|
|
struct timeout_task clip_task;
|
2018-11-29 01:15:53 +00:00
|
|
|
|
2012-06-19 07:34:13 +00:00
|
|
|
void *tom_softc; /* (struct tom_data *) */
|
2011-12-16 02:09:51 +00:00
|
|
|
struct tom_tunables tt;
|
cxgbe(4): Add support for Connection Offload Policy (aka COP).
COP allows fine-grained control on whether to offload a TCP connection
using t4_tom, and what settings to apply to a connection selected for
offload. t4_tom must still be loaded and IFCAP_TOE must still be
enabled for full TCP offload to take place on an interface. The
difference is that IFCAP_TOE used to be the only knob and would enable
TOE for all new connections on the inteface, but now the driver will
also consult the COP, if any, before offloading to the hardware TOE.
A policy is a plain text file with any number of rules, one per line.
Each rule has a "match" part consisting of a socket-type (L = listen,
A = active open, P = passive open, D = don't care) and a pcap-filter(7)
expression, and a "settings" part that specifies whether to offload the
connection or not and the parameters to use if so. The general format
of a rule is: [socket-type] expr => settings
Example. See cxgbetool(8) for more information.
[L] ip && port http => offload
[L] port 443 => !offload
[L] port ssh => offload
[P] src net 192.168/16 && dst port ssh => offload !nagle !timestamp cong newreno
[P] dst port ssh => offload !nagle ecn cong tahoe
[P] dst port http => offload
[A] dst port 443 => offload tls
[A] dst net 192.168/16 => offload !timestamp cong highspeed
The driver processes the rules for each new listen, active open, or
passive open and stops at the first match. There is an implicit rule at
the end of every policy that prohibits offload when no rule in the
policy matches:
[D] all => !offload
This is a reworked and expanded version of a patch submitted by
Krishnamraju Eraparaju @ Chelsio.
Sponsored by: Chelsio Communications
2018-04-14 19:07:56 +00:00
|
|
|
struct t4_offload_policy *policy;
|
|
|
|
struct rwlock policy_lock;
|
|
|
|
|
2013-08-28 20:45:45 +00:00
|
|
|
void *iwarp_softc; /* (struct c4iw_dev *) */
|
cxgbe(4): Add support for Connection Offload Policy (aka COP).
COP allows fine-grained control on whether to offload a TCP connection
using t4_tom, and what settings to apply to a connection selected for
offload. t4_tom must still be loaded and IFCAP_TOE must still be
enabled for full TCP offload to take place on an interface. The
difference is that IFCAP_TOE used to be the only knob and would enable
TOE for all new connections on the inteface, but now the driver will
also consult the COP, if any, before offloading to the hardware TOE.
A policy is a plain text file with any number of rules, one per line.
Each rule has a "match" part consisting of a socket-type (L = listen,
A = active open, P = passive open, D = don't care) and a pcap-filter(7)
expression, and a "settings" part that specifies whether to offload the
connection or not and the parameters to use if so. The general format
of a rule is: [socket-type] expr => settings
Example. See cxgbetool(8) for more information.
[L] ip && port http => offload
[L] port 443 => !offload
[L] port ssh => offload
[P] src net 192.168/16 && dst port ssh => offload !nagle !timestamp cong newreno
[P] dst port ssh => offload !nagle ecn cong tahoe
[P] dst port http => offload
[A] dst port 443 => offload tls
[A] dst net 192.168/16 => offload !timestamp cong highspeed
The driver processes the rules for each new listen, active open, or
passive open and stops at the first match. There is an implicit rule at
the end of every policy that prohibits offload when no rule in the
policy matches:
[D] all => !offload
This is a reworked and expanded version of a patch submitted by
Krishnamraju Eraparaju @ Chelsio.
Sponsored by: Chelsio Communications
2018-04-14 19:07:56 +00:00
|
|
|
struct iw_tunables iwt;
|
2015-12-26 00:26:02 +00:00
|
|
|
void *iscsi_ulp_softc; /* (struct cxgbei_data *) */
|
Add a driver for the Chelsio T6 crypto accelerator engine.
The ccr(4) driver supports use of the crypto accelerator engine on
Chelsio T6 NICs in "lookaside" mode via the opencrypto framework.
Currently, the driver supports AES-CBC, AES-CTR, AES-GCM, and AES-XTS
cipher algorithms as well as the SHA1-HMAC, SHA2-256-HMAC, SHA2-384-HMAC,
and SHA2-512-HMAC authentication algorithms. The driver also supports
chaining one of AES-CBC, AES-CTR, or AES-XTS with an authentication
algorithm for encrypt-then-authenticate operations.
Note that this driver is still under active development and testing and
may not yet be ready for production use. It does pass the tests in
tests/sys/opencrypto with the exception that the AES-GCM implementation
in the driver does not yet support requests with a zero byte payload.
To use this driver currently, the "uwire" configuration must be used
along with explicitly enabling support for lookaside crypto capabilities
in the cxgbe(4) driver. These can be done by setting the following
tunables before loading the cxgbe(4) driver:
hw.cxgbe.config_file=uwire
hw.cxgbe.cryptocaps_allowed=-1
MFC after: 1 month
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D10763
2017-05-17 22:13:07 +00:00
|
|
|
void *ccr_softc; /* (struct ccr_softc *) */
|
2011-05-30 21:07:26 +00:00
|
|
|
struct l2t_data *l2t; /* L2 table */
|
2018-05-31 21:31:08 +00:00
|
|
|
struct smt_data *smt; /* Source MAC Table */
|
2011-02-18 08:00:26 +00:00
|
|
|
struct tid_info tids;
|
2018-11-15 23:00:30 +00:00
|
|
|
vmem_t *key_map;
|
NIC KTLS for Chelsio T6 adapters.
This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters.
Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE
connections.
NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment
the encrypted TLS frames output by the crypto engine. Instead, the
TOE is placed into a special setup to permit "dummy" connections to be
associated with regular sockets using KTLS. This permits using the
TOE to segment the encrypted TLS records. However, this approach does
have some limitations:
1) Regular TOE sockets cannot be used when the TOE is in this special
mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but
not both at the same time.
2) In NIC KTLS mode, the TOE is only able to accept a per-connection
timestamp offset that varies in the upper 4 bits. Put another way,
only connections whose timestamp offset has the 28 lower bits
cleared can use NIC KTLS and generate correct timestamps. The
driver will refuse to enable NIC KTLS on connections with a
timestamp offset with any of the lower 28 bits set. To use NIC
KTLS, users can either disable TCP timestamps by setting the
net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the
tcp_new_ts_offset() function to clear the lower 28 bits of the
generated offset.
3) Because the TCP segmentation relies on fields mirrored in a TCB in
the TOE, not all fields in a TCP packet can be sent in the TCP
segments generated from a TLS record. Specifically, for packets
containing TCP options other than timestamps, the driver will
inject an "empty" TCP packet holding the requested options (e.g. a
SACK scoreboard) along with the segments from the TLS record.
These empty TCP packets are counted by the
dev.cc.N.txq.M.kern_tls_options sysctls.
Unlike TOE TLS which is able to buffer encrypted TLS records in
on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS
records for retransmit requests as well as non-retransmit requests
that do not include the start of a TLS record but do include the
trailer. The T6 NIC KTLS code tries to optimize some of the cases for
requests to transmit partial TLS records. In particular it attempts
to minimize sending "waste" bytes that have to be given as input to
the crypto engine but are not needed on the wire to satisfy mbufs sent
from the TCP stack down to the driver.
TCP packets for TLS requests are broken down into the following
classes (with associated counters):
- Mbufs that send an entire TLS record in full do not have any waste
bytes (dev.cc.N.txq.M.kern_tls_full).
- Mbufs that send a short TLS record that ends before the end of the
trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC,
the encryption must always start at the beginning, so if the mbuf
starts at an offset into the TLS record, the offset bytes will be
"waste" bytes. For sockets using AES-GCM, the encryption can start
at the 16 byte block before the starting offset capping the waste at
15 bytes.
- Mbufs that send a partial TLS record that has a non-zero starting
offset but ends at the end of the trailer
(dev.cc.N.txq.M.kern_tls_partial). In order to compute the
authentication hash stored in the trailer, the entire TLS record
must be sent as input to the crypto engine, so the bytes before the
offset are always "waste" bytes.
In addition, other per-txq sysctls are provided:
- dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq
using AES-CBC.
- dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq
using AES-GCM.
- dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to
compensate for the TOE engine not being able to set FIN on the last
segment of a TLS record if the TLS record mbuf had FIN set.
- dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this
txq including full, short, and partial records.
- dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header
and payload) sent for TLS record requests.
- dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS
record requests.
To enable NIC KTLS with T6, set the following tunables prior to
loading the cxgbe(4) driver:
hw.cxgbe.config_file=kern_tls
hw.cxgbe.kern_tls=1
Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
|
|
|
struct tls_tunables tlst;
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2017-11-15 06:45:33 +00:00
|
|
|
uint8_t doorbells;
|
2021-11-15 18:55:04 +00:00
|
|
|
int offload_map; /* port_id's with IFCAP_TOE enabled */
|
|
|
|
int bt_map; /* tx_chan's with BASE-T */
|
2015-02-08 09:28:55 +00:00
|
|
|
int active_ulds; /* ULDs activated on this adapter */
|
2011-02-18 08:00:26 +00:00
|
|
|
int flags;
|
2015-06-16 12:36:29 +00:00
|
|
|
int debug_flags;
|
2022-02-04 21:16:35 +00:00
|
|
|
int error_flags; /* Used by error handler and live reset. */
|
2011-02-18 08:00:26 +00:00
|
|
|
|
Add support for packet-sniffing tracers to cxgbe(4). This works with
all T4 and T5 based cards and is useful for analyzing TSO, LRO, TOE, and
for general purpose monitoring without tapping any cxgbe or cxl ifnet
directly.
Tracers on the T4/T5 chips provide access to Ethernet frames exactly as
they were received from or transmitted on the wire. On transmit, a
tracer will capture a frame after TSO segmentation, hw VLAN tag
insertion, hw L3 & L4 checksum insertion, etc. It will also capture
frames generated by the TCP offload engine (TOE traffic is normally
invisible to the kernel). On receive, a tracer will capture a frame
before hw VLAN extraction, runt filtering, other badness filtering,
before the steering/drop/L2-rewrite filters or the TOE have had a go at
it, and of course before sw LRO in the driver.
There are 4 tracers on a chip. A tracer can trace only in one direction
(tx or rx). For now cxgbetool will set up tracers to capture the first
128B of every transmitted or received frame on a given port. This is a
small subset of what the hardware can do. A pseudo ifnet with the same
name as the nexus driver (t4nex0 or t5nex0) will be created for tracing.
The data delivered to this ifnet is an additional copy made inside the
chip. Normal delivery to cxgbe<n> or cxl<n> will be made as usual.
/* watch cxl0, which is the first port hanging off t5nex0. */
# cxgbetool t5nex0 tracer 0 tx0 (watch what cxl0 is transmitting)
# cxgbetool t5nex0 tracer 1 rx0 (watch what cxl0 is receiving)
# cxgbetool t5nex0 tracer list
# tcpdump -i t5nex0 <== all that cxl0 sees and puts on the wire
If you were doing TSO, a tcpdump on cxl0 may have shown you ~64K
"frames" with no L3/L4 checksum but this will show you the frames that
were actually transmitted.
/* all done */
# cxgbetool t5nex0 tracer 0 disable
# cxgbetool t5nex0 tracer 1 disable
# cxgbetool t5nex0 tracer list
# ifconfig t5nex0 destroy
2013-07-26 22:04:11 +00:00
|
|
|
char ifp_lockname[16];
|
|
|
|
struct mtx ifp_lock;
|
|
|
|
struct ifnet *ifp; /* tracer ifp */
|
|
|
|
struct ifmedia media;
|
|
|
|
int traceq; /* iq used by all tracers, -1 if none */
|
|
|
|
int tracer_valid; /* bitmap of valid tracers */
|
|
|
|
int tracer_enabled; /* bitmap of enabled tracers */
|
|
|
|
|
2016-03-11 03:15:17 +00:00
|
|
|
char fw_version[16];
|
|
|
|
char tp_version[16];
|
2016-08-27 00:13:41 +00:00
|
|
|
char er_version[16];
|
|
|
|
char bs_version[16];
|
2013-01-26 03:10:28 +00:00
|
|
|
char cfg_file[32];
|
|
|
|
u_int cfcsum;
|
2011-02-18 08:00:26 +00:00
|
|
|
struct adapter_params params;
|
2016-03-04 13:11:13 +00:00
|
|
|
const struct chip_params *chip_params;
|
2011-02-18 08:00:26 +00:00
|
|
|
struct t4_virt_res vres;
|
|
|
|
|
2016-03-12 02:54:55 +00:00
|
|
|
uint16_t nbmcaps;
|
2011-12-16 02:09:51 +00:00
|
|
|
uint16_t linkcaps;
|
2016-03-12 02:54:55 +00:00
|
|
|
uint16_t switchcaps;
|
2011-12-16 02:09:51 +00:00
|
|
|
uint16_t niccaps;
|
|
|
|
uint16_t toecaps;
|
|
|
|
uint16_t rdmacaps;
|
2016-09-12 00:15:40 +00:00
|
|
|
uint16_t cryptocaps;
|
2011-12-16 02:09:51 +00:00
|
|
|
uint16_t iscsicaps;
|
|
|
|
uint16_t fcoecaps;
|
|
|
|
|
2021-04-26 21:03:06 +00:00
|
|
|
struct sysctl_ctx_list ctx;
|
|
|
|
struct sysctl_oid *ctrlq_oid;
|
|
|
|
struct sysctl_oid *fwq_oid;
|
2011-04-19 22:08:28 +00:00
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
struct mtx sc_lock;
|
|
|
|
char lockname[16];
|
2011-12-16 02:09:51 +00:00
|
|
|
|
|
|
|
/* Starving free lists */
|
|
|
|
struct mtx sfl_lock; /* same cache-line as sc_lock? but that's ok */
|
|
|
|
TAILQ_HEAD(, sge_fl) sfl;
|
|
|
|
struct callout sfl_callout;
|
|
|
|
|
2021-04-28 04:33:10 +00:00
|
|
|
/*
|
|
|
|
* Driver code that can run when the adapter is suspended must use this
|
|
|
|
* lock or a synchronized_op and check for HW_OFF_LIMITS before
|
|
|
|
* accessing hardware.
|
|
|
|
*
|
|
|
|
* XXX: could be changed to rwlock. wlock in suspend/resume and for
|
|
|
|
* indirect register access, rlock everywhere else.
|
|
|
|
*/
|
|
|
|
struct mtx reg_lock;
|
2014-09-27 05:50:31 +00:00
|
|
|
|
2016-03-10 06:15:31 +00:00
|
|
|
struct memwin memwin[NUM_MEMWIN]; /* memory windows */
|
|
|
|
|
2017-05-02 20:38:10 +00:00
|
|
|
struct mtx tc_lock;
|
|
|
|
struct task tc_task;
|
|
|
|
|
2022-02-04 21:16:35 +00:00
|
|
|
struct task fatal_error_task;
|
2021-04-28 04:33:10 +00:00
|
|
|
struct task reset_task;
|
|
|
|
const void *reset_thread;
|
|
|
|
int num_resets;
|
|
|
|
int incarnation;
|
|
|
|
|
2013-01-10 23:56:50 +00:00
|
|
|
const char *last_op;
|
|
|
|
const void *last_op_thr;
|
2015-08-19 15:40:03 +00:00
|
|
|
int last_op_flags;
|
2019-03-28 21:22:28 +00:00
|
|
|
|
|
|
|
int swintr;
|
2019-11-24 16:40:54 +00:00
|
|
|
int sensor_resets;
|
NIC KTLS for Chelsio T6 adapters.
This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters.
Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE
connections.
NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment
the encrypted TLS frames output by the crypto engine. Instead, the
TOE is placed into a special setup to permit "dummy" connections to be
associated with regular sockets using KTLS. This permits using the
TOE to segment the encrypted TLS records. However, this approach does
have some limitations:
1) Regular TOE sockets cannot be used when the TOE is in this special
mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but
not both at the same time.
2) In NIC KTLS mode, the TOE is only able to accept a per-connection
timestamp offset that varies in the upper 4 bits. Put another way,
only connections whose timestamp offset has the 28 lower bits
cleared can use NIC KTLS and generate correct timestamps. The
driver will refuse to enable NIC KTLS on connections with a
timestamp offset with any of the lower 28 bits set. To use NIC
KTLS, users can either disable TCP timestamps by setting the
net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the
tcp_new_ts_offset() function to clear the lower 28 bits of the
generated offset.
3) Because the TCP segmentation relies on fields mirrored in a TCB in
the TOE, not all fields in a TCP packet can be sent in the TCP
segments generated from a TLS record. Specifically, for packets
containing TCP options other than timestamps, the driver will
inject an "empty" TCP packet holding the requested options (e.g. a
SACK scoreboard) along with the segments from the TLS record.
These empty TCP packets are counted by the
dev.cc.N.txq.M.kern_tls_options sysctls.
Unlike TOE TLS which is able to buffer encrypted TLS records in
on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS
records for retransmit requests as well as non-retransmit requests
that do not include the start of a TLS record but do include the
trailer. The T6 NIC KTLS code tries to optimize some of the cases for
requests to transmit partial TLS records. In particular it attempts
to minimize sending "waste" bytes that have to be given as input to
the crypto engine but are not needed on the wire to satisfy mbufs sent
from the TCP stack down to the driver.
TCP packets for TLS requests are broken down into the following
classes (with associated counters):
- Mbufs that send an entire TLS record in full do not have any waste
bytes (dev.cc.N.txq.M.kern_tls_full).
- Mbufs that send a short TLS record that ends before the end of the
trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC,
the encryption must always start at the beginning, so if the mbuf
starts at an offset into the TLS record, the offset bytes will be
"waste" bytes. For sockets using AES-GCM, the encryption can start
at the 16 byte block before the starting offset capping the waste at
15 bytes.
- Mbufs that send a partial TLS record that has a non-zero starting
offset but ends at the end of the trailer
(dev.cc.N.txq.M.kern_tls_partial). In order to compute the
authentication hash stored in the trailer, the entire TLS record
must be sent as input to the crypto engine, so the bytes before the
offset are always "waste" bytes.
In addition, other per-txq sysctls are provided:
- dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq
using AES-CBC.
- dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq
using AES-GCM.
- dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to
compensate for the TOE engine not being able to set FIN on the last
segment of a TLS record if the TLS record mbuf had FIN set.
- dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this
txq including full, short, and partial records.
- dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header
and payload) sent for TLS record requests.
- dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS
record requests.
To enable NIC KTLS with T6, set the following tunables prior to
loading the cxgbe(4) driver:
hw.cxgbe.config_file=kern_tls
hw.cxgbe.kern_tls=1
Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
|
|
|
|
|
|
|
struct callout ktls_tick;
|
2011-02-18 08:00:26 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
#define ADAPTER_LOCK(sc) mtx_lock(&(sc)->sc_lock)
|
|
|
|
#define ADAPTER_UNLOCK(sc) mtx_unlock(&(sc)->sc_lock)
|
|
|
|
#define ADAPTER_LOCK_ASSERT_OWNED(sc) mtx_assert(&(sc)->sc_lock, MA_OWNED)
|
|
|
|
#define ADAPTER_LOCK_ASSERT_NOTOWNED(sc) mtx_assert(&(sc)->sc_lock, MA_NOTOWNED)
|
|
|
|
|
2013-01-10 23:56:50 +00:00
|
|
|
#define ASSERT_SYNCHRONIZED_OP(sc) \
|
|
|
|
KASSERT(IS_BUSY(sc) && \
|
|
|
|
(mtx_owned(&(sc)->sc_lock) || sc->last_op_thr == curthread), \
|
|
|
|
("%s: operation not synchronized.", __func__))
|
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
#define PORT_LOCK(pi) mtx_lock(&(pi)->pi_lock)
|
|
|
|
#define PORT_UNLOCK(pi) mtx_unlock(&(pi)->pi_lock)
|
|
|
|
#define PORT_LOCK_ASSERT_OWNED(pi) mtx_assert(&(pi)->pi_lock, MA_OWNED)
|
|
|
|
#define PORT_LOCK_ASSERT_NOTOWNED(pi) mtx_assert(&(pi)->pi_lock, MA_NOTOWNED)
|
|
|
|
|
|
|
|
#define FL_LOCK(fl) mtx_lock(&(fl)->fl_lock)
|
|
|
|
#define FL_TRYLOCK(fl) mtx_trylock(&(fl)->fl_lock)
|
|
|
|
#define FL_UNLOCK(fl) mtx_unlock(&(fl)->fl_lock)
|
|
|
|
#define FL_LOCK_ASSERT_OWNED(fl) mtx_assert(&(fl)->fl_lock, MA_OWNED)
|
|
|
|
#define FL_LOCK_ASSERT_NOTOWNED(fl) mtx_assert(&(fl)->fl_lock, MA_NOTOWNED)
|
|
|
|
|
|
|
|
#define RXQ_FL_LOCK(rxq) FL_LOCK(&(rxq)->fl)
|
|
|
|
#define RXQ_FL_UNLOCK(rxq) FL_UNLOCK(&(rxq)->fl)
|
|
|
|
#define RXQ_FL_LOCK_ASSERT_OWNED(rxq) FL_LOCK_ASSERT_OWNED(&(rxq)->fl)
|
|
|
|
#define RXQ_FL_LOCK_ASSERT_NOTOWNED(rxq) FL_LOCK_ASSERT_NOTOWNED(&(rxq)->fl)
|
|
|
|
|
|
|
|
#define EQ_LOCK(eq) mtx_lock(&(eq)->eq_lock)
|
|
|
|
#define EQ_TRYLOCK(eq) mtx_trylock(&(eq)->eq_lock)
|
|
|
|
#define EQ_UNLOCK(eq) mtx_unlock(&(eq)->eq_lock)
|
|
|
|
#define EQ_LOCK_ASSERT_OWNED(eq) mtx_assert(&(eq)->eq_lock, MA_OWNED)
|
|
|
|
#define EQ_LOCK_ASSERT_NOTOWNED(eq) mtx_assert(&(eq)->eq_lock, MA_NOTOWNED)
|
|
|
|
|
|
|
|
#define TXQ_LOCK(txq) EQ_LOCK(&(txq)->eq)
|
|
|
|
#define TXQ_TRYLOCK(txq) EQ_TRYLOCK(&(txq)->eq)
|
|
|
|
#define TXQ_UNLOCK(txq) EQ_UNLOCK(&(txq)->eq)
|
|
|
|
#define TXQ_LOCK_ASSERT_OWNED(txq) EQ_LOCK_ASSERT_OWNED(&(txq)->eq)
|
|
|
|
#define TXQ_LOCK_ASSERT_NOTOWNED(txq) EQ_LOCK_ASSERT_NOTOWNED(&(txq)->eq)
|
|
|
|
|
2015-12-03 00:02:01 +00:00
|
|
|
#define for_each_txq(vi, iter, q) \
|
2020-06-25 17:04:22 +00:00
|
|
|
for (q = &vi->adapter->sge.txq[vi->first_txq], iter = 0; \
|
2015-12-03 00:02:01 +00:00
|
|
|
iter < vi->ntxq; ++iter, ++q)
|
|
|
|
#define for_each_rxq(vi, iter, q) \
|
2020-06-25 17:04:22 +00:00
|
|
|
for (q = &vi->adapter->sge.rxq[vi->first_rxq], iter = 0; \
|
2015-12-03 00:02:01 +00:00
|
|
|
iter < vi->nrxq; ++iter, ++q)
|
|
|
|
#define for_each_ofld_txq(vi, iter, q) \
|
2020-06-25 17:04:22 +00:00
|
|
|
for (q = &vi->adapter->sge.ofld_txq[vi->first_ofld_txq], iter = 0; \
|
2015-12-03 00:02:01 +00:00
|
|
|
iter < vi->nofldtxq; ++iter, ++q)
|
|
|
|
#define for_each_ofld_rxq(vi, iter, q) \
|
2020-06-25 17:04:22 +00:00
|
|
|
for (q = &vi->adapter->sge.ofld_rxq[vi->first_ofld_rxq], iter = 0; \
|
2015-12-03 00:02:01 +00:00
|
|
|
iter < vi->nofldrxq; ++iter, ++q)
|
|
|
|
#define for_each_nm_txq(vi, iter, q) \
|
2020-06-25 17:04:22 +00:00
|
|
|
for (q = &vi->adapter->sge.nm_txq[vi->first_nm_txq], iter = 0; \
|
2016-06-23 02:53:00 +00:00
|
|
|
iter < vi->nnmtxq; ++iter, ++q)
|
2015-12-03 00:02:01 +00:00
|
|
|
#define for_each_nm_rxq(vi, iter, q) \
|
2020-06-25 17:04:22 +00:00
|
|
|
for (q = &vi->adapter->sge.nm_rxq[vi->first_nm_rxq], iter = 0; \
|
2016-06-23 02:53:00 +00:00
|
|
|
iter < vi->nnmrxq; ++iter, ++q)
|
2015-12-03 00:02:01 +00:00
|
|
|
#define for_each_vi(_pi, _iter, _vi) \
|
|
|
|
for ((_vi) = (_pi)->vi, (_iter) = 0; (_iter) < (_pi)->nvi; \
|
|
|
|
++(_iter), ++(_vi))
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2014-08-02 06:55:36 +00:00
|
|
|
#define IDXINCR(idx, incr, wrap) do { \
|
|
|
|
idx = wrap - idx > incr ? idx + incr : incr - (wrap - idx); \
|
2014-08-02 00:56:34 +00:00
|
|
|
} while (0)
|
|
|
|
#define IDXDIFF(head, tail, wrap) \
|
2014-08-02 06:55:36 +00:00
|
|
|
((head) >= (tail) ? (head) - (tail) : (wrap) - (tail) + (head))
|
2014-08-02 00:56:34 +00:00
|
|
|
|
2011-05-30 21:34:44 +00:00
|
|
|
/* One for errors, one for firmware events */
|
|
|
|
#define T4_EXTRA_INTR 2
|
2011-02-18 08:00:26 +00:00
|
|
|
|
Chelsio T4/T5 VF driver.
The cxgbev/cxlv driver supports Virtual Function devices for Chelsio
T4 and T4 adapters. The VF devices share most of their code with the
existing PF4 driver (cxgbe/cxl) and as such the VF device driver
currently depends on the PF4 driver.
Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf
PCI device driver that attaches to the VF device. It then creates
child cxgbev/cxlv devices representing ports assigned to the VF.
By default, the PF driver assigns a single port to each VF.
t4vf_hw.c contains VF-specific routines from the shared code used to
fetch VF-specific parameters from the firmware.
t4_vf.c contains the VF-specific PCI device driver and includes its
own attach routine.
VF devices are required to use a different firmware request when
transmitting packets (which in turn requires a different CPL message
to encapsulate messages). This alternate firmware request does not
permit chaining multiple packets in a single message, so each packet
results in a firmware request. In addition, the different CPL message
requires more detailed information when enabling hardware checksums,
so parse_pkt() on VF devices must examine L2 and L3 headers for all
packets (not just TSO packets) for VF devices. Finally, L2 checksums
on non-UDP/non-TCP packets do not work reliably (the firmware trashes
the IPv4 fragment field), so IPv4 checksums for such packets are
calculated in software.
Most of the other changes in the non-VF-specific code are to expose
various variables and functions private to the PF driver so that they
can be used by the VF driver.
Note that a limited subset of cxgbetool functions are supported on VF
devices including register dumps, scheduler classes, and clearing of
statistics. In addition, TOE is not supported on VF devices, only for
the PF interfaces.
Reviewed by: np
MFC after: 2 months
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
|
|
|
/* One for firmware events */
|
|
|
|
#define T4VF_EXTRA_INTR 1
|
|
|
|
|
2017-12-22 19:10:19 +00:00
|
|
|
static inline int
|
|
|
|
forwarding_intr_to_fwq(struct adapter *sc)
|
|
|
|
{
|
|
|
|
|
|
|
|
return (sc->intr_count == 1);
|
|
|
|
}
|
|
|
|
|
2021-04-28 04:33:10 +00:00
|
|
|
/* Works reliably inside a sync_op or with reg_lock held. */
|
|
|
|
static inline bool
|
|
|
|
hw_off_limits(struct adapter *sc)
|
|
|
|
{
|
2022-02-04 21:16:35 +00:00
|
|
|
int off_limits = atomic_load_int(&sc->error_flags) & HW_OFF_LIMITS;
|
|
|
|
|
|
|
|
return (__predict_false(off_limits != 0));
|
2021-04-28 04:33:10 +00:00
|
|
|
}
|
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
static inline uint32_t
|
|
|
|
t4_read_reg(struct adapter *sc, uint32_t reg)
|
|
|
|
{
|
2021-04-28 04:33:10 +00:00
|
|
|
if (hw_off_limits(sc))
|
|
|
|
MPASS(curthread == sc->reset_thread);
|
2011-02-18 08:00:26 +00:00
|
|
|
return bus_space_read_4(sc->bt, sc->bh, reg);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
|
|
|
t4_write_reg(struct adapter *sc, uint32_t reg, uint32_t val)
|
|
|
|
{
|
2021-04-28 04:33:10 +00:00
|
|
|
if (hw_off_limits(sc))
|
|
|
|
MPASS(curthread == sc->reset_thread);
|
2011-02-18 08:00:26 +00:00
|
|
|
bus_space_write_4(sc->bt, sc->bh, reg, val);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline uint64_t
|
|
|
|
t4_read_reg64(struct adapter *sc, uint32_t reg)
|
|
|
|
{
|
2021-04-28 04:33:10 +00:00
|
|
|
if (hw_off_limits(sc))
|
|
|
|
MPASS(curthread == sc->reset_thread);
|
2016-12-13 20:35:57 +00:00
|
|
|
#ifdef __LP64__
|
|
|
|
return bus_space_read_8(sc->bt, sc->bh, reg);
|
|
|
|
#else
|
|
|
|
return (uint64_t)bus_space_read_4(sc->bt, sc->bh, reg) +
|
|
|
|
((uint64_t)bus_space_read_4(sc->bt, sc->bh, reg + 4) << 32);
|
|
|
|
|
|
|
|
#endif
|
2011-02-18 08:00:26 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
|
|
|
t4_write_reg64(struct adapter *sc, uint32_t reg, uint64_t val)
|
|
|
|
{
|
2021-04-28 04:33:10 +00:00
|
|
|
if (hw_off_limits(sc))
|
|
|
|
MPASS(curthread == sc->reset_thread);
|
2016-12-13 20:35:57 +00:00
|
|
|
#ifdef __LP64__
|
|
|
|
bus_space_write_8(sc->bt, sc->bh, reg, val);
|
|
|
|
#else
|
|
|
|
bus_space_write_4(sc->bt, sc->bh, reg, val);
|
|
|
|
bus_space_write_4(sc->bt, sc->bh, reg + 4, val>> 32);
|
|
|
|
#endif
|
2011-02-18 08:00:26 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
|
|
|
t4_os_pci_read_cfg1(struct adapter *sc, int reg, uint8_t *val)
|
|
|
|
{
|
2021-04-28 04:33:10 +00:00
|
|
|
if (hw_off_limits(sc))
|
|
|
|
MPASS(curthread == sc->reset_thread);
|
2011-02-18 08:00:26 +00:00
|
|
|
*val = pci_read_config(sc->dev, reg, 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
|
|
|
t4_os_pci_write_cfg1(struct adapter *sc, int reg, uint8_t val)
|
|
|
|
{
|
2021-04-28 04:33:10 +00:00
|
|
|
if (hw_off_limits(sc))
|
|
|
|
MPASS(curthread == sc->reset_thread);
|
2011-02-18 08:00:26 +00:00
|
|
|
pci_write_config(sc->dev, reg, val, 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
|
|
|
t4_os_pci_read_cfg2(struct adapter *sc, int reg, uint16_t *val)
|
|
|
|
{
|
2012-06-19 07:34:13 +00:00
|
|
|
|
2021-04-28 04:33:10 +00:00
|
|
|
if (hw_off_limits(sc))
|
|
|
|
MPASS(curthread == sc->reset_thread);
|
2011-02-18 08:00:26 +00:00
|
|
|
*val = pci_read_config(sc->dev, reg, 2);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
|
|
|
t4_os_pci_write_cfg2(struct adapter *sc, int reg, uint16_t val)
|
|
|
|
{
|
2021-04-28 04:33:10 +00:00
|
|
|
if (hw_off_limits(sc))
|
|
|
|
MPASS(curthread == sc->reset_thread);
|
2011-02-18 08:00:26 +00:00
|
|
|
pci_write_config(sc->dev, reg, val, 2);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
|
|
|
t4_os_pci_read_cfg4(struct adapter *sc, int reg, uint32_t *val)
|
|
|
|
{
|
2021-04-28 04:33:10 +00:00
|
|
|
if (hw_off_limits(sc))
|
|
|
|
MPASS(curthread == sc->reset_thread);
|
2011-02-18 08:00:26 +00:00
|
|
|
*val = pci_read_config(sc->dev, reg, 4);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
|
|
|
t4_os_pci_write_cfg4(struct adapter *sc, int reg, uint32_t val)
|
|
|
|
{
|
2021-04-28 04:33:10 +00:00
|
|
|
if (hw_off_limits(sc))
|
|
|
|
MPASS(curthread == sc->reset_thread);
|
2011-02-18 08:00:26 +00:00
|
|
|
pci_write_config(sc->dev, reg, val, 4);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline struct port_info *
|
|
|
|
adap2pinfo(struct adapter *sc, int idx)
|
|
|
|
{
|
2012-06-19 07:34:13 +00:00
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
return (sc->port[idx]);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
2017-07-17 00:42:13 +00:00
|
|
|
t4_os_set_hw_addr(struct port_info *pi, uint8_t hw_addr[])
|
2011-02-18 08:00:26 +00:00
|
|
|
{
|
2012-06-19 07:34:13 +00:00
|
|
|
|
2017-07-17 00:42:13 +00:00
|
|
|
bcopy(hw_addr, pi->vi[0].hw_addr, ETHER_ADDR_LEN);
|
2011-02-18 08:00:26 +00:00
|
|
|
}
|
|
|
|
|
2013-03-30 02:26:20 +00:00
|
|
|
static inline int
|
|
|
|
tx_resume_threshold(struct sge_eq *eq)
|
2011-12-16 02:09:51 +00:00
|
|
|
{
|
2012-06-19 07:34:13 +00:00
|
|
|
|
2014-12-31 23:19:16 +00:00
|
|
|
/* not quite the same as qsize / 4, but this will do. */
|
|
|
|
return (eq->sidx / 4);
|
2011-12-16 02:09:51 +00:00
|
|
|
}
|
|
|
|
|
2016-03-08 02:04:05 +00:00
|
|
|
static inline int
|
|
|
|
t4_use_ldst(struct adapter *sc)
|
|
|
|
{
|
|
|
|
|
|
|
|
#ifdef notyet
|
|
|
|
return (sc->flags & FW_OK || !sc->use_bd);
|
|
|
|
#else
|
|
|
|
return (0);
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2019-02-01 20:42:49 +00:00
|
|
|
static inline void
|
|
|
|
CH_DUMP_MBOX(struct adapter *sc, int mbox, const int reg,
|
|
|
|
const char *msg, const __be64 *const p, const bool err)
|
|
|
|
{
|
|
|
|
|
|
|
|
if (!(sc->debug_flags & DF_DUMP_MBOX) && !err)
|
|
|
|
return;
|
|
|
|
if (p != NULL) {
|
|
|
|
log(err ? LOG_ERR : LOG_DEBUG,
|
|
|
|
"%s: mbox %u %s %016llx %016llx %016llx %016llx "
|
|
|
|
"%016llx %016llx %016llx %016llx\n",
|
|
|
|
device_get_nameunit(sc->dev), mbox, msg,
|
|
|
|
(long long)be64_to_cpu(p[0]), (long long)be64_to_cpu(p[1]),
|
|
|
|
(long long)be64_to_cpu(p[2]), (long long)be64_to_cpu(p[3]),
|
|
|
|
(long long)be64_to_cpu(p[4]), (long long)be64_to_cpu(p[5]),
|
|
|
|
(long long)be64_to_cpu(p[6]), (long long)be64_to_cpu(p[7]));
|
|
|
|
} else {
|
|
|
|
log(err ? LOG_ERR : LOG_DEBUG,
|
|
|
|
"%s: mbox %u %s %016llx %016llx %016llx %016llx "
|
|
|
|
"%016llx %016llx %016llx %016llx\n",
|
|
|
|
device_get_nameunit(sc->dev), mbox, msg,
|
|
|
|
(long long)t4_read_reg64(sc, reg),
|
|
|
|
(long long)t4_read_reg64(sc, reg + 8),
|
|
|
|
(long long)t4_read_reg64(sc, reg + 16),
|
|
|
|
(long long)t4_read_reg64(sc, reg + 24),
|
|
|
|
(long long)t4_read_reg64(sc, reg + 32),
|
|
|
|
(long long)t4_read_reg64(sc, reg + 40),
|
|
|
|
(long long)t4_read_reg64(sc, reg + 48),
|
|
|
|
(long long)t4_read_reg64(sc, reg + 56));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2011-03-05 03:06:38 +00:00
|
|
|
/* t4_main.c */
|
2017-11-15 23:48:02 +00:00
|
|
|
extern int t4_ntxq;
|
|
|
|
extern int t4_nrxq;
|
Chelsio T4/T5 VF driver.
The cxgbev/cxlv driver supports Virtual Function devices for Chelsio
T4 and T4 adapters. The VF devices share most of their code with the
existing PF4 driver (cxgbe/cxl) and as such the VF device driver
currently depends on the PF4 driver.
Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf
PCI device driver that attaches to the VF device. It then creates
child cxgbev/cxlv devices representing ports assigned to the VF.
By default, the PF driver assigns a single port to each VF.
t4vf_hw.c contains VF-specific routines from the shared code used to
fetch VF-specific parameters from the firmware.
t4_vf.c contains the VF-specific PCI device driver and includes its
own attach routine.
VF devices are required to use a different firmware request when
transmitting packets (which in turn requires a different CPL message
to encapsulate messages). This alternate firmware request does not
permit chaining multiple packets in a single message, so each packet
results in a firmware request. In addition, the different CPL message
requires more detailed information when enabling hardware checksums,
so parse_pkt() on VF devices must examine L2 and L3 headers for all
packets (not just TSO packets) for VF devices. Finally, L2 checksums
on non-UDP/non-TCP packets do not work reliably (the firmware trashes
the IPv4 fragment field), so IPv4 checksums for such packets are
calculated in software.
Most of the other changes in the non-VF-specific code are to expose
various variables and functions private to the PF driver so that they
can be used by the VF driver.
Note that a limited subset of cxgbetool functions are supported on VF
devices including register dumps, scheduler classes, and clearing of
statistics. In addition, TOE is not supported on VF devices, only for
the PF interfaces.
Reviewed by: np
MFC after: 2 months
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
|
|
|
extern int t4_intr_types;
|
2017-11-15 23:48:02 +00:00
|
|
|
extern int t4_tmr_idx;
|
|
|
|
extern int t4_pktc_idx;
|
Chelsio T4/T5 VF driver.
The cxgbev/cxlv driver supports Virtual Function devices for Chelsio
T4 and T4 adapters. The VF devices share most of their code with the
existing PF4 driver (cxgbe/cxl) and as such the VF device driver
currently depends on the PF4 driver.
Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf
PCI device driver that attaches to the VF device. It then creates
child cxgbev/cxlv devices representing ports assigned to the VF.
By default, the PF driver assigns a single port to each VF.
t4vf_hw.c contains VF-specific routines from the shared code used to
fetch VF-specific parameters from the firmware.
t4_vf.c contains the VF-specific PCI device driver and includes its
own attach routine.
VF devices are required to use a different firmware request when
transmitting packets (which in turn requires a different CPL message
to encapsulate messages). This alternate firmware request does not
permit chaining multiple packets in a single message, so each packet
results in a firmware request. In addition, the different CPL message
requires more detailed information when enabling hardware checksums,
so parse_pkt() on VF devices must examine L2 and L3 headers for all
packets (not just TSO packets) for VF devices. Finally, L2 checksums
on non-UDP/non-TCP packets do not work reliably (the firmware trashes
the IPv4 fragment field), so IPv4 checksums for such packets are
calculated in software.
Most of the other changes in the non-VF-specific code are to expose
various variables and functions private to the PF driver so that they
can be used by the VF driver.
Note that a limited subset of cxgbetool functions are supported on VF
devices including register dumps, scheduler classes, and clearing of
statistics. In addition, TOE is not supported on VF devices, only for
the PF interfaces.
Reviewed by: np
MFC after: 2 months
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
|
|
|
extern unsigned int t4_qsize_rxq;
|
|
|
|
extern unsigned int t4_qsize_txq;
|
|
|
|
extern device_method_t cxgbe_methods[];
|
|
|
|
|
2011-02-18 08:00:26 +00:00
|
|
|
int t4_os_find_pci_capability(struct adapter *, int);
|
|
|
|
int t4_os_pci_save_state(struct adapter *);
|
|
|
|
int t4_os_pci_restore_state(struct adapter *);
|
2017-08-12 14:02:19 +00:00
|
|
|
void t4_os_portmod_changed(struct port_info *);
|
|
|
|
void t4_os_link_changed(struct port_info *);
|
2011-12-16 02:09:51 +00:00
|
|
|
void t4_iterate(void (*)(struct adapter *, void *), void *);
|
2016-09-16 00:08:37 +00:00
|
|
|
void t4_init_devnames(struct adapter *);
|
Chelsio T4/T5 VF driver.
The cxgbev/cxlv driver supports Virtual Function devices for Chelsio
T4 and T4 adapters. The VF devices share most of their code with the
existing PF4 driver (cxgbe/cxl) and as such the VF device driver
currently depends on the PF4 driver.
Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf
PCI device driver that attaches to the VF device. It then creates
child cxgbev/cxlv devices representing ports assigned to the VF.
By default, the PF driver assigns a single port to each VF.
t4vf_hw.c contains VF-specific routines from the shared code used to
fetch VF-specific parameters from the firmware.
t4_vf.c contains the VF-specific PCI device driver and includes its
own attach routine.
VF devices are required to use a different firmware request when
transmitting packets (which in turn requires a different CPL message
to encapsulate messages). This alternate firmware request does not
permit chaining multiple packets in a single message, so each packet
results in a firmware request. In addition, the different CPL message
requires more detailed information when enabling hardware checksums,
so parse_pkt() on VF devices must examine L2 and L3 headers for all
packets (not just TSO packets) for VF devices. Finally, L2 checksums
on non-UDP/non-TCP packets do not work reliably (the firmware trashes
the IPv4 fragment field), so IPv4 checksums for such packets are
calculated in software.
Most of the other changes in the non-VF-specific code are to expose
various variables and functions private to the PF driver so that they
can be used by the VF driver.
Note that a limited subset of cxgbetool functions are supported on VF
devices including register dumps, scheduler classes, and clearing of
statistics. In addition, TOE is not supported on VF devices, only for
the PF interfaces.
Reviewed by: np
MFC after: 2 months
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D7599
2016-09-07 18:13:57 +00:00
|
|
|
void t4_add_adapter(struct adapter *);
|
|
|
|
int t4_detach_common(device_t);
|
|
|
|
int t4_map_bars_0_and_4(struct adapter *);
|
|
|
|
int t4_map_bar_2(struct adapter *);
|
|
|
|
int t4_setup_intr_handlers(struct adapter *);
|
|
|
|
void t4_sysctls(struct adapter *);
|
2015-12-03 00:02:01 +00:00
|
|
|
int begin_synchronized_op(struct adapter *, struct vi_info *, int, char *);
|
|
|
|
void doom_vi(struct adapter *, struct vi_info *);
|
2013-01-10 23:56:50 +00:00
|
|
|
void end_synchronized_op(struct adapter *, int);
|
2014-05-27 18:18:41 +00:00
|
|
|
int update_mac_settings(struct ifnet *, int);
|
2021-04-26 21:03:06 +00:00
|
|
|
int adapter_init(struct adapter *);
|
|
|
|
int vi_init(struct vi_info *);
|
2015-12-03 00:02:01 +00:00
|
|
|
void vi_sysctls(struct vi_info *);
|
2018-04-03 01:22:15 +00:00
|
|
|
int rw_via_memwin(struct adapter *, int, uint32_t, uint32_t *, int, int);
|
2018-04-26 19:00:35 +00:00
|
|
|
int alloc_atid(struct adapter *, void *);
|
|
|
|
void *lookup_atid(struct adapter *, int);
|
|
|
|
void free_atid(struct adapter *, int);
|
2018-04-26 22:04:21 +00:00
|
|
|
void release_tid(struct adapter *, int, struct sge_wrq *);
|
2018-08-23 00:58:10 +00:00
|
|
|
int cxgbe_media_change(struct ifnet *);
|
|
|
|
void cxgbe_media_status(struct ifnet *, struct ifmediareq *);
|
2022-02-04 21:16:35 +00:00
|
|
|
void t4_os_cim_err(struct adapter *);
|
2014-05-27 18:18:41 +00:00
|
|
|
|
NIC KTLS for Chelsio T6 adapters.
This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters.
Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE
connections.
NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment
the encrypted TLS frames output by the crypto engine. Instead, the
TOE is placed into a special setup to permit "dummy" connections to be
associated with regular sockets using KTLS. This permits using the
TOE to segment the encrypted TLS records. However, this approach does
have some limitations:
1) Regular TOE sockets cannot be used when the TOE is in this special
mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but
not both at the same time.
2) In NIC KTLS mode, the TOE is only able to accept a per-connection
timestamp offset that varies in the upper 4 bits. Put another way,
only connections whose timestamp offset has the 28 lower bits
cleared can use NIC KTLS and generate correct timestamps. The
driver will refuse to enable NIC KTLS on connections with a
timestamp offset with any of the lower 28 bits set. To use NIC
KTLS, users can either disable TCP timestamps by setting the
net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the
tcp_new_ts_offset() function to clear the lower 28 bits of the
generated offset.
3) Because the TCP segmentation relies on fields mirrored in a TCB in
the TOE, not all fields in a TCP packet can be sent in the TCP
segments generated from a TLS record. Specifically, for packets
containing TCP options other than timestamps, the driver will
inject an "empty" TCP packet holding the requested options (e.g. a
SACK scoreboard) along with the segments from the TLS record.
These empty TCP packets are counted by the
dev.cc.N.txq.M.kern_tls_options sysctls.
Unlike TOE TLS which is able to buffer encrypted TLS records in
on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS
records for retransmit requests as well as non-retransmit requests
that do not include the start of a TLS record but do include the
trailer. The T6 NIC KTLS code tries to optimize some of the cases for
requests to transmit partial TLS records. In particular it attempts
to minimize sending "waste" bytes that have to be given as input to
the crypto engine but are not needed on the wire to satisfy mbufs sent
from the TCP stack down to the driver.
TCP packets for TLS requests are broken down into the following
classes (with associated counters):
- Mbufs that send an entire TLS record in full do not have any waste
bytes (dev.cc.N.txq.M.kern_tls_full).
- Mbufs that send a short TLS record that ends before the end of the
trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC,
the encryption must always start at the beginning, so if the mbuf
starts at an offset into the TLS record, the offset bytes will be
"waste" bytes. For sockets using AES-GCM, the encryption can start
at the 16 byte block before the starting offset capping the waste at
15 bytes.
- Mbufs that send a partial TLS record that has a non-zero starting
offset but ends at the end of the trailer
(dev.cc.N.txq.M.kern_tls_partial). In order to compute the
authentication hash stored in the trailer, the entire TLS record
must be sent as input to the crypto engine, so the bytes before the
offset are always "waste" bytes.
In addition, other per-txq sysctls are provided:
- dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq
using AES-CBC.
- dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq
using AES-GCM.
- dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to
compensate for the TOE engine not being able to set FIN on the last
segment of a TLS record if the TLS record mbuf had FIN set.
- dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this
txq including full, short, and partial records.
- dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header
and payload) sent for TLS record requests.
- dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS
record requests.
To enable NIC KTLS with T6, set the following tunables prior to
loading the cxgbe(4) driver:
hw.cxgbe.config_file=kern_tls
hw.cxgbe.kern_tls=1
Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21962
2019-11-21 19:30:31 +00:00
|
|
|
#ifdef KERN_TLS
|
|
|
|
/* t4_kern_tls.c */
|
|
|
|
int cxgbe_tls_tag_alloc(struct ifnet *, union if_snd_tag_alloc_params *,
|
|
|
|
struct m_snd_tag **);
|
|
|
|
void t6_ktls_modload(void);
|
|
|
|
void t6_ktls_modunload(void);
|
|
|
|
int t6_ktls_try(struct ifnet *, struct socket *, struct ktls_session *);
|
|
|
|
int t6_ktls_parse_pkt(struct mbuf *, int *, int *);
|
|
|
|
int t6_ktls_write_wr(struct sge_txq *, void *, struct mbuf *, u_int, u_int);
|
|
|
|
#endif
|
|
|
|
|
2019-11-13 00:53:45 +00:00
|
|
|
/* t4_keyctx.c */
|
|
|
|
struct auth_hash;
|
|
|
|
union authctx;
|
2021-06-08 18:11:51 +00:00
|
|
|
#ifdef KERN_TLS
|
|
|
|
struct ktls_session;
|
|
|
|
struct tls_key_req;
|
|
|
|
struct tls_keyctx;
|
|
|
|
#endif
|
2019-11-13 00:53:45 +00:00
|
|
|
|
|
|
|
void t4_aes_getdeckey(void *, const void *, unsigned int);
|
|
|
|
void t4_copy_partial_hash(int, union authctx *, void *);
|
|
|
|
void t4_init_gmac_hash(const char *, int, char *);
|
2021-07-26 20:41:05 +00:00
|
|
|
void t4_init_hmac_digest(const struct auth_hash *, u_int, const char *, int,
|
|
|
|
char *);
|
2021-06-08 18:11:51 +00:00
|
|
|
#ifdef KERN_TLS
|
|
|
|
u_int t4_tls_key_info_size(const struct ktls_session *);
|
|
|
|
int t4_tls_proto_ver(const struct ktls_session *);
|
|
|
|
int t4_tls_cipher_mode(const struct ktls_session *);
|
|
|
|
int t4_tls_auth_mode(const struct ktls_session *);
|
|
|
|
int t4_tls_hmac_ctrl(const struct ktls_session *);
|
|
|
|
void t4_tls_key_ctx(const struct ktls_session *, int, struct tls_keyctx *);
|
|
|
|
int t4_alloc_tls_keyid(struct adapter *);
|
|
|
|
void t4_free_tls_keyid(struct adapter *, int);
|
|
|
|
void t4_write_tlskey_wr(const struct ktls_session *, int, int, int, int,
|
|
|
|
struct tls_key_req *);
|
|
|
|
#endif
|
2019-11-13 00:53:45 +00:00
|
|
|
|
2014-05-27 18:18:41 +00:00
|
|
|
#ifdef DEV_NETMAP
|
|
|
|
/* t4_netmap.c */
|
2018-08-11 04:55:47 +00:00
|
|
|
struct sge_nm_rxq;
|
2016-06-23 02:53:00 +00:00
|
|
|
void cxgbe_nm_attach(struct vi_info *);
|
|
|
|
void cxgbe_nm_detach(struct vi_info *);
|
2018-08-11 04:55:47 +00:00
|
|
|
void service_nm_rxq(struct sge_nm_rxq *);
|
2021-04-26 21:03:06 +00:00
|
|
|
int alloc_nm_rxq(struct vi_info *, struct sge_nm_rxq *, int, int);
|
2020-12-03 08:30:29 +00:00
|
|
|
int free_nm_rxq(struct vi_info *, struct sge_nm_rxq *);
|
2021-04-26 21:03:06 +00:00
|
|
|
int alloc_nm_txq(struct vi_info *, struct sge_nm_txq *, int, int);
|
2020-12-03 08:30:29 +00:00
|
|
|
int free_nm_txq(struct vi_info *, struct sge_nm_txq *);
|
2014-05-27 18:18:41 +00:00
|
|
|
#endif
|
2011-02-18 08:00:26 +00:00
|
|
|
|
2011-03-05 03:06:38 +00:00
|
|
|
/* t4_sge.c */
|
2011-03-08 03:04:07 +00:00
|
|
|
void t4_sge_modload(void);
|
2014-07-23 22:29:22 +00:00
|
|
|
void t4_sge_modunload(void);
|
|
|
|
uint64_t t4_sge_extfree_refs(void);
|
2013-03-30 02:26:20 +00:00
|
|
|
void t4_tweak_chip_settings(struct adapter *);
|
2021-02-18 09:15:46 +00:00
|
|
|
int t4_verify_chip_settings(struct adapter *);
|
|
|
|
void t4_init_rx_buf_info(struct adapter *);
|
2011-02-18 08:00:26 +00:00
|
|
|
int t4_create_dma_tag(struct adapter *);
|
2013-07-31 05:12:51 +00:00
|
|
|
void t4_sge_sysctls(struct adapter *, struct sysctl_ctx_list *,
|
|
|
|
struct sysctl_oid_list *);
|
2011-02-18 08:00:26 +00:00
|
|
|
int t4_destroy_dma_tag(struct adapter *);
|
2020-12-03 08:30:29 +00:00
|
|
|
int alloc_ring(struct adapter *, size_t, bus_dma_tag_t *, bus_dmamap_t *,
|
|
|
|
bus_addr_t *, void **);
|
|
|
|
int free_ring(struct adapter *, bus_dma_tag_t, bus_dmamap_t, bus_addr_t,
|
|
|
|
void *);
|
2021-04-26 21:03:06 +00:00
|
|
|
void free_fl_buffers(struct adapter *, struct sge_fl *);
|
2011-04-19 22:08:28 +00:00
|
|
|
int t4_setup_adapter_queues(struct adapter *);
|
|
|
|
int t4_teardown_adapter_queues(struct adapter *);
|
2015-12-03 00:02:01 +00:00
|
|
|
int t4_setup_vi_queues(struct vi_info *);
|
|
|
|
int t4_teardown_vi_queues(struct vi_info *);
|
2011-02-18 08:00:26 +00:00
|
|
|
void t4_intr_all(void *);
|
2011-05-30 21:34:44 +00:00
|
|
|
void t4_intr(void *);
|
2018-08-11 04:55:47 +00:00
|
|
|
#ifdef DEV_NETMAP
|
|
|
|
void t4_nm_intr(void *);
|
2016-06-23 02:53:00 +00:00
|
|
|
void t4_vi_intr(void *);
|
2018-08-11 04:55:47 +00:00
|
|
|
#endif
|
2011-02-18 08:00:26 +00:00
|
|
|
void t4_intr_err(void *);
|
|
|
|
void t4_intr_evt(void *);
|
2012-06-19 07:34:13 +00:00
|
|
|
void t4_wrq_tx_locked(struct adapter *, struct sge_wrq *, struct wrqe *);
|
2011-02-18 08:00:26 +00:00
|
|
|
void t4_update_fl_bufsize(struct ifnet *);
|
2018-11-06 00:11:36 +00:00
|
|
|
struct mbuf *alloc_wr_mbuf(int, int);
|
2020-09-22 04:16:40 +00:00
|
|
|
int parse_pkt(struct mbuf **, bool);
|
2014-12-31 23:19:16 +00:00
|
|
|
void *start_wrq_wr(struct sge_wrq *, int, struct wrq_cookie *);
|
|
|
|
void commit_wrq_wr(struct sge_wrq *, void *, struct wrq_cookie *);
|
2015-07-06 20:56:59 +00:00
|
|
|
int tnl_cong(struct port_info *, int);
|
2018-04-30 15:18:38 +00:00
|
|
|
void t4_register_an_handler(an_handler_t);
|
|
|
|
void t4_register_fw_msg_handler(int, fw_msg_handler_t);
|
|
|
|
void t4_register_cpl_handler(int, cpl_handler_t);
|
|
|
|
void t4_register_shared_cpl_handler(int, cpl_handler_t, int);
|
2018-05-24 10:18:14 +00:00
|
|
|
#ifdef RATELIMIT
|
|
|
|
int ethofld_transmit(struct ifnet *, struct mbuf *);
|
2019-10-22 20:41:54 +00:00
|
|
|
void send_etid_flush_wr(struct cxgbe_rate_tag *);
|
2018-05-24 10:18:14 +00:00
|
|
|
#endif
|
2011-12-16 02:09:51 +00:00
|
|
|
|
Add support for packet-sniffing tracers to cxgbe(4). This works with
all T4 and T5 based cards and is useful for analyzing TSO, LRO, TOE, and
for general purpose monitoring without tapping any cxgbe or cxl ifnet
directly.
Tracers on the T4/T5 chips provide access to Ethernet frames exactly as
they were received from or transmitted on the wire. On transmit, a
tracer will capture a frame after TSO segmentation, hw VLAN tag
insertion, hw L3 & L4 checksum insertion, etc. It will also capture
frames generated by the TCP offload engine (TOE traffic is normally
invisible to the kernel). On receive, a tracer will capture a frame
before hw VLAN extraction, runt filtering, other badness filtering,
before the steering/drop/L2-rewrite filters or the TOE have had a go at
it, and of course before sw LRO in the driver.
There are 4 tracers on a chip. A tracer can trace only in one direction
(tx or rx). For now cxgbetool will set up tracers to capture the first
128B of every transmitted or received frame on a given port. This is a
small subset of what the hardware can do. A pseudo ifnet with the same
name as the nexus driver (t4nex0 or t5nex0) will be created for tracing.
The data delivered to this ifnet is an additional copy made inside the
chip. Normal delivery to cxgbe<n> or cxl<n> will be made as usual.
/* watch cxl0, which is the first port hanging off t5nex0. */
# cxgbetool t5nex0 tracer 0 tx0 (watch what cxl0 is transmitting)
# cxgbetool t5nex0 tracer 1 rx0 (watch what cxl0 is receiving)
# cxgbetool t5nex0 tracer list
# tcpdump -i t5nex0 <== all that cxl0 sees and puts on the wire
If you were doing TSO, a tcpdump on cxl0 may have shown you ~64K
"frames" with no L3/L4 checksum but this will show you the frames that
were actually transmitted.
/* all done */
# cxgbetool t5nex0 tracer 0 disable
# cxgbetool t5nex0 tracer 1 disable
# cxgbetool t5nex0 tracer list
# ifconfig t5nex0 destroy
2013-07-26 22:04:11 +00:00
|
|
|
/* t4_tracer.c */
|
|
|
|
struct t4_tracer;
|
|
|
|
void t4_tracer_modload(void);
|
|
|
|
void t4_tracer_modunload(void);
|
|
|
|
void t4_tracer_port_detach(struct adapter *);
|
|
|
|
int t4_get_tracer(struct adapter *, struct t4_tracer *);
|
|
|
|
int t4_set_tracer(struct adapter *, struct t4_tracer *);
|
|
|
|
int t4_trace_pkt(struct sge_iq *, const struct rss_header *, struct mbuf *);
|
|
|
|
int t5_trace_pkt(struct sge_iq *, const struct rss_header *, struct mbuf *);
|
|
|
|
|
2017-05-02 20:38:10 +00:00
|
|
|
/* t4_sched.c */
|
|
|
|
int t4_set_sched_class(struct adapter *, struct t4_sched_params *);
|
|
|
|
int t4_set_sched_queue(struct adapter *, struct t4_sched_queue *);
|
|
|
|
int t4_init_tx_sched(struct adapter *);
|
|
|
|
int t4_free_tx_sched(struct adapter *);
|
|
|
|
void t4_update_tx_sched(struct adapter *);
|
|
|
|
int t4_reserve_cl_rl_kbps(struct adapter *, int, u_int, int *);
|
2018-08-06 23:21:13 +00:00
|
|
|
void t4_release_cl_rl(struct adapter *, int, int);
|
|
|
|
int sysctl_tc(SYSCTL_HANDLER_ARGS);
|
|
|
|
int sysctl_tc_params(SYSCTL_HANDLER_ARGS);
|
2018-05-18 06:09:15 +00:00
|
|
|
#ifdef RATELIMIT
|
|
|
|
void t4_init_etid_table(struct adapter *);
|
|
|
|
void t4_free_etid_table(struct adapter *);
|
2019-10-22 20:41:54 +00:00
|
|
|
struct cxgbe_rate_tag *lookup_etid(struct adapter *, int);
|
|
|
|
int cxgbe_rate_tag_alloc(struct ifnet *, union if_snd_tag_alloc_params *,
|
2018-05-18 06:09:15 +00:00
|
|
|
struct m_snd_tag **);
|
2019-10-22 20:41:54 +00:00
|
|
|
void cxgbe_rate_tag_free_locked(struct cxgbe_rate_tag *);
|
2019-08-01 14:17:31 +00:00
|
|
|
void cxgbe_ratelimit_query(struct ifnet *, struct if_ratelimit_query_results *);
|
2018-05-18 06:09:15 +00:00
|
|
|
#endif
|
2017-05-02 20:38:10 +00:00
|
|
|
|
2018-05-01 20:17:22 +00:00
|
|
|
/* t4_filter.c */
|
|
|
|
int get_filter_mode(struct adapter *, uint32_t *);
|
|
|
|
int set_filter_mode(struct adapter *, uint32_t);
|
2021-02-19 21:47:18 +00:00
|
|
|
int set_filter_mask(struct adapter *, uint32_t);
|
2018-05-01 20:17:22 +00:00
|
|
|
int get_filter(struct adapter *, struct t4_filter *);
|
|
|
|
int set_filter(struct adapter *, struct t4_filter *);
|
|
|
|
int del_filter(struct adapter *, struct t4_filter *);
|
|
|
|
int t4_filter_rpl(struct sge_iq *, const struct rss_header *, struct mbuf *);
|
cxgbe(4): Add support for hash filters.
These filters reside in the card's memory instead of its TCAM and can be
configured via a new "hashfilter" subcommand in cxgbetool. Hash and
normal TCAM filters can be used together. The hardware does an
exact-match of packet fields for hash filters, unlike the masked match
performed for TCAM filters. Any T5/T6 card with memory can support at
least half a million hash filters. The sample config file with the
driver configures 512K of these, it is possible to double this to 1
million+ in some cases.
The chip does an exact-match of fields of incoming datagrams with hash
filters and performs the action configured for the filter if it matches.
The fields to match are specified in a "filter mask" in the firmware
config file. The filter mask always includes the 5-tuple (sip, dip,
sport, dport, ipproto). It can, optionally, also include any subset of
the filter mode (see filterMode and filterMask in the firmware config
file).
For example:
filterMode = fragmentation, mpshittype, protocol, vlan, port, fcoe
filterMask = protocol, port, vlan
Exact values of the 5-tuple, the physical port, and VLAN tag would have
to be provided while setting up a hash filter with the chip
configuration above.
Hash filters support all actions supported by TCAM filters. A packet
that hits a hash filter can be dropped, let through (with optional
steering to a specific queue or RSS region), switched out of another
port (with optional L2 rewrite of DMAC, SMAC, VLAN tag), or get NAT'ed.
(Support for some of these will show up in the driver in a follow-up
commit very shortly).
Sponsored by: Chelsio Communications
2018-05-09 04:09:49 +00:00
|
|
|
int t4_hashfilter_ao_rpl(struct sge_iq *, const struct rss_header *, struct mbuf *);
|
|
|
|
int t4_hashfilter_tcb_rpl(struct sge_iq *, const struct rss_header *, struct mbuf *);
|
|
|
|
int t4_del_hashfilter_rpl(struct sge_iq *, const struct rss_header *, struct mbuf *);
|
2018-08-15 03:03:01 +00:00
|
|
|
void free_hftid_hash(struct tid_info *);
|
2018-05-01 20:17:22 +00:00
|
|
|
|
2012-06-19 07:34:13 +00:00
|
|
|
static inline struct wrqe *
|
|
|
|
alloc_wrqe(int wr_len, struct sge_wrq *wrq)
|
2011-12-16 02:09:51 +00:00
|
|
|
{
|
2012-06-19 07:34:13 +00:00
|
|
|
int len = offsetof(struct wrqe, wr) + wr_len;
|
|
|
|
struct wrqe *wr;
|
|
|
|
|
|
|
|
wr = malloc(len, M_CXGBE, M_NOWAIT);
|
|
|
|
if (__predict_false(wr == NULL))
|
|
|
|
return (NULL);
|
|
|
|
wr->wr_len = wr_len;
|
|
|
|
wr->wrq = wrq;
|
|
|
|
return (wr);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void *
|
|
|
|
wrtod(struct wrqe *wr)
|
|
|
|
{
|
|
|
|
return (&wr->wr[0]);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
|
|
|
free_wrqe(struct wrqe *wr)
|
|
|
|
{
|
|
|
|
free(wr, M_CXGBE);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
|
|
|
t4_wrq_tx(struct adapter *sc, struct wrqe *wr)
|
|
|
|
{
|
|
|
|
struct sge_wrq *wrq = wr->wrq;
|
2011-12-16 02:09:51 +00:00
|
|
|
|
|
|
|
TXQ_LOCK(wrq);
|
2012-06-19 07:34:13 +00:00
|
|
|
t4_wrq_tx_locked(sc, wrq, wr);
|
2011-12-16 02:09:51 +00:00
|
|
|
TXQ_UNLOCK(wrq);
|
|
|
|
}
|
|
|
|
|
2018-04-03 01:22:15 +00:00
|
|
|
static inline int
|
|
|
|
read_via_memwin(struct adapter *sc, int idx, uint32_t addr, uint32_t *val,
|
|
|
|
int len)
|
|
|
|
{
|
|
|
|
|
|
|
|
return (rw_via_memwin(sc, idx, addr, val, len, 0));
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline int
|
|
|
|
write_via_memwin(struct adapter *sc, int idx, uint32_t addr,
|
|
|
|
const uint32_t *val, int len)
|
|
|
|
{
|
|
|
|
|
|
|
|
return (rw_via_memwin(sc, idx, addr, (void *)(uintptr_t)val, len, 1));
|
|
|
|
}
|
2020-06-23 07:33:29 +00:00
|
|
|
|
|
|
|
/* Number of len16 -> number of descriptors */
|
|
|
|
static inline int
|
|
|
|
tx_len16_to_desc(int len16)
|
|
|
|
{
|
|
|
|
|
|
|
|
return (howmany(len16, EQ_ESIZE / 16));
|
|
|
|
}
|
2011-02-18 08:00:26 +00:00
|
|
|
#endif
|