d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

cxgbe(4): Combine all _10g and _1g tunables and drop the suffix from

Browse Source 
their names.  The finer-grained knobs weren't practically useful.

Sponsored by:	Chelsio Communications
This commit is contained in:
Navdeep Parhar 2017-11-15 23:48:02 +00:00
parent f95f6841c8
commit 8c61c6bbda
5 changed files with 159 additions and 350 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

37
share/man/man4/cxgbe.4
View File

@ -172,37 +172,22 @@ types.
A negative value for such a tunable instructs the driver to create
up to that many queues if there are enough CPU cores available.
.Bl -tag -width indent
.It Va hw.cxgbe.ntxq10g
Number of tx queues used for a 10Gb or higher-speed port.
.It Va hw.cxgbe.ntxq
Number of NIC tx queues used for a port.
The default is 16 or the number
of CPU cores in the system, whichever is less.
.It Va hw.cxgbe.nrxq10g
Number of rx queues used for a 10Gb or higher-speed port.
.It Va hw.cxgbe.nrxq
Number of NIC rx queues used for a port.
The default is 8 or the number
of CPU cores in the system, whichever is less.
.It Va hw.cxgbe.ntxq1g
Number of tx queues used for a 1Gb port.
The default is 4 or the number
of CPU cores in the system, whichever is less.
.It Va hw.cxgbe.nrxq1g
Number of rx queues used for a 1Gb port.
The default is 2 or the number
of CPU cores in the system, whichever is less.
.It Va hw.cxgbe.nofldtxq10g
Number of TOE tx queues used for a 10Gb or higher-speed port.
.It Va hw.cxgbe.nofldtxq
Number of TOE tx queues used for a port.
The default is 8 or the
number of CPU cores in the system, whichever is less.
.It Va hw.cxgbe.nofldrxq10g
Number of TOE rx queues used for a 10Gb or higher-speed port.
.It Va hw.cxgbe.nofldrxq
Number of TOE rx queues used for a port.
The default is 2 or the
number of CPU cores in the system, whichever is less.
.It Va hw.cxgbe.nofldtxq1g
Number of TOE tx queues used for a 1Gb port.
The default is 2 or the
number of CPU cores in the system, whichever is less.
.It Va hw.cxgbe.nofldrxq1g
Number of TOE rx queues used for a 1Gb port.
The default is 1.
.It Va hw.cxgbe.num_vis
Number of virtual interfaces (VIs) created for each port.
Each virtual interface creates a separate network interface.
@ -213,8 +198,7 @@ name from the table above.
Additional virtual interfaces use a single pair of queues
for rx and tx as well an additional pair of queues for TOE rx and tx.
The default is 1.
.It Va hw.cxgbe.holdoff_timer_idx_10G
.It Va hw.cxgbe.holdoff_timer_idx_1G
.It Va hw.cxgbe.holdoff_timer_idx
.It Va hw.cxgbe.holdoff_timer_idx_ofld
Timer index value used to delay interrupts.
The holdoff timer list has the values 1, 5, 10, 50, 100, and 200
@ -224,8 +208,7 @@ holdoff_timer_idx_ofld applies to queues used for TOE rx.
The default value is 1 which means the timer value is 5us.
Different interfaces can be assigned different values at any time via the
dev.<port>.X.holdoff_tmr_idx and dev.<port>.X.holdoff_tmr_idx_ofld sysctls.
.It Va hw.cxgbe.holdoff_pktc_idx_10G
.It Va hw.cxgbe.holdoff_pktc_idx_1G
.It Va hw.cxgbe.holdoff_pktc_idx
.It Va hw.cxgbe.holdoff_pktc_idx_ofld
Packet-count index value used to delay interrupts.
The packet-count list has the values 1, 8, 16, and 32 by default,

22
share/man/man4/cxgbev.4
View File

@ -172,24 +172,15 @@ Tunables can be set at the
prompt before booting the kernel or stored in
.Xr loader.conf 5 .
.Bl -tag -width indent
.It Va hw.cxgbe.ntxq10g
Number of tx queues used for a 10Gb or higher-speed port.
.It Va hw.cxgbe.ntxq
Number of tx queues used for a port.
The default is 16 or the number
of CPU cores in the system, whichever is less.
.It Va hw.cxgbe.nrxq10g
Number of rx queues used for a 10Gb or higher-speed port.
.It Va hw.cxgbe.nrxq
Number of rx queues used for a port.
The default is 8 or the number
of CPU cores in the system, whichever is less.
.It Va hw.cxgbe.ntxq1g
Number of tx queues used for a 1Gb port.
The default is 4 or the number
of CPU cores in the system, whichever is less.
.It Va hw.cxgbe.nrxq1g
Number of rx queues used for a 1Gb port.
The default is 2 or the number
of CPU cores in the system, whichever is less.
.It Va hw.cxgbe.holdoff_timer_idx_10G
.It Va hw.cxgbe.holdoff_timer_idx_1G
.It Va hw.cxgbe.holdoff_timer_idx
Timer index value used to delay interrupts.
The holdoff timer list has the values 1, 5, 10, 50, 100, and 200
by default (all values are in microseconds) and the index selects a
@ -197,8 +188,7 @@ value from this list.
The default value is 1 which means the timer value is 5us.
Different interfaces can be assigned different values at any time via the
dev.<port>.X.holdoff_tmr_idx sysctl.
.It Va hw.cxgbe.holdoff_pktc_idx_10G
.It Va hw.cxgbe.holdoff_pktc_idx_1G
.It Va hw.cxgbe.holdoff_pktc_idx
Packet-count index value used to delay interrupts.
The packet-count list has the values 1, 8, 16, and 32 by default,
and the index selects a value from this list.

12
sys/dev/cxgbe/adapter.h
View File

@ -1114,15 +1114,11 @@ t4_use_ldst(struct adapter *sc)
}
/* t4_main.c */
extern int t4_ntxq10g;
extern int t4_nrxq10g;
extern int t4_ntxq1g;
extern int t4_nrxq1g;
extern int t4_ntxq;
extern int t4_nrxq;
extern int t4_intr_types;
extern int t4_tmr_idx_10g;
extern int t4_pktc_idx_10g;
extern int t4_tmr_idx_1g;
extern int t4_pktc_idx_1g;
extern int t4_tmr_idx;
extern int t4_pktc_idx;
extern unsigned int t4_qsize_rxq;
extern unsigned int t4_qsize_txq;
extern device_method_t cxgbe_methods[];

315
sys/dev/cxgbe/t4_main.c
View File

@ -241,23 +241,17 @@ SLIST_HEAD(, uld_info) t4_uld_list;
*/
/*
* Number of queues for tx and rx, 10G and 1G, NIC and offload.
* Number of queues for tx and rx, NIC and offload.
*/
#define NTXQ_10G 16
int t4_ntxq10g = -NTXQ_10G;
TUNABLE_INT("hw.cxgbe.ntxq10g", &t4_ntxq10g);
#define NTXQ 16
int t4_ntxq = -NTXQ;
TUNABLE_INT("hw.cxgbe.ntxq", &t4_ntxq);
TUNABLE_INT("hw.cxgbe.ntxq10g", &t4_ntxq); /* Old name, undocumented */
#define NRXQ_10G 8
int t4_nrxq10g = -NRXQ_10G;
TUNABLE_INT("hw.cxgbe.nrxq10g", &t4_nrxq10g);
#define NTXQ_1G 4
int t4_ntxq1g = -NTXQ_1G;
TUNABLE_INT("hw.cxgbe.ntxq1g", &t4_ntxq1g);
#define NRXQ_1G 2
int t4_nrxq1g = -NRXQ_1G;
TUNABLE_INT("hw.cxgbe.nrxq1g", &t4_nrxq1g);
#define NRXQ 8
int t4_nrxq = -NRXQ;
TUNABLE_INT("hw.cxgbe.nrxq", &t4_nrxq);
TUNABLE_INT("hw.cxgbe.nrxq10g", &t4_nrxq); /* Old name, undocumented */
#define NTXQ_VI 1
static int t4_ntxq_vi = -NTXQ_VI;
@ -271,21 +265,13 @@ static int t4_rsrv_noflowq = 0;
TUNABLE_INT("hw.cxgbe.rsrv_noflowq", &t4_rsrv_noflowq);
#ifdef TCP_OFFLOAD
#define NOFLDTXQ_10G 8
static int t4_nofldtxq10g = -NOFLDTXQ_10G;
TUNABLE_INT("hw.cxgbe.nofldtxq10g", &t4_nofldtxq10g);
#define NOFLDTXQ 8
static int t4_nofldtxq = -NOFLDTXQ;
TUNABLE_INT("hw.cxgbe.nofldtxq", &t4_nofldtxq);
#define NOFLDRXQ_10G 2
static int t4_nofldrxq10g = -NOFLDRXQ_10G;
TUNABLE_INT("hw.cxgbe.nofldrxq10g", &t4_nofldrxq10g);
#define NOFLDTXQ_1G 2
static int t4_nofldtxq1g = -NOFLDTXQ_1G;
TUNABLE_INT("hw.cxgbe.nofldtxq1g", &t4_nofldtxq1g);
#define NOFLDRXQ_1G 1
static int t4_nofldrxq1g = -NOFLDRXQ_1G;
TUNABLE_INT("hw.cxgbe.nofldrxq1g", &t4_nofldrxq1g);
#define NOFLDRXQ 2
static int t4_nofldrxq = -NOFLDRXQ;
TUNABLE_INT("hw.cxgbe.nofldrxq", &t4_nofldrxq);
#define NOFLDTXQ_VI 1
static int t4_nofldtxq_vi = -NOFLDTXQ_VI;
@ -360,23 +346,15 @@ TUNABLE_INT("hw.cxgbe.nnmrxq_vi", &t4_nnmrxq_vi);
#endif
/*
* Holdoff parameters for 10G and 1G ports.
* Holdoff parameters for ports.
*/
#define TMR_IDX_10G 1
int t4_tmr_idx_10g = TMR_IDX_10G;
TUNABLE_INT("hw.cxgbe.holdoff_timer_idx_10G", &t4_tmr_idx_10g);
#define TMR_IDX 1
int t4_tmr_idx = TMR_IDX;
TUNABLE_INT("hw.cxgbe.holdoff_timer_idx", &t4_tmr_idx);
#define PKTC_IDX_10G (-1)
int t4_pktc_idx_10g = PKTC_IDX_10G;
TUNABLE_INT("hw.cxgbe.holdoff_pktc_idx_10G", &t4_pktc_idx_10g);
#define TMR_IDX_1G 1
int t4_tmr_idx_1g = TMR_IDX_1G;
TUNABLE_INT("hw.cxgbe.holdoff_timer_idx_1G", &t4_tmr_idx_1g);
#define PKTC_IDX_1G (-1)
int t4_pktc_idx_1g = PKTC_IDX_1G;
TUNABLE_INT("hw.cxgbe.holdoff_pktc_idx_1G", &t4_pktc_idx_1g);
#define PKTC_IDX (-1)
int t4_pktc_idx = PKTC_IDX;
TUNABLE_INT("hw.cxgbe.holdoff_pktc_idx", &t4_pktc_idx);
/*
* Size (# of entries) of each tx and rx queue.
@ -489,17 +467,12 @@ static int vi_mac_funcs[] = {
struct intrs_and_queues {
uint16_t intr_type; /* INTx, MSI, or MSI-X */
uint16_t nirq; /* Total # of vectors */
uint16_t intr_flags_10g;/* Interrupt flags for each 10G port */
uint16_t intr_flags_1g; /* Interrupt flags for each 1G port */
uint16_t ntxq10g; /* # of NIC txq's for each 10G port */
uint16_t nrxq10g; /* # of NIC rxq's for each 10G port */
uint16_t ntxq1g; /* # of NIC txq's for each 1G port */
uint16_t nrxq1g; /* # of NIC rxq's for each 1G port */
uint16_t intr_flags; /* Interrupt flags for each port */
uint16_t ntxq; /* # of NIC txq's for each port */
uint16_t nrxq; /* # of NIC rxq's for each port */
uint16_t rsrv_noflowq; /* Flag whether to reserve queue 0 */
uint16_t nofldtxq10g; /* # of TOE txq's for each 10G port */
uint16_t nofldrxq10g; /* # of TOE rxq's for each 10G port */
uint16_t nofldtxq1g; /* # of TOE txq's for each 1G port */
uint16_t nofldrxq1g; /* # of TOE rxq's for each 1G port */
uint16_t nofldtxq; /* # of TOE txq's for each port */
uint16_t nofldrxq; /* # of TOE rxq's for each port */
/* The vcxgbe/vcxl interfaces use these and not the ones above. */
uint16_t ntxq_vi; /* # of NIC txq's */
@ -532,7 +505,7 @@ static int fwmtype_to_hwmtype(int);
static int validate_mt_off_len(struct adapter *, int, uint32_t, int,
uint32_t *);
static int fixup_devlog_params(struct adapter *);
static int cfg_itype_and_nqueues(struct adapter *, int, int, int,
static int cfg_itype_and_nqueues(struct adapter *, int, int,
struct intrs_and_queues *);
static int prep_firmware(struct adapter *);
static int partition_resources(struct adapter *, const struct firmware *,
@ -857,7 +830,7 @@ static int
t4_attach(device_t dev)
{
struct adapter *sc;
int rc = 0, i, j, n10g, n1g, rqidx, tqidx;
int rc = 0, i, j, rqidx, tqidx, nports;
struct make_dev_args mda;
struct intrs_and_queues iaq;
struct sge *s;
@ -1011,11 +984,8 @@ t4_attach(device_t dev)
/*
* First pass over all the ports - allocate VIs and initialize some
* basic parameters like mac address, port type, etc. We also figure
* out whether a port is 10G or 1G and use that information when
* calculating how many interrupts to attempt to allocate.
* basic parameters like mac address, port type, etc.
*/
n10g = n1g = 0;
for_each_port(sc, i) {
struct port_info *pi;
@ -1051,12 +1021,6 @@ t4_attach(device_t dev)
mtx_init(&pi->pi_lock, pi->lockname, 0, MTX_DEF);
sc->chan_map[pi->tx_chan] = i;
if (port_top_speed(pi) >= 10) {
n10g++;
} else {
n1g++;
}
/* All VIs on this port share this media. */
ifmedia_init(&pi->media, IFM_IMASK, cxgbe_media_change,
cxgbe_media_status);
@ -1075,7 +1039,8 @@ t4_attach(device_t dev)
/*
* Interrupt type, # of interrupts, # of rx/tx queues, etc.
*/
rc = cfg_itype_and_nqueues(sc, n10g, n1g, num_vis, &iaq);
nports = sc->params.nports;
rc = cfg_itype_and_nqueues(sc, nports, num_vis, &iaq);
if (rc != 0)
goto done; /* error message displayed already */
if (iaq.nrxq_vi + iaq.nofldrxq_vi + iaq.nnmrxq_vi == 0)
@ -1085,24 +1050,22 @@ t4_attach(device_t dev)
sc->intr_count = iaq.nirq;
s = &sc->sge;
s->nrxq = n10g * iaq.nrxq10g + n1g * iaq.nrxq1g;
s->ntxq = n10g * iaq.ntxq10g + n1g * iaq.ntxq1g;
s->nrxq = nports * iaq.nrxq;
s->ntxq = nports * iaq.ntxq;
if (num_vis > 1) {
s->nrxq += (n10g + n1g) * (num_vis - 1) * iaq.nrxq_vi;
s->ntxq += (n10g + n1g) * (num_vis - 1) * iaq.ntxq_vi;
s->nrxq += nports * (num_vis - 1) * iaq.nrxq_vi;
s->ntxq += nports * (num_vis - 1) * iaq.ntxq_vi;
}
s->neq = s->ntxq + s->nrxq; /* the free list in an rxq is an eq */
s->neq += sc->params.nports + 1;/* ctrl queues: 1 per port + 1 mgmt */
s->neq += nports + 1;/* ctrl queues: 1 per port + 1 mgmt */
s->niq = s->nrxq + 1; /* 1 extra for firmware event queue */
#ifdef TCP_OFFLOAD
if (is_offload(sc)) {
s->nofldrxq = n10g * iaq.nofldrxq10g + n1g * iaq.nofldrxq1g;
s->nofldtxq = n10g * iaq.nofldtxq10g + n1g * iaq.nofldtxq1g;
s->nofldrxq = nports * iaq.nofldrxq;
s->nofldtxq = nports * iaq.nofldtxq;
if (num_vis > 1) {
s->nofldrxq += (n10g + n1g) * (num_vis - 1) *
iaq.nofldrxq_vi;
s->nofldtxq += (n10g + n1g) * (num_vis - 1) *
iaq.nofldtxq_vi;
s->nofldrxq += nports * (num_vis - 1) * iaq.nofldrxq_vi;
s->nofldtxq += nports * (num_vis - 1) * iaq.nofldtxq_vi;
}
s->neq += s->nofldtxq + s->nofldrxq;
s->niq += s->nofldrxq;
@ -1115,8 +1078,8 @@ t4_attach(device_t dev)
#endif
#ifdef DEV_NETMAP
if (num_vis > 1) {
s->nnmrxq = (n10g + n1g) * (num_vis - 1) * iaq.nnmrxq_vi;
s->nnmtxq = (n10g + n1g) * (num_vis - 1) * iaq.nnmtxq_vi;
s->nnmrxq = nports * (num_vis - 1) * iaq.nnmrxq_vi;
s->nnmtxq = nports * (num_vis - 1) * iaq.nnmtxq_vi;
}
s->neq += s->nnmtxq + s->nnmrxq;
s->niq += s->nnmrxq;
@ -1127,7 +1090,7 @@ t4_attach(device_t dev)
M_CXGBE, M_ZERO | M_WAITOK);
#endif
s->ctrlq = malloc(sc->params.nports * sizeof(struct sge_wrq), M_CXGBE,
s->ctrlq = malloc(nports * sizeof(struct sge_wrq), M_CXGBE,
M_ZERO | M_WAITOK);
s->rxq = malloc(s->nrxq * sizeof(struct sge_rxq), M_CXGBE,
M_ZERO | M_WAITOK);
@ -1170,19 +1133,12 @@ t4_attach(device_t dev)
vi->first_rxq = rqidx;
vi->first_txq = tqidx;
if (port_top_speed(pi) >= 10) {
vi->tmr_idx = t4_tmr_idx_10g;
vi->pktc_idx = t4_pktc_idx_10g;
vi->flags |= iaq.intr_flags_10g & INTR_RXQ;
vi->nrxq = j == 0 ? iaq.nrxq10g : iaq.nrxq_vi;
vi->ntxq = j == 0 ? iaq.ntxq10g : iaq.ntxq_vi;
} else {
vi->tmr_idx = t4_tmr_idx_1g;
vi->pktc_idx = t4_pktc_idx_1g;
vi->flags |= iaq.intr_flags_1g & INTR_RXQ;
vi->nrxq = j == 0 ? iaq.nrxq1g : iaq.nrxq_vi;
vi->ntxq = j == 0 ? iaq.ntxq1g : iaq.ntxq_vi;
}
vi->tmr_idx = t4_tmr_idx;
vi->pktc_idx = t4_pktc_idx;
vi->flags |= iaq.intr_flags & INTR_RXQ;
vi->nrxq = j == 0 ? iaq.nrxq : iaq.nrxq_vi;
vi->ntxq = j == 0 ? iaq.ntxq : iaq.ntxq_vi;
rqidx += vi->nrxq;
tqidx += vi->ntxq;
@ -1196,19 +1152,10 @@ t4_attach(device_t dev)
vi->ofld_pktc_idx = t4_pktc_idx_ofld;
vi->first_ofld_rxq = ofld_rqidx;
vi->first_ofld_txq = ofld_tqidx;
if (port_top_speed(pi) >= 10) {
vi->flags |= iaq.intr_flags_10g & INTR_OFLD_RXQ;
vi->nofldrxq = j == 0 ? iaq.nofldrxq10g :
iaq.nofldrxq_vi;
vi->nofldtxq = j == 0 ? iaq.nofldtxq10g :
iaq.nofldtxq_vi;
} else {
vi->flags |= iaq.intr_flags_1g & INTR_OFLD_RXQ;
vi->nofldrxq = j == 0 ? iaq.nofldrxq1g :
iaq.nofldrxq_vi;
vi->nofldtxq = j == 0 ? iaq.nofldtxq1g :
iaq.nofldtxq_vi;
}
vi->flags |= iaq.intr_flags & INTR_OFLD_RXQ;
vi->nofldrxq = j == 0 ? iaq.nofldrxq : iaq.nofldrxq_vi;
vi->nofldtxq = j == 0 ? iaq.nofldtxq : iaq.nofldtxq_vi;
ofld_rqidx += vi->nofldrxq;
ofld_tqidx += vi->nofldtxq;
#endif
@ -2715,28 +2662,25 @@ fixup_devlog_params(struct adapter *sc)
}
static int
cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g, int num_vis,
cfg_itype_and_nqueues(struct adapter *sc, int nports, int num_vis,
struct intrs_and_queues *iaq)
{
int rc, itype, navail, nrxq10g, nrxq1g, n;
int nofldrxq10g = 0, nofldrxq1g = 0;
int rc, itype, navail, nrxq, n;
int nofldrxq = 0;
MPASS(nports > 0);
bzero(iaq, sizeof(*iaq));
iaq->ntxq10g = t4_ntxq10g;
iaq->ntxq1g = t4_ntxq1g;
iaq->ntxq = t4_ntxq;
iaq->ntxq_vi = t4_ntxq_vi;
iaq->nrxq10g = nrxq10g = t4_nrxq10g;
iaq->nrxq1g = nrxq1g = t4_nrxq1g;
iaq->nrxq = nrxq = t4_nrxq;
iaq->nrxq_vi = t4_nrxq_vi;
iaq->rsrv_noflowq = t4_rsrv_noflowq;
#ifdef TCP_OFFLOAD
if (is_offload(sc)) {
iaq->nofldtxq10g = t4_nofldtxq10g;
iaq->nofldtxq1g = t4_nofldtxq1g;
iaq->nofldtxq = t4_nofldtxq;
iaq->nofldtxq_vi = t4_nofldtxq_vi;
iaq->nofldrxq10g = nofldrxq10g = t4_nofldrxq10g;
iaq->nofldrxq1g = nofldrxq1g = t4_nofldrxq1g;
iaq->nofldrxq = nofldrxq = t4_nofldrxq;
iaq->nofldrxq_vi = t4_nofldrxq_vi;
}
#endif
@ -2761,8 +2705,7 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g, int num_vis,
continue;
iaq->intr_type = itype;
iaq->intr_flags_10g = 0;
iaq->intr_flags_1g = 0;
iaq->intr_flags = 0;
/*
* Best option: an interrupt vector for errors, one for the
@ -2772,15 +2715,13 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g, int num_vis,
* because only one set of queues is active at a time.
*/
iaq->nirq = T4_EXTRA_INTR;
iaq->nirq += n10g * (nrxq10g + nofldrxq10g);
iaq->nirq += n1g * (nrxq1g + nofldrxq1g);
iaq->nirq += (n10g + n1g) * (num_vis - 1) *
iaq->nirq += nports * (nrxq + nofldrxq);
iaq->nirq += nports * (num_vis - 1) *
max(iaq->nrxq_vi, iaq->nnmrxq_vi); /* See comment above. */
iaq->nirq += (n10g + n1g) * (num_vis - 1) * iaq->nofldrxq_vi;
iaq->nirq += nports * (num_vis - 1) * iaq->nofldrxq_vi;
if (iaq->nirq <= navail &&
(itype != INTR_MSI || powerof2(iaq->nirq))) {
iaq->intr_flags_10g = INTR_ALL;
iaq->intr_flags_1g = INTR_ALL;
iaq->intr_flags = INTR_ALL;
goto allocate;
}
@ -2788,13 +2729,11 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g, int num_vis,
if (num_vis > 1) {
device_printf(sc->dev, "virtual interfaces disabled "
"because num_vis=%u with current settings "
"(nrxq10g=%u, nrxq1g=%u, nofldrxq10g=%u, "
"nofldrxq1g=%u, nrxq_vi=%u nofldrxq_vi=%u, "
"(nrxq=%u, nofldrxq=%u, nrxq_vi=%u nofldrxq_vi=%u, "
"nnmrxq_vi=%u) would need %u interrupts but "
"only %u are available.\n", num_vis, nrxq10g,
nrxq1g, nofldrxq10g, nofldrxq1g, iaq->nrxq_vi,
iaq->nofldrxq_vi, iaq->nnmrxq_vi, iaq->nirq,
navail);
"only %u are available.\n", num_vis, nrxq,
nofldrxq, iaq->nrxq_vi, iaq->nofldrxq_vi,
iaq->nnmrxq_vi, iaq->nirq, navail);
num_vis = 1;
iaq->ntxq_vi = iaq->nrxq_vi = 0;
iaq->nofldtxq_vi = iaq->nofldrxq_vi = 0;
@ -2809,19 +2748,12 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g, int num_vis,
* will forward their interrupts to those that do.
*/
iaq->nirq = T4_EXTRA_INTR;
if (nrxq10g >= nofldrxq10g) {
iaq->intr_flags_10g = INTR_RXQ;
iaq->nirq += n10g * nrxq10g;
if (nrxq >= nofldrxq) {
iaq->intr_flags = INTR_RXQ;
iaq->nirq += nports * nrxq;
} else {
iaq->intr_flags_10g = INTR_OFLD_RXQ;
iaq->nirq += n10g * nofldrxq10g;
}
if (nrxq1g >= nofldrxq1g) {
iaq->intr_flags_1g = INTR_RXQ;
iaq->nirq += n1g * nrxq1g;
} else {
iaq->intr_flags_1g = INTR_OFLD_RXQ;
iaq->nirq += n1g * nofldrxq1g;
iaq->intr_flags = INTR_OFLD_RXQ;
iaq->nirq += nports * nofldrxq;
}
if (iaq->nirq <= navail &&
(itype != INTR_MSI || powerof2(iaq->nirq)))
@ -2834,45 +2766,24 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g, int num_vis,
* fit what's available to us.
*/
iaq->nirq = T4_EXTRA_INTR;
iaq->nirq += n10g + n1g;
iaq->nirq += nports;
if (iaq->nirq <= navail) {
int leftover = navail - iaq->nirq;
int target = max(nrxq, nofldrxq);
if (n10g > 0) {
int target = max(nrxq10g, nofldrxq10g);
iaq->intr_flags_10g = nrxq10g >= nofldrxq10g ?
iaq->intr_flags = nrxq >= nofldrxq ?
INTR_RXQ : INTR_OFLD_RXQ;
n = 1;
while (n < target && leftover >= n10g) {
leftover -= n10g;
iaq->nirq += n10g;
while (n < target && leftover >= nports) {
leftover -= nports;
iaq->nirq += nports;
n++;
}
iaq->nrxq10g = min(n, nrxq10g);
iaq->nrxq = min(n, nrxq);
#ifdef TCP_OFFLOAD
iaq->nofldrxq10g = min(n, nofldrxq10g);
iaq->nofldrxq = min(n, nofldrxq);
#endif
}
if (n1g > 0) {
int target = max(nrxq1g, nofldrxq1g);
iaq->intr_flags_1g = nrxq1g >= nofldrxq1g ?
INTR_RXQ : INTR_OFLD_RXQ;
n = 1;
while (n < target && leftover >= n1g) {
leftover -= n1g;
iaq->nirq += n1g;
n++;
}
iaq->nrxq1g = min(n, nrxq1g);
#ifdef TCP_OFFLOAD
iaq->nofldrxq1g = min(n, nofldrxq1g);
#endif
}
if (itype != INTR_MSI || powerof2(iaq->nirq))
goto allocate;
@ -2881,11 +2792,11 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g, int num_vis,
/*
* Least desirable option: one interrupt vector for everything.
*/
iaq->nirq = iaq->nrxq10g = iaq->nrxq1g = 1;
iaq->intr_flags_10g = iaq->intr_flags_1g = 0;
iaq->nirq = iaq->nrxq = 1;
iaq->intr_flags = 0;
#ifdef TCP_OFFLOAD
if (is_offload(sc))
iaq->nofldrxq10g = iaq->nofldrxq1g = 1;
iaq->nofldrxq = 1;
#endif
allocate:
navail = iaq->nirq;
@ -9941,50 +9852,30 @@ tweak_tunables(void)
{
int nc = mp_ncpus; /* our snapshot of the number of CPUs */
if (t4_ntxq10g < 1) {
if (t4_ntxq < 1) {
#ifdef RSS
t4_ntxq10g = rss_getnumbuckets();
t4_ntxq = rss_getnumbuckets();
#else
calculate_nqueues(&t4_ntxq10g, nc, NTXQ_10G);
#endif
}
if (t4_ntxq1g < 1) {
#ifdef RSS
/* XXX: way too many for 1GbE? */
t4_ntxq1g = rss_getnumbuckets();
#else
calculate_nqueues(&t4_ntxq1g, nc, NTXQ_1G);
calculate_nqueues(&t4_ntxq, nc, NTXQ);
#endif
}
calculate_nqueues(&t4_ntxq_vi, nc, NTXQ_VI);
if (t4_nrxq10g < 1) {
if (t4_nrxq < 1) {
#ifdef RSS
t4_nrxq10g = rss_getnumbuckets();
t4_nrxq = rss_getnumbuckets();
#else
calculate_nqueues(&t4_nrxq10g, nc, NRXQ_10G);
#endif
}
if (t4_nrxq1g < 1) {
#ifdef RSS
/* XXX: way too many for 1GbE? */
t4_nrxq1g = rss_getnumbuckets();
#else
calculate_nqueues(&t4_nrxq1g, nc, NRXQ_1G);
calculate_nqueues(&t4_nrxq, nc, NRXQ);
#endif
}
calculate_nqueues(&t4_nrxq_vi, nc, NRXQ_VI);
#ifdef TCP_OFFLOAD
calculate_nqueues(&t4_nofldtxq10g, nc, NOFLDTXQ_10G);
calculate_nqueues(&t4_nofldtxq1g, nc, NOFLDTXQ_1G);
calculate_nqueues(&t4_nofldtxq, nc, NOFLDTXQ);
calculate_nqueues(&t4_nofldtxq_vi, nc, NOFLDTXQ_VI);
calculate_nqueues(&t4_nofldrxq10g, nc, NOFLDRXQ_10G);
calculate_nqueues(&t4_nofldrxq1g, nc, NOFLDRXQ_1G);
calculate_nqueues(&t4_nofldrxq, nc, NOFLDRXQ);
calculate_nqueues(&t4_nofldrxq_vi, nc, NOFLDRXQ_VI);
if (t4_toecaps_allowed == -1)
@ -10022,17 +9913,11 @@ tweak_tunables(void)
calculate_nqueues(&t4_nnmrxq_vi, nc, NNMRXQ_VI);
#endif
if (t4_tmr_idx_10g < 0 || t4_tmr_idx_10g >= SGE_NTIMERS)
t4_tmr_idx_10g = TMR_IDX_10G;
if (t4_tmr_idx < 0 || t4_tmr_idx >= SGE_NTIMERS)
t4_tmr_idx = TMR_IDX;
if (t4_pktc_idx_10g < -1 || t4_pktc_idx_10g >= SGE_NCOUNTERS)
t4_pktc_idx_10g = PKTC_IDX_10G;
if (t4_tmr_idx_1g < 0 || t4_tmr_idx_1g >= SGE_NTIMERS)
t4_tmr_idx_1g = TMR_IDX_1G;
if (t4_pktc_idx_1g < -1 || t4_pktc_idx_1g >= SGE_NCOUNTERS)
t4_pktc_idx_1g = PKTC_IDX_1G;
if (t4_pktc_idx < -1 || t4_pktc_idx >= SGE_NCOUNTERS)
t4_pktc_idx = PKTC_IDX;
if (t4_qsize_txq < 128)
t4_qsize_txq = 128;

117
sys/dev/cxgbe/t4_vf.c
View File

@ -62,12 +62,9 @@ __FBSDID("$FreeBSD$");
struct intrs_and_queues {
uint16_t intr_type; /* MSI, or MSI-X */
uint16_t nirq; /* Total # of vectors */
uint16_t intr_flags_10g;/* Interrupt flags for each 10G port */
uint16_t intr_flags_1g; /* Interrupt flags for each 1G port */
uint16_t ntxq10g; /* # of NIC txq's for each 10G port */
uint16_t nrxq10g; /* # of NIC rxq's for each 10G port */
uint16_t ntxq1g; /* # of NIC txq's for each 1G port */
uint16_t nrxq1g; /* # of NIC rxq's for each 1G port */
uint16_t intr_flags; /* Interrupt flags for each port */
uint16_t ntxq; /* # of NIC txq's for each port */
uint16_t nrxq; /* # of NIC rxq's for each port */
};
struct {
@ -306,12 +303,10 @@ set_params__post_init(struct adapter *sc)
#undef FW_PARAM_DEV
static int
cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g,
struct intrs_and_queues *iaq)
cfg_itype_and_nqueues(struct adapter *sc, struct intrs_and_queues *iaq)
{
struct vf_resources *vfres;
int nrxq10g, nrxq1g, nrxq;
int ntxq10g, ntxq1g, ntxq;
int nrxq, ntxq, nports;
int itype, iq_avail, navail, rc;
/*
@ -319,6 +314,7 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g,
* we can allocate enough interrupts for our layout.
*/
vfres = &sc->params.vfres;
nports = sc->params.nports;
bzero(iaq, sizeof(*iaq));
for (itype = INTR_MSIX; itype != 0; itype >>= 1) {
@ -334,8 +330,7 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g,
continue;
iaq->intr_type = itype;
iaq->intr_flags_10g = 0;
iaq->intr_flags_1g = 0;
iaq->intr_flags = 0;
/*
* XXX: The Linux driver reserves an Ingress Queue for
@ -358,10 +353,10 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g,
* limit on ingress queues.
*/
iq_avail = vfres->niqflint - iaq->nirq;
if (iq_avail < n10g + n1g) {
if (iq_avail < nports) {
device_printf(sc->dev,
"Not enough ingress queues (%d) for %d ports\n",
vfres->niqflint, n10g + n1g);
vfres->niqflint, nports);
return (ENXIO);
}
@ -371,26 +366,17 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g,
* port, then don't bother, we will just forward all
* interrupts to one interrupt in that case.
*/
if (iaq->nirq + n10g + n1g <= navail) {
if (iaq->nirq + nports <= navail) {
if (iq_avail > navail - iaq->nirq)
iq_avail = navail - iaq->nirq;
}
nrxq10g = t4_nrxq10g;
nrxq1g = t4_nrxq1g;
nrxq = n10g * nrxq10g + n1g * nrxq1g;
if (nrxq > iq_avail && nrxq1g > 1) {
/* Too many ingress queues. Try just 1 for 1G. */
nrxq1g = 1;
nrxq = n10g * nrxq10g + n1g * nrxq1g;
}
nrxq = nports * t4_nrxq;
if (nrxq > iq_avail) {
/*
* Still too many ingress queues. Use what we
* can for each 10G port.
* Too many ingress queues. Use what we can.
*/
nrxq10g = (iq_avail - n1g) / n10g;
nrxq = n10g * nrxq10g + n1g * nrxq1g;
nrxq = (iq_avail / nports) * nports;
}
KASSERT(nrxq <= iq_avail, ("too many ingress queues"));
@ -398,45 +384,34 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g,
* Next, determine the upper bound on txqs from the limit
* on ETH queues.
*/
if (vfres->nethctrl < n10g + n1g) {
if (vfres->nethctrl < nports) {
device_printf(sc->dev,
"Not enough ETH queues (%d) for %d ports\n",
vfres->nethctrl, n10g + n1g);
vfres->nethctrl, nports);
return (ENXIO);
}
ntxq10g = t4_ntxq10g;
ntxq1g = t4_ntxq1g;
ntxq = n10g * ntxq10g + n1g * ntxq1g;
if (ntxq > vfres->nethctrl) {
/* Too many ETH queues. Try just 1 for 1G. */
ntxq1g = 1;
ntxq = n10g * ntxq10g + n1g * ntxq1g;
}
ntxq = nports * t4_ntxq;
if (ntxq > vfres->nethctrl) {
/*
* Still too many ETH queues. Use what we
* can for each 10G port.
* Too many ETH queues. Use what we can.
*/
ntxq10g = (vfres->nethctrl - n1g) / n10g;
ntxq = n10g * ntxq10g + n1g * ntxq1g;
ntxq = (vfres->nethctrl / nports) * nports;
}
KASSERT(ntxq <= vfres->nethctrl, ("too many ETH queues"));
/*
* Finally, ensure we have enough egress queues.
*/
if (vfres->neq < (n10g + n1g) * 2) {
if (vfres->neq < nports * 2) {
device_printf(sc->dev,
"Not enough egress queues (%d) for %d ports\n",
vfres->neq, n10g + n1g);
vfres->neq, nports);
return (ENXIO);
}
if (nrxq + ntxq > vfres->neq) {
/* Just punt and use 1 for everything. */
nrxq1g = ntxq1g = nrxq10g = ntxq10g = 1;
nrxq = n10g * nrxq10g + n1g * nrxq1g;
ntxq = n10g * ntxq10g + n1g * ntxq1g;
nrxq = ntxq = nports;
}
KASSERT(nrxq <= iq_avail, ("too many ingress queues"));
KASSERT(ntxq <= vfres->nethctrl, ("too many ETH queues"));
@ -447,10 +422,8 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g,
* have to be a power of 2 as well.
*/
iaq->nirq += nrxq;
iaq->ntxq10g = ntxq10g;
iaq->ntxq1g = ntxq1g;
iaq->nrxq10g = nrxq10g;
iaq->nrxq1g = nrxq1g;
iaq->ntxq = ntxq;
iaq->nrxq = nrxq;
if (iaq->nirq <= navail &&
(itype != INTR_MSI || powerof2(iaq->nirq))) {
navail = iaq->nirq;
@ -465,8 +438,7 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g,
return (rc);
}
if (navail == iaq->nirq) {
iaq->intr_flags_10g = INTR_RXQ;
iaq->intr_flags_1g = INTR_RXQ;
iaq->intr_flags = INTR_RXQ;
return (0);
}
pci_release_msi(sc->dev);
@ -483,8 +455,7 @@ cfg_itype_and_nqueues(struct adapter *sc, int n10g, int n1g,
device_printf(sc->dev,
"failed to allocate vectors:%d, type=%d, req=%d, rcvd=%d\n",
itype, rc, iaq->nirq, navail);
iaq->intr_flags_10g = 0;
iaq->intr_flags_1g = 0;
iaq->intr_flags = 0;
return (rc);
}
@ -500,7 +471,7 @@ static int
t4vf_attach(device_t dev)
{
struct adapter *sc;
int rc = 0, i, j, n10g, n1g, rqidx, tqidx;
int rc = 0, i, j, rqidx, tqidx;
struct make_dev_args mda;
struct intrs_and_queues iaq;
struct sge *s;
@ -634,11 +605,8 @@ t4vf_attach(device_t dev)
/*
* First pass over all the ports - allocate VIs and initialize some
* basic parameters like mac address, port type, etc. We also figure
* out whether a port is 10G or 1G and use that information when
* calculating how many interrupts to attempt to allocate.
* basic parameters like mac address, port type, etc.
*/
n10g = n1g = 0;
for_each_port(sc, i) {
struct port_info *pi;
@ -673,12 +641,6 @@ t4vf_attach(device_t dev)
mtx_init(&pi->pi_lock, pi->lockname, 0, MTX_DEF);
sc->chan_map[pi->tx_chan] = i;
if (port_top_speed(pi) >= 10) {
n10g++;
} else {
n1g++;
}
pi->dev = device_add_child(dev, sc->names->vf_ifnet_name, -1);
if (pi->dev == NULL) {
device_printf(dev,
@ -693,7 +655,7 @@ t4vf_attach(device_t dev)
/*
* Interrupt type, # of interrupts, # of rx/tx queues, etc.
*/
rc = cfg_itype_and_nqueues(sc, n10g, n1g, &iaq);
rc = cfg_itype_and_nqueues(sc, &iaq);
if (rc != 0)
goto done; /* error message displayed already */
@ -701,8 +663,8 @@ t4vf_attach(device_t dev)
sc->intr_count = iaq.nirq;
s = &sc->sge;
s->nrxq = n10g * iaq.nrxq10g + n1g * iaq.nrxq1g;
s->ntxq = n10g * iaq.ntxq10g + n1g * iaq.ntxq1g;
s->nrxq = sc->params.nports * iaq.nrxq;
s->ntxq = sc->params.nports * iaq.ntxq;
s->neq = s->ntxq + s->nrxq; /* the free list in an rxq is an eq */
s->neq += sc->params.nports + 1;/* ctrl queues: 1 per port + 1 mgmt */
s->niq = s->nrxq + 1; /* 1 extra for firmware event queue */
@ -738,19 +700,12 @@ t4vf_attach(device_t dev)
vi->first_rxq = rqidx;
vi->first_txq = tqidx;
if (port_top_speed(pi) >= 10) {
vi->tmr_idx = t4_tmr_idx_10g;
vi->pktc_idx = t4_pktc_idx_10g;
vi->flags |= iaq.intr_flags_10g & INTR_RXQ;
vi->nrxq = j == 0 ? iaq.nrxq10g : 1;
vi->ntxq = j == 0 ? iaq.ntxq10g : 1;
} else {
vi->tmr_idx = t4_tmr_idx_1g;
vi->pktc_idx = t4_pktc_idx_1g;
vi->flags |= iaq.intr_flags_1g & INTR_RXQ;
vi->nrxq = j == 0 ? iaq.nrxq1g : 1;
vi->ntxq = j == 0 ? iaq.ntxq1g : 1;
}
vi->tmr_idx = t4_tmr_idx;
vi->pktc_idx = t4_pktc_idx;
vi->flags |= iaq.intr_flags & INTR_RXQ;
vi->nrxq = j == 0 ? iaq.nrxq: 1;
vi->ntxq = j == 0 ? iaq.ntxq: 1;
rqidx += vi->nrxq;
tqidx += vi->ntxq;