Commit Graph

65 Commits

Author SHA1 Message Date
Gleb Smirnoff
c578b6aca0 Provide a set of inline functions to manage simple mbuf(9) queues, based
on queue(3)'s STAILQ.  Utilize them in cxgb(4) and Xen, deleting home
grown implementations.

Sponsored by:	Netflix
Sponsored by:	Nginx, Inc.
2015-02-19 01:19:42 +00:00
Navdeep Parhar
e26e6373c8 cxgb(4): implement if_get_counter. 2014-09-27 18:35:16 +00:00
Gleb Smirnoff
76039bc84f The r48589 promised to remove implicit inclusion of if_var.h soon. Prepare
to this event, adding if_var.h to files that do need it. Also, include
all includes that now are included due to implicit pollution via if_var.h

Sponsored by:	Netflix
Sponsored by:	Nginx, Inc.
2013-10-26 17:58:36 +00:00
Navdeep Parhar
0a7049095f cxgb(4): IPv6 rx/tx hw checksum, IPv6 TSO and LRO too.
(Some parts already worked, this makes it complete).
2012-06-30 02:11:53 +00:00
Navdeep Parhar
09fe63205c - Updated TOE support in the kernel.
- Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs.
  These are available as t3_tom and t4_tom modules that augment cxgb(4)
  and cxgbe(4) respectively.  The cxgb/cxgbe drivers continue to work as
  usual with or without these extra features.

- iWARP driver for Terminator 3 ASIC (kernel verbs).  T4 iWARP in the
  works and will follow soon.

Build-tested with make universe.

30s overview
============
What interfaces support TCP offload?  Look for TOE4 and/or TOE6 in the
capabilities of an interface:
# ifconfig -m | grep TOE

Enable/disable TCP offload on an interface (just like any other ifnet
capability):
# ifconfig cxgbe0 toe
# ifconfig cxgbe0 -toe

Which connections are offloaded?  Look for toe4 and/or toe6 in the
output of netstat and sockstat:
# netstat -np tcp | grep toe
# sockstat -46c | grep toe

Reviewed by:	bz, gnn
Sponsored by:	Chelsio communications.
MFC after:	~3 months (after 9.1, and after ensuring MFC is feasible)
2012-06-19 07:34:13 +00:00
Bjoern A. Zeeb
62b5b6ecd0 MFp4 bz_ipv6_fast:
Significantly update tcp_lro for mostly two things:
  1) introduce basic support for IPv6 without extension headers.
  2) try hard to also get the incremental checksum updates right,
     especially also in the IPv4 case for the IP and TCP header.

  Move variables around for better locality, factor things out into
  functions, allow checksum updates to be compiled out, ...

  Leave a few comments on further things to look at in the future,
  though that is not the full list.

  Update drivers with appropriate #includes as needed for IPv6 data
  type in LRO.

  Sponsored by:	The FreeBSD Foundation
  Sponsored by:	iXsystems

Reviewed by:	gnn (as part of the whole)
MFC After:	3 days
2012-05-24 23:03:23 +00:00
Navdeep Parhar
65d43cc6e7 Remove if_start from cxgb and cxgbe.
Submitted by:	jhb
MFC after:	3 days
2012-02-07 07:32:39 +00:00
Navdeep Parhar
7eeb16cee7 t3_free_sge_resources should be given the number of qsets it needs to free.
MFC after:	1 week
2011-03-24 01:16:48 +00:00
Navdeep Parhar
bd1a9fbad6 Improve cxgb(4)'s behaviour when faced with temporarily "bouncy" links:
- Run the adapter's tick at 1Hz and remove link state checks from it.
  Instead, have each port check its link state.  Delay the check so that
  it takes place slightly after the driver is notified of a change in
  link state.  This is a cheap way to debounce these notifications if
  many are received in rapid succession.  POLL_LINK_1ST_TIME flag can
  also be eliminated as a side effect of these changes.
- Do not reset the PHY when link goes down.
- Clear port's link_fault flag if the PHY indicates link is down.
- get_link_status_r should leave speed and duplex alone when link is down.

MFC after:	1 month
2010-07-09 00:38:00 +00:00
Navdeep Parhar
2c32b50248 Eliminate ext_intr_task. The "slow" interrupt handler is already
running on the adapter's task queue.  Just do what the task does
instead of enqueueing it.

MFC after:	3 days
2010-07-09 00:36:35 +00:00
Navdeep Parhar
27d1c65e8e cxgb(4): add knob to get packet timestamps from the hardware.
The T3 ASIC can provide an incoming packet's timestamp instead of its RSS hash.
The timestamp is just a counter running off the card's clock.  With a 175MHz
clock an increment represents ~5.7ns and the 32 bit value wraps around in ~25s.

# sysctl -d dev.cxgbc.0.pkt_timestamp
dev.cxgbc.0.pkt_timestamp: provide packet timestamp instead of connection hash

# sysctl -d dev.cxgbc.0.core_clock
dev.cxgbc.0.core_clock: core clock frequency (in KHz)
# sysctl dev.cxgbc.0.core_clock
dev.cxgbc.0.core_clock: 175000
2010-06-12 22:33:04 +00:00
Navdeep Parhar
1d4942f42f Don't ring the tx doorbell for every frame when we know more frames
will follow.  Adjust the freelist and response queue doorbells too.

Discussed with:	kmacy
2010-05-05 22:52:06 +00:00
Navdeep Parhar
489ca05be7 Increase response queue size to avoid starvation, add a counter
to track it when it does occur.
2010-04-02 17:50:52 +00:00
Navdeep Parhar
97ae3bc359 Multiple fixes related to queue set sizing and resources:
- Only the tunnelq (TXQ_ETH) requires a buf_ring, an ifq, and the watchdog/timer
  callouts.  Do not allocate these for the other tx queues.

- Use 16k jumbo clusters only on offload capable cards by default.

- Do not allocate a full tx ring for the offload queue if the card is not
  offload capable.

- Slightly better freelist size calculation.

- Fix nmbjumbo4 typo, remove unneeded global variables.

MFC after:	3 days
2010-03-31 00:27:49 +00:00
Navdeep Parhar
92f61ecb4b Fix tx drop statistics.
MFC after:	3 days
2010-03-31 00:26:02 +00:00
Navdeep Parhar
e83ec3e5c3 There is no need to test __FreeBSD_version for features that have
been around for a long time now (7.1-ish or even earlier); assume
they are present.  These includes MSI, TSO, LRO, VLAN, INTR_FILTERS,
FIRMWARE, etc.

Also, eliminate some dead code and clean up in other places as part
of this quick once-over.

MFC after:	1 week
2010-02-24 10:16:18 +00:00
Navdeep Parhar
b302b77ca7 Fix for a cxgb(4) panic. cxgb_ioctl can be called by the IP and IPv6
layers with non-sleepable locks held.  Don't (potentially) sleep in
those situations.
2010-01-20 03:40:43 +00:00
Navdeep Parhar
c01f2b8301 cxgb(4) updates, including:
- support for the new Gen-2, BT, and LP-CR cards.
- T3 firmware 7.7.0
- shared "common code" updates.

Approved by:	gnn (mentor)
Obtained from:	Chelsio
MFC after:	1 month
2009-10-05 20:21:41 +00:00
Robert Watson
5157862aa2 Use if_maddr_rlock() instead of IF_ADDR_LOCK() to protect access to
if_multiaddrs in if_cxgb.

Approved by:	re (kib)
MFC after:	6 weeks
2009-06-26 19:04:08 +00:00
Kip Macy
3f345a5d09 Greatly simplify cxgb by removing almost all of the custom mbuf management logic
- remove mbuf iovec - useful, but adds too much complexity when isolated to
   the driver

- remove driver private caching - insufficient benefit over UMA to justify
  the added complexity and maintenance overhead

- remove separate logic for managing multiple transmit queues, with the
  new drbr routines the control flow can be made to much more closely resemble
  legacy drivers

- remove dedicated service threads, with per-cpu callouts one can get the same
  benefit much more simply by registering a callout 1 tick in the future if there
  are still buffered packets

- remove embedded mbuf usage - Jeffr's changes will (I hope) soon be integrated
  greatly reducing the overhead of using kernel APIs for reference counting
  clusters

- add hysteresis to descriptor coalescing logic

- add coalesce threshold sysctls to allow users to decide at run-time
  between optimizing for forwarding / UDP or optimizing for TCP

- add once per second watchdog to effectively close the very rare races
  occurring from coalescing

- incorporate Navdeep's changes to the initialization path required to
  convert port and adapter locks back to ordinary mutexes (silencing BPF
  LOR complaints)

- enable prefetches in get_packet and tx cleaning

Reviewed by:	navdeep@
MFC after:	2 weeks
2009-06-19 23:34:32 +00:00
George V. Neville-Neil
0bbdea7776 Integrate three changes from Chelsio.
1) Add a sysctl that will say what type of PHYs exist on the card.
2) Fix a bug that occurs when an AEL 2005 PHY resets without a transciever
in the card.
3) Unify the PHY link detection code.

Obtained from:	Navdeep Parhar
MFC after:	10 days
2009-05-21 15:08:03 +00:00
George V. Neville-Neil
f2d8ff04fe Update the Chelsio driver to the latest bits from Chelsio
Firmware upgraded to 7.1.0 (from 5.0.0).
T3C EEPROM and SRAM added; Code to update eeprom/sram fixed.
fl_empty and rx_fifo_ovfl counters can be observed via sysctl.
Two new cxgbtool commands to get uP logic analyzer info and uP IOQs
Synced up with Chelsio's "common code" (as of 03/03/09)

Submitted by:	 Navdeep Parhar at Chelsio
Reviewed by:	gnn
MFC after:	2 weeks
2009-03-10 19:22:45 +00:00
Kip Macy
a02573bc06 - enable multiple transmit queues
- invert sense of hw.cxgb.singleq tunable to hw.cxgb.multiq
- don't wake up transmitting thread by default
- add per tx queue ifaltq to handle ALTQ
- remove several unused functions in cxgb_multiq.c
- add several sysctls: multiq_tx_enable, coalesce_tx_enable,
  and wakeup_tx_thread
- this obsoletes the hw.cxgb.snd_queue_len as ifq is replaced
  by a buf_ring
2008-11-22 08:05:05 +00:00
Kip Macy
db7f0b974f - bump __FreeBSD version to reflect added buf_ring, memory barriers,
and ifnet functions

- add memory barriers to <machine/atomic.h>
- update drivers to only conditionally define their own

- add lockless producer / consumer ring buffer
- remove ring buffer implementation from cxgb and update its callers

- add if_transmit(struct ifnet *ifp, struct mbuf *m) to ifnet to
  allow drivers to efficiently manage multiple hardware queues
  (i.e. not serialize all packets through one ifq)
- expose if_qflush to allow drivers to flush any driver managed queues

This work was supported by Bitgravity Inc. and Chelsio Inc.
2008-11-22 05:55:56 +00:00
Kip Macy
a7db7fbd35 - Remove default NIC dependency on ulp headers
- make toe module build dependent on kernel support

Submitted by:	Chelsio Inc.
MFC after:	1 week
2008-09-24 01:19:08 +00:00
Kip Macy
79775f8f1b Update cxgb include paths to not require prefixing with dev/cxgb
Submitted by:	Chelsio Inc.
2008-09-23 03:16:54 +00:00
Kip Macy
e97121da99 Allow cxgb to be unified across versions by making newer features conditional
Submitted by:	Chelsio Inc
MFC after:	3 days
2008-09-23 02:22:24 +00:00
Kip Macy
aa819acf89 White space cleanups to bring closer to RELENG_7 2008-09-16 02:03:28 +00:00
Kip Macy
af9b081c37 Remove some dead code along with gratuitous differences between HEAD and 7 2008-09-16 01:02:17 +00:00
Kip Macy
d5e2c3dd04 Fix issue with recovering from transient jumbo mbuf shortage.
Submitted by:	Chelsio Inc.
MFC after:	3 days
2008-09-09 01:36:02 +00:00
Kip Macy
706cb31f0a Add LRO and MAC statistics to exported sysctls.
Obtained from:	Chelsio Inc.
MFC after:	1 week
2008-08-13 01:30:41 +00:00
Kip Macy
25292deb42 Remove cxgb private lro implementation and switch to using system implementation.
Obtained from:	Chelsio Inc.
MFC after:	1 week
2008-08-12 00:27:32 +00:00
Kip Macy
9b4de886f9 Vendor fix for PHY problem.
Obtained from:	Chelsio Inc.
MFC after:	3 days
2008-08-11 23:01:34 +00:00
Kip Macy
4af83c8cff import vendor fixes to cxgb 2008-07-18 06:12:31 +00:00
Kip Macy
ef9e6d4c6c reduce the size of the jumbo ring on i386 and disable pcpu cluster caching 2008-03-31 21:02:27 +00:00
Kip Macy
8e10660f12 - update firmware to 5.0
- add support for T3C
- add DDP support (zero-copy receive)
- fix TOE transmit of large requests
- fix shutdown so that sockets don't remain in CLOSING state indefinitely
- register listeners when an interface is brought up after tom is loaded
- fix setting of multicast filter
- enable link at device attach
- exit tick handler if shutdown is in progress
- add helper for logging TCB
- add sysctls for dumping transmit queues

- note that TOE wxill not be MFC'd until after 7.0 has been finalized

MFC after: 3 days
2008-02-23 01:06:17 +00:00
Kip Macy
2fd79ec2de put tx queue size back to 1024 2008-01-16 00:26:04 +00:00
Kip Macy
139edb19d9 - Simplify mb_free_ext_fast
- increase asserts for mbuf accounting
- track outstanding mbufs (maps very closely to leaked)
- actually only create one thread per port if !multiq
    Oddly enough this fixes the use after free

- move txq_segs to stack in t3_encap
- add checks that pidx doesn't move pass cidx
- simplify mbuf free logic in collapse mbufs routine
2008-01-15 08:08:09 +00:00
Kip Macy
60f1e27625 - move WR_LEN in to cxgb_adapter.h add PIO_LEN to make intent clearer
- move cxgb_tx_common in to cxgb_multiq.c and rename to cxgb_tx
- move cxgb_tx_common dependencies
- further simplify cxgb_dequeue_packet for the non-multiqueue case
- only launch one service thread per port in the non-multiq case
- remove dead cleaning code from cxgb_sge.c
- simplify PIO case substantially in by returning directly from mbuf collapse
  and just using m_copydata
- remove gratuitous m_gethdr in the rx path
- clarify freeing of mbufs in collapse
2008-01-15 03:27:42 +00:00
Kip Macy
5ef02c0223 Convert over to using the multiqueue infrastructure although all calls going
through cxgb_start still end up using queue 0
2008-01-13 22:38:17 +00:00
Kip Macy
9330dbc3a1 Be more aggressive about tx cleaning - when multiples streams were running the tx
queue could fill up and stop getting cleaned.
2008-01-11 02:26:54 +00:00
Kip Macy
10b16b260c Add support for selectively dumping the state of the hardware response queue.
Change ordering of a couple of types.
2008-01-10 06:54:20 +00:00
Kip Macy
8d5754afc5 make nqsets a uint32_t so that sysctl will work
add 2 fields for allowing queue dumping
2008-01-09 08:12:24 +00:00
Kip Macy
8090c9f504 Make TCP offload work on HEAD (modulo negative interaction between sbcompress
and t3_push_frames).
 - Import latest changes to cxgb_main.c and cxgb_sge.c from toestack p4 branch
 - make driver local copy of tcp_subr.c and tcp_usrreq.c and override tcp_usrreqs so
   TOE can also functions on versions with unmodified TCP

- add cxgb back to the build
2007-12-17 08:17:51 +00:00
Kip Macy
3e96c7e790 Import updated support code for the TOM (tcp offload module). 2007-12-15 21:54:59 +00:00
Kip Macy
2de1fa86d7 pull in changes made to RELENG_6 version in the process of doing the MFC
Supported by: Chelsio
Approved by: re (blanket)
2007-09-10 00:59:51 +00:00
Kip Macy
8adc65adda Add back in support for normal mbuf chaining on RX under DISABLE_MBUF_IOVEC
Approved by: re(blanket)
Supported by: Chelsio
2007-09-09 04:34:03 +00:00
Kip Macy
5c5df3da16 - fix qset to port binding as a proper fix for the problems encountered on the 4-port
- fix the use after free seen when sending packets small enough to fit as an immediate
   and bpf peers are present
 - update to firmware rev 4.7 along with various small vendor fixes

Supported by: Chelsio
Approved by: re (blanket)
MFC after: 3 days
2007-09-09 01:28:03 +00:00
Kip Macy
7ac2e6c362 Fixes for 4 port and small packet optimization
- remove cpl->iff panic - we can't know the port number from the rspq on the 4-port
- pick the ifnet based on the interface in the CPL header
- switch to using qset 0 for egress on the 4-port for now - may change
  when we start using RSS
- move ether_ifdetach to before the port lock gets deinitialized to avoid
  hang in the case where there are BPF peers (cxgb_ioctl is called indirectly
  when BPF peers are present)
- don't call t3_mac_reset if multiport is set, this was causing tx errors
  by misconfiguring the MAC on the 4-port
- change V_TXPKT_INTF to use txpkt_intf as the interfaces are not contiguous
- free the mbuf immediately in the case where the payload is small enough to be copied
  into the rspq
- only update the coalesce timer if for a queue if packets were taken off of it
- add in missed 20ms DELAY in initializaton vsc8211

- prompt MFC as this only applies to the 4-port which is currently completely
  broken - OK'd by kensmith

Supported by: Chelsio
Approved by: re (blanket)
MFC after: 0 days
2007-08-25 21:07:37 +00:00
Kip Macy
7aff6d8ed3 forward port signedness fixes from RELENG_6
fix compile error for case where MSI_SUPPORTED not defined

Approved by: re (blanket)
2007-08-17 05:57:04 +00:00