629 Commits

Author SHA1 Message Date
jhb
70a3719bb8 Add a driver for the Chelsio T6 crypto accelerator engine.
The ccr(4) driver supports use of the crypto accelerator engine on
Chelsio T6 NICs in "lookaside" mode via the opencrypto framework.

Currently, the driver supports AES-CBC, AES-CTR, AES-GCM, and AES-XTS
cipher algorithms as well as the SHA1-HMAC, SHA2-256-HMAC, SHA2-384-HMAC,
and SHA2-512-HMAC authentication algorithms.  The driver also supports
chaining one of AES-CBC, AES-CTR, or AES-XTS with an authentication
algorithm for encrypt-then-authenticate operations.

Note that this driver is still under active development and testing and
may not yet be ready for production use.  It does pass the tests in
tests/sys/opencrypto with the exception that the AES-GCM implementation
in the driver does not yet support requests with a zero byte payload.

To use this driver currently, the "uwire" configuration must be used
along with explicitly enabling support for lookaside crypto capabilities
in the cxgbe(4) driver.  These can be done by setting the following
tunables before loading the cxgbe(4) driver:

    hw.cxgbe.config_file=uwire
    hw.cxgbe.cryptocaps_allowed=-1

MFC after:	1 month
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D10763
2017-05-17 22:13:07 +00:00
jhb
3b9fd823d7 Add support for child devices that aren't ports.
Invoke any identify routines of child drivers during attach before attaching
children, and delete any remaining devices after deleting ports.

MFC after:	1 month
Sponsored by:	Chelsio Communications
2017-05-16 23:18:50 +00:00
np
38452b717f cxgbe(4): Avoid an out of bounds access when an attempt to unbind a tx
queue from a traffic class fails.

Reported by:	x ksi <s3810 at pjwstk edu pl>
MFC after:	3 days
2017-05-15 18:18:32 +00:00
np
2910c97c08 cxgbe(4): netmap-only interrupts for a VI do not have an associated rxq
or ofld_rxq and should be ignored by vi_intr_iq.

MFC after:	3 days.
Sponsored by:	Chelsio Communications
2017-05-14 09:07:13 +00:00
np
2a0af286de Adjust whitespace and fix a comment. No functional change.
MFC after:	3 days
2017-05-10 00:42:28 +00:00
np
1fa4e09fb2 cxgbe(4): Do not assume that if_qflush is always followed by inteface-down.
MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-05-09 18:33:41 +00:00
np
bbcaf715fe cxgbe(4): Fixes related to the knob that controls link autonegotiation.
- Do not leak the adapter lock in sysctl_autoneg.
- Accept only 0 or 1 as valid settings for autonegotiation.
- A fixed speed must be requested by the driver when autonegotiation is
  disabled otherwise the firmware will reject the l1cfg command.  Use
  the top speed supported by the port for now.

MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-05-09 08:08:28 +00:00
np
a1d65b9b7e cxgbe/t4_tom: Per-connection rate limiting for TCP sockets handled by
the TOE.  For now this capability is always enabled in kernels with
options RATELIMIT.  t4_tom will check if_capenable once the base driver
gets code to support rate limiting for any socket (TOE or not).

This was tested with iperf3 and netperf ToT as they already support
SO_MAX_PACING_RATE sockopt.  There is a bug in firmwares prior to
1.16.45.0 that affects the BSD driver only and results in rate-limiting
at an incorrect rate.  This will resolve by itself as soon as 1.16.45.0
or later firmware shows up in the driver.

Relnotes:	Yes
Sponsored by:	Chelsio Communications
2017-05-05 20:06:49 +00:00
np
5d1cab1e8c cxgbe(4): The Tx scheduler initialization either works or doesn't. It
doesn't need a refresh in either case.

Sponsored by:	Chelsio Communications
2017-05-05 19:34:05 +00:00
np
911432e2f3 cxgbe(4): Update the VF device ids too. This should have been part
of r317820.

Reported by:	jhb@
MFC after:	1 week
Sponsored by:	Chelsio Communications
2017-05-05 16:52:25 +00:00
np
c69686cf8e cxgbe(4): Update the list of PCIe devices claimed by the driver. At
this point any board with a T6 should just work.

Obtained from:	Chelsio Communications
MFC after:	1 week
Sponsored by:	Chelsio Communications
2017-05-05 00:54:23 +00:00
np
e18d78b84b cxgbe(4): Support routines for Tx traffic scheduling.
- Create a new file, t4_sched.c, and move all of the code related to
  traffic management from t4_main.c and t4_sge.c to this file.
- Track both Channel Rate Limiter (ch_rl) and Class Rate Limiter (cl_rl)
  parameters in the PF driver.
- Initialize all the cl_rl limiters with somewhat arbitrary default
  rates and provide routines to update them on the fly.
- Provide routines to reserve and release traffic classes.

MFC after:	1 month
Sponsored by:	Chelsio Communications
2017-05-02 20:38:10 +00:00
np
6e2033d0db cxgbe/iw_cxgbe: Pull in some updates to c4iw_wait_for_reply from the
iw_cxgb4 Linux driver.

Obtained from:	Chelsio Communications
MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-04-25 16:54:27 +00:00
mav
587866de56 Use proper alignment constant for uma_zcreate().
Previous code panicked on KASSERT with INVARIANTS enabled.

MFC after:	2 weeks
2017-04-24 08:44:51 +00:00
np
9ad04f7ac0 cxgbe: Add tunables to control the number of LRO entries and the number
of rx mbufs that should be presorted before LRO.  There is no change in
default behavior.

MFC after:	1 week
Sponsored by:	Chelsio Communications
2017-04-17 09:00:20 +00:00
np
66dab3d0be cxgbe: Add a tunable to configure the SGE time scaler, which is
available starting with T6.  The values in the timer holdoff registers
are multiplied by the scaling factor before use.

dev.<nexus>.<n>.holdoff_timers shows the final values of the
timers in microseconds.

MFC after:	1 week
Sponsored by:	Chelsio Communications
2017-04-15 17:00:50 +00:00
np
b03d870c8b cxgbe/iw_cxgbe: Report the actual values of various parameters as
configured by the firmware.

MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-04-14 20:15:17 +00:00
np
400490a9ed cxgbe/iw_cxgbe: Report accurate page_size_cap in ib_query_device.
MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-04-14 19:18:50 +00:00
np
ebc68d03bd cxgbe/iw_cxgbe: hw supports 64K (not 32K) Protection Domains.
MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-04-14 19:15:31 +00:00
np
6193444cfe cxgbe: Query some more RDMA related parameters from the firmware.
MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-04-13 17:16:36 +00:00
np
e5490560a2 cxgbe/iw_cxgbe: Remove another bad cast. This should have been
included in r316571.

MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-04-06 16:19:19 +00:00
np
0edd303bef cxgbe/iw_cxgbe: Replace a magic constant with something more readable
(and accurate).

T4 and later have an extra bit for page shift so the maximum page size
is 8TB (shift of 12 + 31) instead of 128MB (12 + 15).  This saves space
in the chip's PBL (physical buffer list) when registering very large
memory regions.

MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-04-06 14:50:15 +00:00
np
6b66ab3ae3 cxgbe/iw_cxgbe: Remove bad cast that resulted in incorrect length for
memory regions larger than 4GB.

MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-04-06 13:58:59 +00:00
np
a931b768e6 cxgbe(4): Program the global RSS key once instead of once per ifnet.
MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-04-04 23:14:03 +00:00
np
56479f976b cxgbe: Don't call t4_edc_err_read for errors not related to the EDCs.
MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-03-29 19:25:31 +00:00
np
9423eb55d8 cxgbe/iw_cxgbe: T6 has no limit on the amount of memory that can be
registered in one ib_reg_phys_mr.
2017-03-28 23:39:11 +00:00
np
ed8215cf86 cxgbe/iw_cxgbe: Defer the handling of error CQEs and RDMA_TERMINATE to
the thread that deals with socket state changes.  This eliminates
various bad races with the ithread.

Submitted by:	KrishnamRaju ErapaRaju @ Chelsio (original version)
MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-03-27 22:00:03 +00:00
np
1aa8f06390 cxgbe/iw_cxgbe: Remove unused code.
MFC after:	3 days
2017-03-27 03:11:51 +00:00
np
ac53ab4652 cxgbe/iw_cxgbe: allocations that use GFP_KERNEL (which is M_WAITOK on
FreeBSD) cannot fail.

MFC after:	3 days
2017-03-25 02:28:21 +00:00
np
c4f39bcc70 cxgbe/iw_cxgbe: alloc_ep expects a gfp_t, and it's always ok to sleep during
alloc_ep.
2017-03-25 01:45:04 +00:00
np
2deba1790b cxgbe/iw_cxgbe: c4iw_connect should always returns a -ve errno on failure.
MFC after:	3 days
2017-03-25 01:38:17 +00:00
np
12ad4b9b06 cxgbe/iw_cxgbe: Use the socket and not the toepcb to reach for the
inpcb.  t4_tom detaches the inpcb from the toepcb as soon as the
hardware is done with the connection (in final_cpl_received) but the
socket is around as long as the cm_id and the rest of iWARP state is.

This fixes an intermittent NULL dereference during abort.

Submitted by:	KrishnamRaju ErapaRaju @ Chelsio
MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-03-15 19:10:04 +00:00
np
41ef1a63d4 cxgbe(4): Fix an always-true assertion (reported by PVS-Studio).
sys/dev/cxgbe/t4_main.c: PVS-Studio: Expression is Always True (CWE-571) (3)

PR:		217746
Submitted by:	Svyatoslav (razmyslov at viva64 com)
MFC after:	1 week
2017-03-13 17:16:29 +00:00
np
cf8bef5834 cxgbe/iw_cxgbe: Abort connection if there is an error during c4iw_modify_qp.
Submitted by:	KrishnamRaju ErapaRaju @ Chelsio
MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-03-07 01:13:26 +00:00
np
59a86e7558 cxgbe/iw_cxgbe: Implement sq/rq drain operation.
ULPs can set a qp's state to ERROR and then post a work request on the
sq and/or rq.  When the reply for that work request comes back it is
guaranteed that all previous work requests posted on that queue have
been drained.

Obtained from:	Chelsio Communications
MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-03-03 03:07:54 +00:00
np
4647782637 cxgbe/iw_cxgbe: Do not check the size of the memory region being
registered.  T4/5/6 have no internal limit on this size.  This is
probably a copy paste from the T3 iw_cxgb driver.

MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-03-01 20:23:21 +00:00
np
30189d73e6 cxgbe/iw_cxgbe: fix various double-close panics with iWARP sockets.
Sockets representing the TCP endpoints for iWARP connections are
allocated by the ibcore module.  Before this revision they were closed
either by the ibcore module or the iw_cxgbe hardware driver depending on
the state transitions during connection teardown.  This is error prone
and there were cases where both iw_cxgbe and ibcore closed the socket
leading to double-free panics.  The fix is to let ibcore close the
sockets it creates and never do it in the driver.

- Use sodisconnect instead of soclose (preceded by solinger = 0) in the
  driver to tear down an RDMA connection abruptly.  This does what's
  intended without releasing the socket's fd reference.

- Close the socket in ibcore when the iWARP iw_cm_id is destroyed.  This
  works for all kinds of sockets: clients that initiate connections,
  listeners, and sockets accepted off of listeners.

Reviewed by:	Steve Wise @ Open Grid Computing, hselasky@
MFC after:	3 days
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D9796
2017-02-28 19:27:41 +00:00
np
402e9063ac cxgbe/iw_cxgbe: Minor changes for T6.
Submitted by:	Krishnamraju Eraparaju at Chelsio
Sponsored by:	Chelsio Communications
2017-02-23 19:02:40 +00:00
np
109b9048f8 cxgbe/t4_tom: Fix CLIP entry refcounting on the passive side. Every
IPv6 connection being handled by the TOE should have a reference on its
CLIP entry.

Sponsored by:	Chelsio Communications
2017-02-06 17:48:25 +00:00
np
603f7d66d4 cxgbe(4): Allow tunables that control the number of queues to be set to
'-n' to tell the driver to create _up to_ 'n' queues if enough cores are
available.  For example, setting hw.cxgbe.nrxq10g="-32" will result in
16 queues if the system has 16 cores, 32 if it has 32.

There is no change in the default number of queues of any type.

MFC after:	1 week
Sponsored by:	Chelsio Communications
2017-02-06 05:19:29 +00:00
jhb
ff2da518de Fix a couple of issues with t4iov probe and attach.
- Check for Chelsio vendor ID in probe routines.
- Fail attach instead of faulting if pci_find_dbsf() doesn't find a
  device.

PR:		216539
Reported by:	asomers
Tested by:	Dave Baukus <daveb@spectralogic.com>
MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-01-31 18:54:13 +00:00
jhb
ba9656082f Unregister CPL handlers for TOE-related messages when unloading TOM.
MFC after:	1 week
Sponsored by:	Chelsio Communications
2017-01-27 23:08:30 +00:00
jhb
1b5ced141f Don't drop a reference to the TOE PCB in undo_offload_socket().
undo_offload_socket() is only called by t4_connect() during a connection
setup failure, but t4_connect() still owns the TOE PCB and frees ita
after undo_offload_socket() returns.  Release a reference in
undo_offload_socket() resulted in a double-free which panicked when
t4_connect() performed the second free.  The reference release was
added to undo_offload_socket() incorrectly in r299210.

MFC after:	1 week
Sponsored by:	Chelsio Communications
2017-01-27 23:03:28 +00:00
np
88dc5fd40f cxgbe/tom: Fix a case where do_pass_accept_req wasn't properly restoring
the VNET.  This should have been in r311949.

MFC after:	2 days
2017-01-18 03:35:42 +00:00
np
989ae30975 cxgbe/tom: Add VIMAGE support to the TOE driver.
Active Open:
- Save the socket's vnet at the time of the active open (t4_connect) and
  switch to it when processing the reply (do_act_open_rpl or
  do_act_establish).

Passive Open:
- Save the listening socket's vnet in the driver's listen_ctx and switch
  to it when processing incoming SYNs for the socket.
- Reject SYNs that arrive on an ifnet that's not in the same vnet as the
  listening socket.

CLIP (Compressed Local IPv6) table:
- Add only those IPv6 addresses to the CLIP that are in a vnet
  associated with one of the card's ifnets.

Misc:
- Set vnet from the toepcb when processing TCP state transitions.
- The kernel sets the vnet when calling the driver's output routine
  so t4_push_frames runs in proper vnet context already.  One exception
  is when incoming credits trigger tx within the driver's ithread.  Set
  the vnet explicitly in do_fw4_ack for that case.

MFC after:	3 days
Sponsored by:	Chelsio Communications
2017-01-11 23:48:17 +00:00
np
6110c4230d The iw_cxgb and iw_cxgbe drivers should not use a FreeBSD device_t where
a linuxkpi style device is expected.  If OFED/linuxkpi actually starts
using this field then we'll have to figure out whether to create fake
devices for these drivers or have linuxkpi deal with NULL device.

This mismatch was first reported as part of D6585.
2017-01-10 18:39:53 +00:00
np
72aca66b1d cxgbe(4): Attach to the 2x25 debug card. This is for internal use only.
MFC after:	3 days
2017-01-10 01:36:50 +00:00
np
e5ecbb057d cxgbe(4): Refresh t4_msg.h, mainly for definitions related to the crypto
engine.

Obtained from:	Chelsio Communications
MFC after:	2 weeks
Sponsored by:	Chelsio Communications
2017-01-10 01:30:41 +00:00
np
3d57498691 cxgbe(4): Enable automatic cidx flush for all control queues.
MFC after:	3 days
2017-01-09 22:20:09 +00:00
np
429237b58f cxgbe(4): The wraparound logic in start_wrq_wr() should not get involved
in work requests that end at the end of the descriptor ring, even though
the pidx wraps around to 0.

MFC after:	3 days
2017-01-09 22:18:08 +00:00