This consolidates all of the DDP state in one place. Also, the code has
now been fixed to ensure that DDP state is only accessed for DDP
connections. This should not be a functional change but makes it cleaner
and easier to add state for other TOE socket modes in the future.
MFC after: 1 month
Sponsored by: Chelsio Communications
This used to work by accident with ld.bfd even though always_keepalive
was marked as static. LLD honors static more correctly, so export this
variable properly (including moving it into the tcp_* namespace).
Reviewed by: bz, emaste
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D14129
Properly honor the lack of the CRD_F_IV_PRESENT flag in the GCM
software fallback case for encryption requests.
Submitted by: Harsh Jain @ Chelsio
Sponsored by: Chelsio Communications
In particular, this avoids edge cases where a generated IV might be
written into the output buffer even though the request is failed with
an error.
Sponsored by: Chelsio Communications
- Extend ccr_gcm_soft() to handle requests with a non-empty payload.
While here, switch to allocating the GMAC context instead of placing
it on the stack since it is over 1KB in size.
- Allow ccr_gcm() to return a special error value (EMSGSIZE) which
triggers a fallback to ccr_gcm_soft(). Move the existing empty
payload check into ccr_gcm() and change a few other cases
(e.g. large AAD) to fallback to software via EMSGSIZE as well.
- Add a new 'sw_fallback' stat to count the number of requests
processed via the software fallback.
Submitted by: Harsh Jain @ Chelsio (original version)
Sponsored by: Chelsio Communications
This works around an issue in the T6 that can result in DMA engine
stalls if an error occurs while processing a DSGL entry with a length
larger than 2KB.
Submitted by: Harsh Jain @ Chelsio
Sponsored by: Chelsio Communications
Most crypto requests will not trigger this condition, but a request
with a highly-fragmented data buffer (and a resulting "large" S/G
list) could trigger it.
Sponsored by: Chelsio Communications
The T6 can hang when processing certain AEAD requests if the request
sets a flag asking the crypto engine to discard the input IV and AAD
rather than copying them into the output buffer. The existing driver
always discards the IV and AAD as we do not need it. As a workaround,
allocate a single "dummy" buffer when the ccr driver attaches and
change all AEAD requests to write the IV and AAD to this scratch
buffer. The contents of the scratch buffer are never used (similar to
"bogus_page"), and it is ok for multiple in-flight requests to share
this dummy buffer.
Submitted by: Harsh Jain @ Chelsio (original version)
Sponsored by: Chelsio Communications
The T6 crypto engine's control messages only support a total AAD
length (including the prefixed IV) of 511 bytes. Reject requests with
large AAD rather than returning incorrect results.
Sponsored by: Chelsio Communications
Combined authentication-encryption and GCM requests already stored the
IV in the immediate explicitly. This extends this behavior to block
cipher requests to work around a firmware bug. While here, simplify
the AEAD and GCM handlers to not include always-true conditions.
Submitted by: Harsh Jain @ Chelsio
Sponsored by: Chelsio Communications
Uses of mallocarray(9).
The use of mallocarray(9) has rocketed the required swap to build FreeBSD.
This is likely caused by the allocation size attributes which put extra pressure
on the compiler.
Given that most of these checks are superfluous we have to choose better
where to use mallocarray(9). We still have more uses of mallocarray(9) but
hopefully this is enough to bring swap usage to a reasonable level.
Reported by: wosch
PR: 225197
Focus on code where we are doing multiplications within malloc(9). None of
these is likely to overflow, however the change is still useful as some
static checkers can benefit from the allocation attributes we use for
mallocarray.
This initial sweep only covers malloc(9) calls with M_NOWAIT. No good
reason but I started doing the changes before r327796 and at that time it
was convenient to make sure the sorrounding code could handle NULL values.
Unconditional 32-bit shift is not endianness-safe.
Modify the logic to work both on LE and BE.
Submitted by: Wojciech Macek <wma@freebsd.org>
Reviewed by: np
Obtained from: Semihalf
Sponsored by: IBM, QCM Technologies
Differential revision: https://reviews.freebsd.org/D13102
leaves the firmware event queue (fwq) as the only queue that can take
interrupts for others.
This simplifies cfg_itype_and_nqueues and queue allocation in the driver
at the cost of a little (never?) used configuration. It also allows
service_iq to be split into two specialized variants in the future.
MFC after: 2 months
Sponsored by: Chelsio Communications
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
tunables. Add num_vis to the intrs_and_queues structure as it affects
the number of interrupts requested and queues created. In future
cfg_itype_and_nqueues might lower it incrementally instead of going
straight to 1 when enough interrupts aren't available.
Sponsored by: Chelsio Communications
The setbit/clearbit pair casts the bitfield pointer
to uint8_t* which effectively treats its contents as
little-endian variable. The ffs() function accepts int as
the parameter, which is big-endian. Use uint8_t here to
avoid mismatch, as we have only 4 doorbells.
Submitted by: Wojciech Macek <wma@freebsd.org>
Reviewed by: np
Obtained from: Semihalf
Sponsored by: QCM Technologies
Differential revision: https://reviews.freebsd.org/D13084
different from hardware defaults. The congestion channel map, which is
still fixed, needs to be tracked separately now. Change the congestion
setting for TOE rx queues to match the drivers on other OSes while here.
MFC after: 2 weeks
Sponsored by: Chelsio Communications
iWarp and RoCE in ibcore. The selection of RDMA_PS_TCP can not be used
to indicate iWarp protocol use. Backport the proper IB device
capabilities from Linux upstream to distinguish between iWarp and
RoCE. Only allocate the additional socket required for iWarp for RDMA
IDs when at least one iWarp device present. This resolves
interopability issues between iWarp and RoCE in ibcore
Reviewed by: np @
Differential Revision: https://reviews.freebsd.org/D12563
Sponsored by: Mellanox Technologies
MFC after: 3 days
r324539 gathered some vnet decls into netinet/tcp_var.h, so that they
are now redundant in dev/cxgbe/tom/{t4_cpl_io.c,t4_ddp.c}. This triggers
gcc -Wredundant-decls.
Reviewed by: np
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12674
All of these arguments are stored in m_ext, so there is no reason
to pass them in the argument list. Not all functions need the second
argument, some don't even need the first one. The second argument
lives in next cache line, so not dereferencing it is a performance
gain. This was discovered in sendfile(2), which will be covered by
next commits.
The second goal of this commit is to bring even more flexibility
to m_ext mbufs, allowing to create more fields in m_ext, opaque to
the generic mbuf code, and potentially set and dereferenced by
subsystems.
Reviewed by: gallatin, kbowling
Differential Revision: https://reviews.freebsd.org/D12615
Changes since 1.16.26.0 for all three firmwares are listed below. This
list was obtained from the Release Notes of the Chelsio Unified Wire
v3.5.05 release for Linux.
T6 Firmware
++++++++++++
================================================================================
Version : 1.16.63.0
Date : 09/29/2017
================================================================================
Fixes
-----
BASE:
- Fixed a fw crash when configured traffic rate limit is less than 10kbps.
- Fixed traffic rate limiting for smaller traffic rate value.
ETH:
- Fixed 40G link failure when interface is toggled.
- Fixed adapter crash when interface is toggled during traffic.
- Fixed 25G link failure when PEER only supports consortium mode autoneg
for 25G.
- Fixed 100G optics link failure when cable is plugged in after bringing up
the interface.
- Enable RS FEC as default if speed is 100G.
- Fixed DCBX configuration refresh failure.
OFLD
- Fixed 0B iWARP ingress read failure.
- Fixed iWARP SRQ reuse failure.
FOiSCSI:
- Fixed vlan interface ping failure.
- Fixed target discovery failures.
- Fixed mutual chap login failure.
================================================================================
Version : 1.16.59.0
Date : 09/05/2017
================================================================================
FIXES
-----
BASE:
- Fixed fw crash caused by MC parity error in SO adapters.
- Generate Timer0Int interrupt if fw crashes due to unaligned access error. Host
driver must look into PCIE_FW register to see if any fw fatal error has
encountered. If PCIE_FW doesn't indicate any error then driver must ignore this
interrupt.
- Fixed receive buffer threshold settings which was resulting in error frames on
receive side.
ETH:
- Fixed an issue in connection traffic shaping when
FLOWC_WR->FW_FLOWC_MNEM_SCHEDCLASS is not received in first WR on the connection.
- Fixed link failure when speed is changed from 10G-1G-10G due to incorrect flag
check.
- Fixed improper LED behaviour for blink test and when traffic is running.
- Removed storage of previous fec settings from fw. Driver needs to pass the user
settings whenever a new module is plugged in as fw resets these when a module is
unplugged.
OFLD
- OVS offload: TP cache is flushed periodically to get the accuate filters stats
(hit count).
ENHANCEMENTS
------------
BASE:
- Ring backbone feature added. New FW_PARAMS_PARAM_DEV_RING_BACKBONE param type
added to query and enable ring backbone support.
- VNI support added for filtering. New entry_type FW_VI_MAC_TYPE_EXACTMAC_VNI
added to FW_VI_MAC_CMD.
- Added new API FW_PARAM_PARAM_DEV_MPSBGMAP to read the priority to buffer group
mapping for the ports.
- FW_PARAMS_PARAM_DEV_TPCHMAP API added to read the port to channel mapping.
- HMA (Host memory access) support added. New FW_HMA_CMD and
FW_PARAMS_PARAM_DEV_HMA_SIZE added to query and configure the HMA. It
enables the memfree support (256 connections) for iwarp.
- PTP support enabled.
ETH:
- Added consortium mode 50G support.
- Added the ability to allow only selected speeds to be advertised during auto
negotiation.
- Increased port capability from 16 to 32 bits to support more speeds.
FW_PARAMS_PARAM_PFVF_PORT_CAPS32 added to query whether fw supports 16 or 32
bit port capability.
OFLD:
- RDMA Write with immediate support added (iwarp 2.0 feature)
- FW_TLS_KEYCTX_TX_WR removed and security key management moved to driver.
- 256 offloaded connections support for iwarp on SO adapters.
iSCSI:
- New param FW_PARAMS_PARAM_DEV_PPOD_EDRAM added for iscsi ppod configuration
in EDRAM (performance improvement).
FOiSCSI:
- iSCSI Command offload target support added.
FOFCoE:
- FCoE support enabled.
================================================================================
Version : 1.16.43.0
Date : 05/05/2017
================================================================================
FIXES
-----
BASE:
- Fixed default DCB mode to AUTO.
- Fixed DCBX bugs when AUTO mode is configured in config file.
- Fixed an issue where even after removing PFC from switch, PFC wasn't getting
reset.
- Fixed DDR3/DDR4 ECC errors.
- Fixed an FLR issue where FLR completion was going to host before FLR
processing is finished in fw.
ETH:
- Fixed bug in writing multi-bytes using i2c interface.
- Fixed the link failure when optical cable is inserted into the QSA module
after loading the driver.
- Fixed false link up when peer interface was brought down.
- Enabling RS FEC by default for 100Gbase-SR4 according to 802.3BJ standard.
- Fixed bugs related to negotiated fec based local/peer fec ability and request.
- Fixed auto-neg failure with few switches.
- T6 Performance improvement fixes.
OFLD
- Fixed an extra credit issue if FW_RI_TYPE_FINI is delayed in fw due to
backpressure.
- Added a new queue type FW_IQ_TYPE_VF_CQ to handle the FW_PARAMS_PARAM_DMAQ*
commands. queue type will be part of the FW_PARAMS_PARAM_DMAQ_IQ_INTIDX
value. Used in guest RDMA (RDMA from VM/VF) usecase.
- T6 Crypto Coprocessor mode bug fixes.
- T6 Crypto TLS-inline mode bug fixes.
ENHANCEMENTS
------------
BASE:
- Added new API FW_PARAM_PARAM_DEV_MPSBGMAP to read the priority to buffer
group mapping for the ports.
ETH:
- Added broadcom consortium next page support for 25G CR.
This can be enabled using flags=an_brcm option in the t6-config.txt file.
- Added spider mode support.
- Added support for 10G-BaseT converter sfp+ module.
- Added support for additional 25G/100G cables.
- Added support to enable/disable auto-neg using ethtool.
================================================================================
Version : 1.16.33.0
Date : 02/24/2017
================================================================================
Fixes
-----
BASE:
- Fixed DDR4 uncorrectable errors.
ETH:
- Enabled link auto negotiation (AN) by default in config file.
- Added AN and FEC control api. Host driver and application can enable/disable
AN and FEC.
ENHANCEMENTS
------------
BASE:
- Enabled High priorty filter.
- Added T6425 adapter support.
ETH:
- Added new workrequest ETH_TX_PKTS2_WR (see fw api document for more details).
================================================================================
Version : 1.16.29.0
Date : 01/27/2017
================================================================================
FIXES
-----
BASE:
- Set multiple fec values only if AN is enabled in config file and when module
is connected.
- Fixed intermittent DDR3/4 ECC errors.
- max number of ethctrl queue in VF set to 2 (reverted the last change
because it causes problem in VF drivers).
ETH:
- Made devlog more verbose by printing cable information in redable form.
- Updated AN settings to work with more 25G/100G switches.
- Added support for more SFP28/QSFP28 cables.
- Fixed an issue of link going down after few hours of idle time.
OFLD:
- Fixed an issue in TLS which was causing fw crash on running TLS traffic.
FOiSCSI:
- Fixed the failure of PXE boot OS install on an iscsi lun.
ENHANCEMENTS
------------
OFLD:
- Added filtering support for NAT. New WR FW_FILTER2_WR and
FW_PARAMS_PARAM_DEV_FILTER2_WR added for the same.
- Added RDMA guest mode (mode 3 or RDMA from VF) support.
================================================================================
T5 Firmware
++++++++++++
================================================================================
Version : 1.16.63.0
Date : 09/29/2017
================================================================================
Fixes
-----
BASE:
- Fixed offload memory overcommit in case of SO adapter.
ETH:
- Fixed DCBX configuration refresh failure.
OFLD
- Fixed 0B iWARP ingress read failure.
FOiSCSI:
- Fixed vlan interface ping failure.
================================================================================
Version : 1.16.59.0
Date : 09/05/2017
================================================================================
FIXES
-----
BASE:
- Fixed an FLR issue which was causing error when VF attached VM was powered on.
ETH:
- Fixed an issue in connection traffic shaping when
FLOWC_WR->FW_FLOWC_MNEM_SCHEDCLASS is not received in first WR on the connection.
- Fixed link failure when speed is changed from 10G-1G-10G due to incorrect flag
check.
- Fixed T580 link failure with few switches which take more time for
establishing link.
ENHANCEMENTS
------------
BASE:
- Ring backbone feature added. New FW_PARAMS_PARAM_DEV_RING_BACKBONE param type
added to query and enable ring backbone support.
- Added new API FW_PARAM_PARAM_DEV_MPSBGMAP to read the priority to buffer group
mapping for the ports.
- FW_PARAMS_PARAM_DEV_TPCHMAP API added to read the port to channel mapping.
FOiSCSI:
- iSCSI Command offload target support added.
================================================================================
Version : 1.16.43.0
Date : 05/05/2017
================================================================================
FIXES
-----
BASE:
- Fixed default DCB mode to AUTO.
- Fixed DCBX bugs when AUTO mode is configured in config file.
- Fixed an issue where even after removing PFC from switch, PFC wasn't getting
reset.
ETH:
- Fixed bug in writing multi-bytes using i2c interface.
- Fixed the link failure when optical cable is inserted into the QSA module
after loading the driver.
OFLD
- Fixed an extra credit issue if FW_RI_TYPE_FINI is delayed in fw due to
backpressure.
- Added a new queue type FW_IQ_TYPE_VF_CQ to handle the FW_PARAMS_PARAM_DMAQ*
commands. queue type will be part of the FW_PARAMS_PARAM_DMAQ_IQ_INTIDX
value. Used in guest RDMA (RDMA from VM/VF) usecase.
ENHANCEMENTS
------------
BASE:
- Added new API FW_PARAM_PARAM_DEV_MPSBGMAP to read the priority to buffer
group mapping for the ports.
================================================================================
Version : 1.16.33.0
Date : 02/24/2017
================================================================================
ENHANCEMENTS
------------
ETH:
- Added new workrequest ETH_TX_PKTS2_WR (see fw api document for more details).
================================================================================
Version : 1.16.29.0
Date : 01/27/2017
================================================================================
FIXES
-----
BASE:
- max number of ethctrl queue in VF set to 2 (reverted the last change
because it causes problem in VF drivers).
FOiSCSI:
- Fixed the failure of PXE boot OS install on an iscsi lun.
ENHANCEMENTS
------------
OFLD:
- Added filtering support for NAT. New WR FW_FILTER2_WR and
FW_PARAMS_PARAM_DEV_FILTER2_WR added for the same.
- Added RDMA guest mode (mode 3 or RDMA from VF) support.
================================================================================
T4 Firmware
+++++++++++
================================================================================
Version : 1.16.63.0
Date : 09/29/2017
================================================================================
Fixes
-----
ETH:
- Fixed DCBX configuration refresh failure.
FOiSCSI:
- Fixed vlan interface ping failure.
================================================================================
Version : 1.16.59.0
Date : 09/05/2017
================================================================================
FIXES
-----
ETH:
- Fixed an issue in connection traffic shaping when
FLOWC_WR->FW_FLOWC_MNEM_SCHEDCLASS is not received in first WR on the connection.
ENHANCEMENTS
------------
BASE:
- FW_PARAMS_PARAM_DEV_TPCHMAP API added to read the port to channel mapping.
================================================================================
Version : 1.16.43.0
Date : 05/05/2017
================================================================================
FIXES
-----
BASE:
- Fixed default DCB mode to AUTO.
- Fixed DCBX bugs when AUTO mode is configured in config file.
- Fixed an issue where even after removing PFC from switch, PFC wasn't getting
reset.
ETH:
- Fixed bug in writing multi-bytes using i2c interface.
OFLD
- Fixed an extra credit issue if FW_RI_TYPE_FINI is delayed in fw due to
backpressure.
- Added a new queue type FW_IQ_TYPE_VF_CQ to handle the FW_PARAMS_PARAM_DMAQ*
commands. queue type will be part of the FW_PARAMS_PARAM_DMAQ_IQ_INTIDX
value. Used in guest RDMA (RDMA from VM/VF) usecase.
ENHANCEMENTS
------------
BASE:
- Added new API FW_PARAM_PARAM_DEV_MPSBGMAP to read the priority to buffer
group mapping for the ports.
Obtained from: Chelsio Communications
MFC after: 2 weeks
Sponsored by: Chelsio Communications
To optimize the case of ping-ponging between two buffers, the DDP code
caches the last two buffers used keeping the pages wired and page pods
stored in the NIC's RAM. If a new aio_read() request uses one of the
same buffers, then the work of holding pages, etc. can be avoided.
However, the starting virtual address of an aio buffer was not saved,
only the page count, length, and initial page offset. Thus, an
aio_read() request could match a different buffer in the address
space. (Earlier during development vm_fault_hold_quick_pages() was
always called and the vm_page_t values were compared, but that was
eventually removed without being adequately replaced.) Fix by storing
the starting virtual address and comparing that (along with other
fields) to determine if a buffer can be reused.
MFC after: 3 days
Sponsored by: Chelsio Communications
