d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Support for TLS offload of TOE connections on T6 adapters.

Browse Source 
The TOE engine in Chelsio T6 adapters supports offloading of TLS
encryption and TCP segmentation for offloaded connections.  Sockets
using TLS are required to use a set of custom socket options to upload
RX and TX keys to the NIC and to enable RX processing.  Currently
these socket options are implemented as TCP options in the vendor
specific range.  A patched OpenSSL library will be made available in a
port / package for use with the TLS TOE support.

TOE sockets can either offload both transmit and reception of TLS
records or just transmit.  TLS offload (both RX and TX) is enabled by
setting the dev.t6nex.<x>.tls sysctl to 1 and requires TOE to be
enabled on the relevant interface.  Transmit offload can be used on
any "normal" or TLS TOE socket by using the custom socket option to
program a transmit key.  This permits most TOE sockets to
transparently offload TLS when applications use a patched SSL library
(e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
library).  Receive offload can only be used with TOE sockets using the
TLS mode.  The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
list of TCP port numbers.  Any connection with either a local or
remote port number in that list will be created as a TLS socket rather
than a plain TOE socket.  Note that although this sysctl accepts an
arbitrary list of port numbers, the sysctl(8) tool is only able to set
sysctl nodes to a single value.  A TLS socket will hang without
receiving data if used by an application that is not using a patched
SSL library.  Thus, the tls_rx_ports node should be used with care.
For a server mostly concerned with offloading TLS transmit, this node
is not needed as plain TOE sockets will fall back to software crypto
when using an unpatched SSL library.

New per-interface statistics nodes are added giving counts of TLS
packets and payload bytes (payload bytes do not include TLS headers or
authentication tags/MACs) offloaded via the TOE engine, e.g.:

dev.cc.0.stats.rx_tls_octets: 149
dev.cc.0.stats.rx_tls_records: 13
dev.cc.0.stats.tx_tls_octets: 26501823
dev.cc.0.stats.tx_tls_records: 1620

TLS transmit work requests are constructed by a new variant of
t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.

TLS transmit work requests require a buffer containing IVs.  If the
IVs are too large to fit into the work request, a separate buffer is
allocated when constructing a work request.  This buffer is associated
with the transmit descriptor and freed when the descriptor is ACKed by
the adapter.

Received TLS frames use two new CPL messages.  The first message is a
CPL_TLS_DATA containing the decryped payload of a single TLS record.
The handler places the mbuf containing the received payload on an
mbufq in the TOE pcb.  The second message is a CPL_RX_TLS_CMP message
which includes a copy of the TLS header and indicates if there were
any errors.  The handler for this message places the TLS header into
the socket buffer followed by the saved mbuf with the payload data.
Both of these handlers are contained in tom/t4_tls.c.

A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
including send_rx_credits(), a new send_rx_modulate(), and
t4_close_conn().

TLS keys for both transmit and receive are stored in onboard memory
in the NIC in the "TLS keys" memory region.

In some cases a TLS socket can hang with pending data available in the
NIC that is not delivered to the host.  As a workaround, TLS sockets
are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
any data is read from a TLS socket.  In addition, a fallback timer will
periodically send CPL_RX_DATA_ACK messages to the NIC for connections
that are still in the handshake phase.  Once the connection has
finished the handshake and programmed RX keys via the socket option,
the timer is stopped.

A new function select_ulp_mode() is used to determine what sub-mode a
given TOE socket should use (plain TOE, DDP, or TLS).  The existing
set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
handles initialization of TLS-specific state when necessary in
addition to DDP-specific state.

Since TLS sockets do not receive individual TCP segments but always
receive full TLS records, they can receive more data than is available
in the current window (e.g. if a 16k TLS record is received but the
socket buffer is itself 16k).  To cope with this, just drop the window
to 0 when this happens, but track the overage and "eat" the overage as
it is read from the socket buffer not opening the window (or adding
rx_credits) for the overage bytes.

Reviewed by:	np (earlier version)
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D14529
This commit is contained in:
jhb 2018-03-13 23:05:51 +00:00
parent 9af12fb91a
commit 0688d78f09
12 changed files with 2566 additions and 50 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
sys/dev/cxgbe/adapter.h
View File

@ -297,6 +297,10 @@ struct port_info {
struct port_stats stats;
u_int tnl_cong_drops;
u_int tx_parse_error;
u_long tx_tls_records;
u_long tx_tls_octets;
u_long rx_tls_records;
u_long rx_tls_octets;
struct callout tick;
};

6
sys/dev/cxgbe/firmware/t6fw_cfg.txt
View File

@ -163,10 +163,12 @@
nserver = 512
nhpfilter = 0
nhash = 16384
protocol = ofld, rddp, rdmac, iscsi_initiator_pdu, iscsi_target_pdu, iscsi_t10dif, crypto_lookaside
protocol = ofld, rddp, rdmac, iscsi_initiator_pdu, iscsi_target_pdu, iscsi_t10dif, tlskeys, crypto_lookaside
tp_l2t = 4096
tp_ddp = 2
tp_ddp_iscsi = 2
tp_tls_key = 3
tp_tls_mxrxsize = 17408 # 16384 + 1024, governs max rx data, pm max xfer len, rx coalesce sizes
tp_stag = 2
tp_pbl = 5
tp_rq = 7
@ -273,7 +275,7 @@
[fini]
version = 0x1
checksum = 0x7191019f
checksum = 0x9e8952d2
#
# $FreeBSD$
#

3
sys/dev/cxgbe/offload.h
View File

@ -151,6 +151,9 @@ struct tom_tunables {
int sndbuf;
int ddp;
int rx_coalesce;
int tls;
int *tls_rx_ports;
int num_tls_rx_ports;
int tx_align;
int tx_zcopy;
};

85
sys/dev/cxgbe/t4_main.c
View File

@ -591,6 +591,7 @@ static int sysctl_wcwr_stats(SYSCTL_HANDLER_ARGS);
static int sysctl_tc_params(SYSCTL_HANDLER_ARGS);
#endif
#ifdef TCP_OFFLOAD
static int sysctl_tls_rx_ports(SYSCTL_HANDLER_ARGS);
static int sysctl_tp_tick(SYSCTL_HANDLER_ARGS);
static int sysctl_tp_dack_timer(SYSCTL_HANDLER_ARGS);
static int sysctl_tp_timer(SYSCTL_HANDLER_ARGS);
@ -1390,6 +1391,7 @@ t4_detach_common(device_t dev)
free(sc->sge.iqmap, M_CXGBE);
free(sc->sge.eqmap, M_CXGBE);
free(sc->tids.ftid_tab, M_CXGBE);
free(sc->tt.tls_rx_ports, M_CXGBE);
t4_destroy_dma_tag(sc);
if (mtx_initialized(&sc->sc_lock)) {
sx_xlock(&t4_list_lock);
@ -5433,6 +5435,14 @@ t4_sysctls(struct adapter *sc)
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "rx_coalesce",
CTLFLAG_RW, &sc->tt.rx_coalesce, 0, "receive coalescing");
sc->tt.tls = 0;
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tls", CTLFLAG_RW,
&sc->tt.tls, 0, "Inline TLS allowed");
SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "tls_rx_ports",
CTLTYPE_INT | CTLFLAG_RW, sc, 0, sysctl_tls_rx_ports,
"I", "TCP ports that use inline TLS+TOE RX");
sc->tt.tx_align = 1;
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_align",
CTLFLAG_RW, &sc->tt.tx_align, 0, "chop and align payload");
@ -5836,6 +5846,19 @@ cxgbe_sysctls(struct port_info *pi)
"# of buffer-group 3 truncated packets");
#undef SYSCTL_ADD_T4_PORTSTAT
SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "tx_tls_records",
CTLFLAG_RD, &pi->tx_tls_records,
"# of TLS records transmitted");
SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "tx_tls_octets",
CTLFLAG_RD, &pi->tx_tls_octets,
"# of payload octets in transmitted TLS records");
SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "rx_tls_records",
CTLFLAG_RD, &pi->rx_tls_records,
"# of TLS records received");
SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "rx_tls_octets",
CTLFLAG_RD, &pi->rx_tls_octets,
"# of payload octets in received TLS records");
}
static int
@ -8257,6 +8280,68 @@ sysctl_tc_params(SYSCTL_HANDLER_ARGS)
#endif
#ifdef TCP_OFFLOAD
static int
sysctl_tls_rx_ports(SYSCTL_HANDLER_ARGS)
{
struct adapter *sc = arg1;
int *old_ports, *new_ports;
int i, new_count, rc;
if (req->newptr == NULL && req->oldptr == NULL)
return (SYSCTL_OUT(req, NULL, imax(sc->tt.num_tls_rx_ports, 1) *
sizeof(sc->tt.tls_rx_ports[0])));
rc = begin_synchronized_op(sc, NULL, SLEEP_OK | INTR_OK, "t4tlsrx");
if (rc)
return (rc);
if (sc->tt.num_tls_rx_ports == 0) {
i = -1;
rc = SYSCTL_OUT(req, &i, sizeof(i));
} else
rc = SYSCTL_OUT(req, sc->tt.tls_rx_ports,
sc->tt.num_tls_rx_ports * sizeof(sc->tt.tls_rx_ports[0]));
if (rc == 0 && req->newptr != NULL) {
new_count = req->newlen / sizeof(new_ports[0]);
new_ports = malloc(new_count * sizeof(new_ports[0]), M_CXGBE,
M_WAITOK);
rc = SYSCTL_IN(req, new_ports, new_count *
sizeof(new_ports[0]));
if (rc)
goto err;
/* Allow setting to a single '-1' to clear the list. */
if (new_count == 1 && new_ports[0] == -1) {
ADAPTER_LOCK(sc);
old_ports = sc->tt.tls_rx_ports;
sc->tt.tls_rx_ports = NULL;
sc->tt.num_tls_rx_ports = 0;
ADAPTER_UNLOCK(sc);
free(old_ports, M_CXGBE);
} else {
for (i = 0; i < new_count; i++) {
if (new_ports[i] < 1 ||
new_ports[i] > IPPORT_MAX) {
rc = EINVAL;
goto err;
}
}
ADAPTER_LOCK(sc);
old_ports = sc->tt.tls_rx_ports;
sc->tt.tls_rx_ports = new_ports;
sc->tt.num_tls_rx_ports = new_count;
ADAPTER_UNLOCK(sc);
free(old_ports, M_CXGBE);
new_ports = NULL;
}
err:
free(new_ports, M_CXGBE);
}
end_synchronized_op(sc, 0);
return (rc);
}
static void
unit_conv(char *buf, size_t len, u_int val, u_int factor)
{

14
sys/dev/cxgbe/tom/t4_connect.c
View File

@ -142,6 +142,10 @@ do_act_establish(struct sge_iq *iq, const struct rss_header *rss,
}
make_established(toep, cpl->snd_isn, cpl->rcv_isn, cpl->tcp_opt);
if (toep->ulp_mode == ULP_MODE_TLS)
tls_establish(toep);
done:
INP_WUNLOCK(inp);
CURVNET_RESTORE();
@ -268,6 +272,11 @@ calc_opt2a(struct socket *so, struct toepcb *toep)
if (toep->ulp_mode == ULP_MODE_TCPDDP)
opt2 |= F_RX_FC_VALID | F_RX_FC_DDP;
#endif
if (toep->ulp_mode == ULP_MODE_TLS) {
opt2 |= F_RX_FC_VALID;
opt2 &= ~V_RX_COALESCE(M_RX_COALESCE);
opt2 |= F_RX_FC_DISABLE;
}
return (htobe32(opt2));
}
@ -378,10 +387,7 @@ t4_connect(struct toedev *tod, struct socket *so, struct rtentry *rt,
DONT_OFFLOAD_ACTIVE_OPEN(ENOMEM);
toep->vnet = so->so_vnet;
if (sc->tt.ddp && (so->so_options & SO_NO_DDP) == 0)
set_tcpddp_ulp_mode(toep);
else
toep->ulp_mode = ULP_MODE_NONE;
set_ulp_mode(toep, select_ulp_mode(so, sc));
SOCKBUF_LOCK(&so->so_rcv);
/* opt0 rcv_bufsiz initially, assumes its normal meaning later */
toep->rx_credits = min(select_rcv_wnd(so) >> 10, M_RCV_BUFSIZ);

122
sys/dev/cxgbe/tom/t4_cpl_io.c
View File

@ -73,9 +73,6 @@ __FBSDID("$FreeBSD$");
#include "tom/t4_tom_l2t.h"
#include "tom/t4_tom.h"
#define IS_AIOTX_MBUF(m) \
((m)->m_flags & M_EXT && (m)->m_ext.ext_flags & EXT_FLAG_AIOTX)
static void t4_aiotx_cancel(struct kaiocb *job);
static void t4_aiotx_queue_toep(struct toepcb *toep);
@ -106,7 +103,7 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_params *ftxp)
{
struct wrqe *wr;
struct fw_flowc_wr *flowc;
unsigned int nparams = ftxp ? 8 : 6, flowclen;
unsigned int nparams, flowclen, paramidx;
struct vi_info *vi = toep->vi;
struct port_info *pi = vi->pi;
struct adapter *sc = pi->adapter;
@ -116,6 +113,15 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_params *ftxp)
KASSERT(!(toep->flags & TPF_FLOWC_WR_SENT),
("%s: flowc for tid %u sent already", __func__, toep->tid));
if (ftxp != NULL)
nparams = 8;
else
nparams = 6;
if (toep->ulp_mode == ULP_MODE_TLS)
nparams++;
if (toep->tls.fcplenmax != 0)
nparams++;
flowclen = sizeof(*flowc) + nparams * sizeof(struct fw_flowc_mnemval);
wr = alloc_wrqe(roundup2(flowclen, 16), toep->ofld_txq);
@ -131,38 +137,44 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_params *ftxp)
flowc->flowid_len16 = htonl(V_FW_WR_LEN16(howmany(flowclen, 16)) |
V_FW_WR_FLOWID(toep->tid));
flowc->mnemval[0].mnemonic = FW_FLOWC_MNEM_PFNVFN;
flowc->mnemval[0].val = htobe32(pfvf);
flowc->mnemval[1].mnemonic = FW_FLOWC_MNEM_CH;
flowc->mnemval[1].val = htobe32(pi->tx_chan);
flowc->mnemval[2].mnemonic = FW_FLOWC_MNEM_PORT;
flowc->mnemval[2].val = htobe32(pi->tx_chan);
flowc->mnemval[3].mnemonic = FW_FLOWC_MNEM_IQID;
flowc->mnemval[3].val = htobe32(toep->ofld_rxq->iq.abs_id);
#define FLOWC_PARAM(__m, __v) \
do { \
flowc->mnemval[paramidx].mnemonic = FW_FLOWC_MNEM_##__m; \
flowc->mnemval[paramidx].val = htobe32(__v); \
paramidx++; \
} while (0)
paramidx = 0;
FLOWC_PARAM(PFNVFN, pfvf);
FLOWC_PARAM(CH, pi->tx_chan);
FLOWC_PARAM(PORT, pi->tx_chan);
FLOWC_PARAM(IQID, toep->ofld_rxq->iq.abs_id);
if (ftxp) {
uint32_t sndbuf = min(ftxp->snd_space, sc->tt.sndbuf);
flowc->mnemval[4].mnemonic = FW_FLOWC_MNEM_SNDNXT;
flowc->mnemval[4].val = htobe32(ftxp->snd_nxt);
flowc->mnemval[5].mnemonic = FW_FLOWC_MNEM_RCVNXT;
flowc->mnemval[5].val = htobe32(ftxp->rcv_nxt);
flowc->mnemval[6].mnemonic = FW_FLOWC_MNEM_SNDBUF;
flowc->mnemval[6].val = htobe32(sndbuf);
flowc->mnemval[7].mnemonic = FW_FLOWC_MNEM_MSS;
flowc->mnemval[7].val = htobe32(ftxp->mss);
FLOWC_PARAM(SNDNXT, ftxp->snd_nxt);
FLOWC_PARAM(RCVNXT, ftxp->rcv_nxt);
FLOWC_PARAM(SNDBUF, sndbuf);
FLOWC_PARAM(MSS, ftxp->mss);
CTR6(KTR_CXGBE,
"%s: tid %u, mss %u, sndbuf %u, snd_nxt 0x%x, rcv_nxt 0x%x",
__func__, toep->tid, ftxp->mss, sndbuf, ftxp->snd_nxt,
ftxp->rcv_nxt);
} else {
flowc->mnemval[4].mnemonic = FW_FLOWC_MNEM_SNDBUF;
flowc->mnemval[4].val = htobe32(512);
flowc->mnemval[5].mnemonic = FW_FLOWC_MNEM_MSS;
flowc->mnemval[5].val = htobe32(512);
FLOWC_PARAM(SNDBUF, 512);
FLOWC_PARAM(MSS, 512);
CTR2(KTR_CXGBE, "%s: tid %u", __func__, toep->tid);
}
if (toep->ulp_mode == ULP_MODE_TLS)
FLOWC_PARAM(ULP_MODE, toep->ulp_mode);
if (toep->tls.fcplenmax != 0)
FLOWC_PARAM(TXDATAPLEN_MAX, toep->tls.fcplenmax);
#undef FLOWC_PARAM
KASSERT(paramidx == nparams, ("nparams mismatch"));
txsd->tx_credits = howmany(flowclen, 16);
txsd->plen = 0;
@ -421,7 +433,7 @@ make_established(struct toepcb *toep, uint32_t snd_isn, uint32_t rcv_isn,
soisconnected(so);
}
static int
int
send_rx_credits(struct adapter *sc, struct toepcb *toep, int credits)
{
struct wrqe *wr;
@ -442,6 +454,23 @@ send_rx_credits(struct adapter *sc, struct toepcb *toep, int credits)
return (credits);
}
void
send_rx_modulate(struct adapter *sc, struct toepcb *toep)
{
struct wrqe *wr;
struct cpl_rx_data_ack *req;
wr = alloc_wrqe(sizeof(*req), toep->ctrlq);
if (wr == NULL)
return;
req = wrtod(wr);
INIT_TP_WR_MIT_CPL(req, CPL_RX_DATA_ACK, toep->tid);
req->credit_dack = htobe32(F_RX_MODULATE_RX);
t4_wrq_tx(sc, wr);
}
void
t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
{
@ -459,8 +488,18 @@ t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
("%s: sb %p has more data (%d) than last time (%d).",
__func__, sb, sbused(sb), toep->sb_cc));
toep->rx_credits += toep->sb_cc - sbused(sb);
credits = toep->sb_cc - sbused(sb);
toep->sb_cc = sbused(sb);
if (toep->ulp_mode == ULP_MODE_TLS) {
if (toep->tls.rcv_over >= credits) {
toep->tls.rcv_over -= credits;
credits = 0;
} else {
credits -= toep->tls.rcv_over;
toep->tls.rcv_over = 0;
}
}
toep->rx_credits += credits;
if (toep->rx_credits > 0 &&
(tp->rcv_wnd <= 32 * 1024 || toep->rx_credits >= 64 * 1024 ||
@ -471,7 +510,8 @@ t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
toep->rx_credits -= credits;
tp->rcv_wnd += credits;
tp->rcv_adv += credits;
}
} else if (toep->flags & TPF_FORCE_CREDITS)
send_rx_modulate(sc, toep);
}
void
@ -489,8 +529,8 @@ t4_rcvd(struct toedev *tod, struct tcpcb *tp)
/*
* Close a connection by sending a CPL_CLOSE_CON_REQ message.
*/
static int
close_conn(struct adapter *sc, struct toepcb *toep)
int
t4_close_conn(struct adapter *sc, struct toepcb *toep)
{
struct wrqe *wr;
struct cpl_close_con_req *req;
@ -691,6 +731,7 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep, int drop)
KASSERT(toep->ulp_mode == ULP_MODE_NONE ||
toep->ulp_mode == ULP_MODE_TCPDDP ||
toep->ulp_mode == ULP_MODE_TLS ||
toep->ulp_mode == ULP_MODE_RDMA,
("%s: ulp_mode %u for toep %p", __func__, toep->ulp_mode, toep));
@ -905,7 +946,7 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep, int drop)
/* Send a FIN if requested, but only if there's no more data to send */
if (m == NULL && toep->flags & TPF_SEND_FIN)
close_conn(sc, toep);
t4_close_conn(sc, toep);
}
static inline void
@ -1097,7 +1138,7 @@ t4_push_pdus(struct adapter *sc, struct toepcb *toep, int drop)
/* Send a FIN if requested, but only if there are no more PDUs to send */
if (mbufq_first(pduq) == NULL && toep->flags & TPF_SEND_FIN)
close_conn(sc, toep);
t4_close_conn(sc, toep);
}
int
@ -1116,6 +1157,8 @@ t4_tod_output(struct toedev *tod, struct tcpcb *tp)
if (toep->ulp_mode == ULP_MODE_ISCSI)
t4_push_pdus(sc, toep, 0);
else if (tls_tx_key(toep))
t4_push_tls_records(sc, toep, 0);
else
t4_push_frames(sc, toep, 0);
@ -1140,6 +1183,8 @@ t4_send_fin(struct toedev *tod, struct tcpcb *tp)
if (tp->t_state >= TCPS_ESTABLISHED) {
if (toep->ulp_mode == ULP_MODE_ISCSI)
t4_push_pdus(sc, toep, 0);
else if (tls_tx_key(toep))
t4_push_tls_records(sc, toep, 0);
else
t4_push_frames(sc, toep, 0);
}
@ -1772,6 +1817,10 @@ do_fw4_ack(struct sge_iq *iq, const struct rss_header *rss, struct mbuf *m)
credits -= txsd->tx_credits;
toep->tx_credits += txsd->tx_credits;
plen += txsd->plen;
if (txsd->iv_buffer) {
free(txsd->iv_buffer, M_CXGBE);
txsd->iv_buffer = NULL;
}
txsd++;
toep->txsd_avail++;
KASSERT(toep->txsd_avail <= toep->txsd_total,
@ -1797,6 +1846,8 @@ do_fw4_ack(struct sge_iq *iq, const struct rss_header *rss, struct mbuf *m)
CURVNET_SET(toep->vnet);
if (toep->ulp_mode == ULP_MODE_ISCSI)
t4_push_pdus(sc, toep, plen);
else if (tls_tx_key(toep))
t4_push_tls_records(sc, toep, plen);
else
t4_push_frames(sc, toep, plen);
CURVNET_RESTORE();
@ -1826,6 +1877,12 @@ do_fw4_ack(struct sge_iq *iq, const struct rss_header *rss, struct mbuf *m)
tid, plen);
#endif
sbdrop_locked(sb, plen);
if (tls_tx_key(toep)) {
struct tls_ofld_info *tls_ofld = &toep->tls;
MPASS(tls_ofld->sb_off >= plen);
tls_ofld->sb_off -= plen;
}
if (!TAILQ_EMPTY(&toep->aiotx_jobq))
t4_aiotx_queue_toep(toep);
sowwakeup_locked(so); /* unlocks so_snd */
@ -2300,6 +2357,9 @@ t4_aio_queue_aiotx(struct socket *so, struct kaiocb *job)
if (!sc->tt.tx_zcopy)
return (EOPNOTSUPP);
if (is_tls_offload(toep) || tls_tx_key(toep))
return (EOPNOTSUPP);
SOCKBUF_LOCK(&so->so_snd);
#ifdef VERBOSE_TRACES
CTR2(KTR_CXGBE, "%s: queueing %p", __func__, job);

27
sys/dev/cxgbe/tom/t4_listen.c
View File

@ -1056,6 +1056,11 @@ calc_opt2p(struct adapter *sc, struct port_info *pi, int rxqid,
if (ulp_mode == ULP_MODE_TCPDDP)
opt2 |= F_RX_FC_VALID | F_RX_FC_DDP;
#endif
if (ulp_mode == ULP_MODE_TLS) {
opt2 |= F_RX_FC_VALID;
opt2 &= ~V_RX_COALESCE(M_RX_COALESCE);
opt2 |= F_RX_FC_DISABLE;
}
return htobe32(opt2);
}
@ -1347,11 +1352,15 @@ do_pass_accept_req(struct sge_iq *iq, const struct rss_header *rss,
INIT_TP_WR_MIT_CPL(rpl5, CPL_PASS_ACCEPT_RPL, tid);
}
if (sc->tt.ddp && (so->so_options & SO_NO_DDP) == 0) {
ulp_mode = ULP_MODE_TCPDDP;
ulp_mode = select_ulp_mode(so, sc);
switch (ulp_mode) {
case ULP_MODE_TCPDDP:
synqe->flags |= TPF_SYNQE_TCPDDP;
} else
ulp_mode = ULP_MODE_NONE;
break;
case ULP_MODE_TLS:
synqe->flags |= TPF_SYNQE_TLS;
break;
}
rpl->opt0 = calc_opt0(so, vi, e, mtu_idx, rscale, rx_credits, ulp_mode);
rpl->opt2 = calc_opt2p(sc, pi, rxqid, &cpl->tcpopt, &th, ulp_mode);
@ -1407,8 +1416,8 @@ do_pass_accept_req(struct sge_iq *iq, const struct rss_header *rss,
REJECT_PASS_ACCEPT();
}
CTR5(KTR_CXGBE, "%s: stid %u, tid %u, lctx %p, synqe %p, SYNACK",
__func__, stid, tid, lctx, synqe);
CTR6(KTR_CXGBE, "%s: stid %u, tid %u, lctx %p, synqe %p, SYNACK mode %d",
__func__, stid, tid, lctx, synqe, ulp_mode);
INP_WLOCK(inp);
synqe->flags |= TPF_SYNQE_HAS_L2TE;
@ -1557,9 +1566,11 @@ do_pass_establish(struct sge_iq *iq, const struct rss_header *rss,
toep->tid = tid;
toep->l2te = &sc->l2t->l2tab[synqe->l2e_idx];
if (synqe->flags & TPF_SYNQE_TCPDDP)
set_tcpddp_ulp_mode(toep);
set_ulp_mode(toep, ULP_MODE_TCPDDP);
else if (synqe->flags & TPF_SYNQE_TLS)
set_ulp_mode(toep, ULP_MODE_TLS);
else
toep->ulp_mode = ULP_MODE_NONE;
set_ulp_mode(toep, ULP_MODE_NONE);
/* opt0 rcv_bufsiz initially, assumes its normal meaning later */
toep->rx_credits = synqe->rcv_bufsize;

1642
sys/dev/cxgbe/tom/t4_tls.c Normal file
View File

File diff suppressed because it is too large Load Diff

593
sys/dev/cxgbe/tom/t4_tls.h Normal file
View File

@ -0,0 +1,593 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
*
* Copyright (c) 2017-2018 Chelsio Communications, Inc.
* All rights reserved.
* Written by: John Baldwin <jhb@FreeBSD.org>, Atul Gupta
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* $FreeBSD$
*
*/
#ifndef __T4_TLS_H__
#define __T4_TLS_H__
#define TLS1_VERSION 0x0301
#define TLS1_1_VERSION 0x0302
#define TLS1_2_VERSION 0x0303
#define TLS_MAX_VERSION TLS1_2_VERSION
#define DTLS1_VERSION 0xFEFF
#define DTLS1_2_VERSION 0xFEFD
#define DTLS_MAX_VERSION DTLS1_2_VERSION
#define DTLS1_VERSION_MAJOR 0xFE
/* Custom socket options for TLS+TOE. */
#define MAX_MAC_KSZ 64 /*512 bits */
#define MAX_CIPHER_KSZ 32 /* 256 bits */
#define CIPHER_BLOCK_SZ 16
#define SALT_SIZE 4
/* Can accomodate 16, 11-15 are reserved */
enum {
CHSSL_SHA_NOP,
CHSSL_SHA1,
CHSSL_SHA224,
CHSSL_SHA256,
CHSSL_GHASH,
CHSSL_SHA512_224,
CHSSL_SHA512_256,
CHSSL_SHA512_384,
CHSSL_SHA512_512,
CHSSL_CBCMAC,
CHSSL_CMAC,
};
/* Can accomodate 16, 8-15 are reserved */
enum {
CHSSL_CIPH_NOP,
CHSSL_AES_CBC,
CHSSL_AES_GCM,
CHSSL_AES_CTR,
CHSSL_AES_GEN,
CHSSL_IPSEC_ESP,
CHSSL_AES_XTS,
CHSSL_AES_CCM,
};
/* Key Context Programming Operation type */
#define KEY_WRITE_RX 0x1
#define KEY_WRITE_TX 0x2
#define KEY_DELETE_RX 0x4
#define KEY_DELETE_TX 0x8
#define S_KEY_CLR_LOC 4
#define M_KEY_CLR_LOC 0xf
#define V_KEY_CLR_LOC(x) ((x) << S_KEY_CLR_LOC)
#define G_KEY_CLR_LOC(x) (((x) >> S_KEY_CLR_LOC) & M_KEY_CLR_LOC)
#define F_KEY_CLR_LOC V_KEY_CLR_LOC(1U)
#define S_KEY_GET_LOC 0
#define M_KEY_GET_LOC 0xf
#define V_KEY_GET_LOC(x) ((x) << S_KEY_GET_LOC)
#define G_KEY_GET_LOC(x) (((x) >> S_KEY_GET_LOC) & M_KEY_GET_LOC)
struct tls_ofld_state {
unsigned char enc_mode;
unsigned char mac_mode;
unsigned char key_loc;
unsigned char ofld_mode;
unsigned char auth_mode;
unsigned char resv[3];
};
struct tls_tx_ctxt {
unsigned char salt[SALT_SIZE];
unsigned char key[MAX_CIPHER_KSZ];
unsigned char ipad[MAX_MAC_KSZ];
unsigned char opad[MAX_MAC_KSZ];
};
struct tls_rx_ctxt {
unsigned char salt[SALT_SIZE];
unsigned char key[MAX_CIPHER_KSZ];
unsigned char ipad[MAX_MAC_KSZ];
unsigned char opad[MAX_MAC_KSZ];
};
struct tls_key_context {
struct tls_tx_ctxt tx;
struct tls_rx_ctxt rx;
unsigned char l_p_key;
unsigned char hmac_ctrl;
unsigned char mac_first;
unsigned char iv_size;
unsigned char iv_ctrl;
unsigned char iv_algo;
unsigned char tx_seq_no;
unsigned char rx_seq_no;
struct tls_ofld_state state;
unsigned int tx_key_info_size;
unsigned int rx_key_info_size;
unsigned int frag_size;
unsigned int mac_secret_size;
unsigned int cipher_secret_size;
int proto_ver;
unsigned int sock_fd;
unsigned short dtls_epoch;
unsigned short rsv;
};
/* Set with 'struct tls_key_context'. */
#define TCP_TLSOM_SET_TLS_CONTEXT (TCP_VENDOR)
/* Get returns int of enabled (1) / disabled (0). */
#define TCP_TLSOM_GET_TLS_TOM (TCP_VENDOR + 1)
enum {
TLS_TOM_NONE = 0,
TLS_TOM_TXONLY,
TLS_TOM_BOTH
};
/* Set with no value. */
#define TCP_TLSOM_CLR_TLS_TOM (TCP_VENDOR + 2)
/* Set with no value. */
#define TCP_TLSOM_CLR_QUIES (TCP_VENDOR + 3)
#ifdef _KERNEL
/* Timeouts for handshake timer in seconds. */
#define TLS_SRV_HELLO_DONE 9
#define TLS_SRV_HELLO_RD_TM 5
#define TLS_SRV_HELLO_BKOFF_TM 15
#define CONTENT_TYPE_CCS 20
#define CONTENT_TYPE_ALERT 21
#define CONTENT_TYPE_HANDSHAKE 22
#define CONTENT_TYPE_APP_DATA 23
#define CONTENT_TYPE_HEARTBEAT 24
#define CONTENT_TYPE_KEY_CONTEXT 32
#define CONTENT_TYPE_ERROR 127
#define GCM_TAG_SIZE 16
#define AEAD_EXPLICIT_DATA_SIZE 8
#define TLS_HEADER_LENGTH 5
#define TP_TX_PG_SZ 65536
#define FC_TP_PLEN_MAX 17408
#define IPAD_SIZE 64
#define OPAD_SIZE 64
#define KEY_SIZE 32
#define CIPHER_BLOCK_SIZE 16
#define HDR_KCTX_SIZE (IPAD_SIZE + OPAD_SIZE + KEY_SIZE)
#define KEY_IN_DDR_SIZE 16
#define TLS_KEY_CONTEXT_SZ roundup2(sizeof(struct tls_tx_ctxt), 32)
/* MAC KEY SIZE */
#define SHA_NOP 0
#define SHA_GHASH 16
#define SHA_224 28
#define SHA_256 32
#define SHA_384 48
#define SHA_512 64
#define SHA1 20
/* CIPHER KEY SIZE */
#define AES_NOP 0
#define AES_128 16
#define AES_192 24
#define AES_256 32
enum {
TLS_1_2_VERSION,
TLS_1_1_VERSION,
DTLS_1_2_VERSION,
TLS_VERSION_MAX,
};
enum {
CH_EVP_CIPH_STREAM_CIPHER,
CH_EVP_CIPH_CBC_MODE,
CH_EVP_CIPH_GCM_MODE,
CH_EVP_CIPH_CTR_MODE,
};
enum {
TLS_SFO_WR_CONTEXTLOC_DSGL,
TLS_SFO_WR_CONTEXTLOC_IMMEDIATE,
TLS_SFO_WR_CONTEXTLOC_DDR,
};
enum {
CPL_TX_TLS_SFO_TYPE_CCS,
CPL_TX_TLS_SFO_TYPE_ALERT,
CPL_TX_TLS_SFO_TYPE_HANDSHAKE,
CPL_TX_TLS_SFO_TYPE_DATA,
CPL_TX_TLS_SFO_TYPE_HEARTBEAT, /* XXX: Shouldn't this be "CUSTOM"? */
};
enum {
CH_CK_SIZE_128,
CH_CK_SIZE_192,
CH_CK_SIZE_256,
CH_CK_SIZE_NOP,
};
enum {
CH_MK_SIZE_128,
CH_MK_SIZE_160,
CH_MK_SIZE_192,
CH_MK_SIZE_256,
CH_MK_SIZE_512,
CH_MK_SIZE_NOP,
};
#define SCMD_ENCDECCTRL_ENCRYPT 0
#define SCMD_ENCDECCTRL_DECRYPT 1
#define SCMD_CIPH_MODE_NOP 0
#define SCMD_CIPH_MODE_AES_CBC 1
#define SCMD_CIPH_MODE_AES_GCM 2
#define SCMD_CIPH_MODE_AES_CTR 3
#define SCMD_CIPH_MODE_AES_GEN 4
#define SCMD_CIPH_MODE_AES_CCM 7
struct tls_scmd {
__be32 seqno_numivs;
__be32 ivgen_hdrlen;
};
struct tls_ofld_info {
struct tls_key_context k_ctx;
int key_location;
int mac_length;
int rx_key_addr;
int tx_key_addr;
uint64_t tx_seq_no;
unsigned short fcplenmax;
unsigned short adjusted_plen;
unsigned short expn_per_ulp;
unsigned short pdus_per_ulp;
struct tls_scmd scmd0;
u_int sb_off;
struct callout handshake_timer;
u_int rcv_over;
};
struct tls_key_req {
__be32 wr_hi;
__be32 wr_mid;
__be32 ftid;
__u8 reneg_to_write_rx;
__u8 protocol;
__be16 mfs;
/* master command */
__be32 cmd;
__be32 len16; /* command length */
__be32 dlen; /* data length in 32-byte units */
__be32 kaddr;
/* sub-command */
__be32 sc_more;
__be32 sc_len;
}__packed;
struct tls_keyctx {
union key_ctx {
struct tx_keyctx_hdr {
__u8 ctxlen;
__u8 r2;
__be16 dualck_to_txvalid;
__u8 txsalt[4];
__be64 r5;
} txhdr;
struct rx_keyctx_hdr {
__u8 flitcnt_hmacctrl;
__u8 protover_ciphmode;
__u8 authmode_to_rxvalid;
__u8 ivpresent_to_rxmk_size;
__u8 rxsalt[4];
__be64 ivinsert_to_authinsrt;
} rxhdr;
} u;
struct keys {
__u8 edkey[32];
__u8 ipad[64];
__u8 opad[64];
} keys;
};
#define S_TLS_KEYCTX_TX_WR_DUALCK 12
#define M_TLS_KEYCTX_TX_WR_DUALCK 0x1
#define V_TLS_KEYCTX_TX_WR_DUALCK(x) ((x) << S_TLS_KEYCTX_TX_WR_DUALCK)
#define G_TLS_KEYCTX_TX_WR_DUALCK(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_DUALCK) & M_TLS_KEYCTX_TX_WR_DUALCK)
#define F_TLS_KEYCTX_TX_WR_DUALCK V_TLS_KEYCTX_TX_WR_DUALCK(1U)
#define S_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT 11
#define M_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT 0x1
#define V_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(x) \
((x) << S_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT)
#define G_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT) & \
M_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT)
#define F_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT \
V_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(1U)
#define S_TLS_KEYCTX_TX_WR_SALT_PRESENT 10
#define M_TLS_KEYCTX_TX_WR_SALT_PRESENT 0x1
#define V_TLS_KEYCTX_TX_WR_SALT_PRESENT(x) \
((x) << S_TLS_KEYCTX_TX_WR_SALT_PRESENT)
#define G_TLS_KEYCTX_TX_WR_SALT_PRESENT(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_SALT_PRESENT) & \
M_TLS_KEYCTX_TX_WR_SALT_PRESENT)
#define F_TLS_KEYCTX_TX_WR_SALT_PRESENT \
V_TLS_KEYCTX_TX_WR_SALT_PRESENT(1U)
#define S_TLS_KEYCTX_TX_WR_TXCK_SIZE 6
#define M_TLS_KEYCTX_TX_WR_TXCK_SIZE 0xf
#define V_TLS_KEYCTX_TX_WR_TXCK_SIZE(x) \
((x) << S_TLS_KEYCTX_TX_WR_TXCK_SIZE)
#define G_TLS_KEYCTX_TX_WR_TXCK_SIZE(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_TXCK_SIZE) & \
M_TLS_KEYCTX_TX_WR_TXCK_SIZE)
#define S_TLS_KEYCTX_TX_WR_TXMK_SIZE 2
#define M_TLS_KEYCTX_TX_WR_TXMK_SIZE 0xf
#define V_TLS_KEYCTX_TX_WR_TXMK_SIZE(x) \
((x) << S_TLS_KEYCTX_TX_WR_TXMK_SIZE)
#define G_TLS_KEYCTX_TX_WR_TXMK_SIZE(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_TXMK_SIZE) & \
M_TLS_KEYCTX_TX_WR_TXMK_SIZE)
#define S_TLS_KEYCTX_TX_WR_TXVALID 0
#define M_TLS_KEYCTX_TX_WR_TXVALID 0x1
#define V_TLS_KEYCTX_TX_WR_TXVALID(x) \
((x) << S_TLS_KEYCTX_TX_WR_TXVALID)
#define G_TLS_KEYCTX_TX_WR_TXVALID(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_TXVALID) & M_TLS_KEYCTX_TX_WR_TXVALID)
#define F_TLS_KEYCTX_TX_WR_TXVALID V_TLS_KEYCTX_TX_WR_TXVALID(1U)
#define S_TLS_KEYCTX_TX_WR_FLITCNT 3
#define M_TLS_KEYCTX_TX_WR_FLITCNT 0x1f
#define V_TLS_KEYCTX_TX_WR_FLITCNT(x) \
((x) << S_TLS_KEYCTX_TX_WR_FLITCNT)
#define G_TLS_KEYCTX_TX_WR_FLITCNT(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_FLITCNT) & M_TLS_KEYCTX_TX_WR_FLITCNT)
#define S_TLS_KEYCTX_TX_WR_HMACCTRL 0
#define M_TLS_KEYCTX_TX_WR_HMACCTRL 0x7
#define V_TLS_KEYCTX_TX_WR_HMACCTRL(x) \
((x) << S_TLS_KEYCTX_TX_WR_HMACCTRL)
#define G_TLS_KEYCTX_TX_WR_HMACCTRL(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_HMACCTRL) & M_TLS_KEYCTX_TX_WR_HMACCTRL)
#define S_TLS_KEYCTX_TX_WR_PROTOVER 4
#define M_TLS_KEYCTX_TX_WR_PROTOVER 0xf
#define V_TLS_KEYCTX_TX_WR_PROTOVER(x) \
((x) << S_TLS_KEYCTX_TX_WR_PROTOVER)
#define G_TLS_KEYCTX_TX_WR_PROTOVER(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_PROTOVER) & M_TLS_KEYCTX_TX_WR_PROTOVER)
#define S_TLS_KEYCTX_TX_WR_CIPHMODE 0
#define M_TLS_KEYCTX_TX_WR_CIPHMODE 0xf
#define V_TLS_KEYCTX_TX_WR_CIPHMODE(x) \
((x) << S_TLS_KEYCTX_TX_WR_CIPHMODE)
#define G_TLS_KEYCTX_TX_WR_CIPHMODE(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_CIPHMODE) & M_TLS_KEYCTX_TX_WR_CIPHMODE)
#define S_TLS_KEYCTX_TX_WR_AUTHMODE 4
#define M_TLS_KEYCTX_TX_WR_AUTHMODE 0xf
#define V_TLS_KEYCTX_TX_WR_AUTHMODE(x) \
((x) << S_TLS_KEYCTX_TX_WR_AUTHMODE)
#define G_TLS_KEYCTX_TX_WR_AUTHMODE(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_AUTHMODE) & M_TLS_KEYCTX_TX_WR_AUTHMODE)
#define S_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL 3
#define M_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL 0x1
#define V_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL(x) \
((x) << S_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL)
#define G_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL) & \
M_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL)
#define F_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL \
V_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL(1U)
#define S_TLS_KEYCTX_TX_WR_SEQNUMCTRL 1
#define M_TLS_KEYCTX_TX_WR_SEQNUMCTRL 0x3
#define V_TLS_KEYCTX_TX_WR_SEQNUMCTRL(x) \
((x) << S_TLS_KEYCTX_TX_WR_SEQNUMCTRL)
#define G_TLS_KEYCTX_TX_WR_SEQNUMCTRL(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_SEQNUMCTRL) & \
M_TLS_KEYCTX_TX_WR_SEQNUMCTRL)
#define S_TLS_KEYCTX_TX_WR_RXVALID 0
#define M_TLS_KEYCTX_TX_WR_RXVALID 0x1
#define V_TLS_KEYCTX_TX_WR_RXVALID(x) \
((x) << S_TLS_KEYCTX_TX_WR_RXVALID)
#define G_TLS_KEYCTX_TX_WR_RXVALID(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_RXVALID) & M_TLS_KEYCTX_TX_WR_RXVALID)
#define F_TLS_KEYCTX_TX_WR_RXVALID V_TLS_KEYCTX_TX_WR_RXVALID(1U)
#define S_TLS_KEYCTX_TX_WR_IVPRESENT 7
#define M_TLS_KEYCTX_TX_WR_IVPRESENT 0x1
#define V_TLS_KEYCTX_TX_WR_IVPRESENT(x) \
((x) << S_TLS_KEYCTX_TX_WR_IVPRESENT)
#define G_TLS_KEYCTX_TX_WR_IVPRESENT(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_IVPRESENT) & \
M_TLS_KEYCTX_TX_WR_IVPRESENT)
#define F_TLS_KEYCTX_TX_WR_IVPRESENT V_TLS_KEYCTX_TX_WR_IVPRESENT(1U)
#define S_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT 6
#define M_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT 0x1
#define V_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT(x) \
((x) << S_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT)
#define G_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT) & \
M_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT)
#define F_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT \
V_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT(1U)
#define S_TLS_KEYCTX_TX_WR_RXCK_SIZE 3
#define M_TLS_KEYCTX_TX_WR_RXCK_SIZE 0x7
#define V_TLS_KEYCTX_TX_WR_RXCK_SIZE(x) \
((x) << S_TLS_KEYCTX_TX_WR_RXCK_SIZE)
#define G_TLS_KEYCTX_TX_WR_RXCK_SIZE(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_RXCK_SIZE) & \
M_TLS_KEYCTX_TX_WR_RXCK_SIZE)
#define S_TLS_KEYCTX_TX_WR_RXMK_SIZE 0
#define M_TLS_KEYCTX_TX_WR_RXMK_SIZE 0x7
#define V_TLS_KEYCTX_TX_WR_RXMK_SIZE(x) \
((x) << S_TLS_KEYCTX_TX_WR_RXMK_SIZE)
#define G_TLS_KEYCTX_TX_WR_RXMK_SIZE(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_RXMK_SIZE) & \
M_TLS_KEYCTX_TX_WR_RXMK_SIZE)
#define S_TLS_KEYCTX_TX_WR_IVINSERT 55
#define M_TLS_KEYCTX_TX_WR_IVINSERT 0x1ffULL
#define V_TLS_KEYCTX_TX_WR_IVINSERT(x) \
((x) << S_TLS_KEYCTX_TX_WR_IVINSERT)
#define G_TLS_KEYCTX_TX_WR_IVINSERT(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_IVINSERT) & M_TLS_KEYCTX_TX_WR_IVINSERT)
#define S_TLS_KEYCTX_TX_WR_AADSTRTOFST 47
#define M_TLS_KEYCTX_TX_WR_AADSTRTOFST 0xffULL
#define V_TLS_KEYCTX_TX_WR_AADSTRTOFST(x) \
((x) << S_TLS_KEYCTX_TX_WR_AADSTRTOFST)
#define G_TLS_KEYCTX_TX_WR_AADSTRTOFST(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_AADSTRTOFST) & \
M_TLS_KEYCTX_TX_WR_AADSTRTOFST)
#define S_TLS_KEYCTX_TX_WR_AADSTOPOFST 39
#define M_TLS_KEYCTX_TX_WR_AADSTOPOFST 0xffULL
#define V_TLS_KEYCTX_TX_WR_AADSTOPOFST(x) \
((x) << S_TLS_KEYCTX_TX_WR_AADSTOPOFST)
#define G_TLS_KEYCTX_TX_WR_AADSTOPOFST(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_AADSTOPOFST) & \
M_TLS_KEYCTX_TX_WR_AADSTOPOFST)
#define S_TLS_KEYCTX_TX_WR_CIPHERSRTOFST 30
#define M_TLS_KEYCTX_TX_WR_CIPHERSRTOFST 0x1ffULL
#define V_TLS_KEYCTX_TX_WR_CIPHERSRTOFST(x) \
((x) << S_TLS_KEYCTX_TX_WR_CIPHERSRTOFST)
#define G_TLS_KEYCTX_TX_WR_CIPHERSRTOFST(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_CIPHERSRTOFST) & \
M_TLS_KEYCTX_TX_WR_CIPHERSRTOFST)
#define S_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST 23
#define M_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST 0x7f
#define V_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST(x) \
((x) << S_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST)
#define G_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST) & \
M_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST)
#define S_TLS_KEYCTX_TX_WR_AUTHSRTOFST 14
#define M_TLS_KEYCTX_TX_WR_AUTHSRTOFST 0x1ff
#define V_TLS_KEYCTX_TX_WR_AUTHSRTOFST(x) \
((x) << S_TLS_KEYCTX_TX_WR_AUTHSRTOFST)
#define G_TLS_KEYCTX_TX_WR_AUTHSRTOFST(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_AUTHSRTOFST) & \
M_TLS_KEYCTX_TX_WR_AUTHSRTOFST)
#define S_TLS_KEYCTX_TX_WR_AUTHSTOPOFST 7
#define M_TLS_KEYCTX_TX_WR_AUTHSTOPOFST 0x7f
#define V_TLS_KEYCTX_TX_WR_AUTHSTOPOFST(x) \
((x) << S_TLS_KEYCTX_TX_WR_AUTHSTOPOFST)
#define G_TLS_KEYCTX_TX_WR_AUTHSTOPOFST(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_AUTHSTOPOFST) & \
M_TLS_KEYCTX_TX_WR_AUTHSTOPOFST)
#define S_TLS_KEYCTX_TX_WR_AUTHINSRT 0
#define M_TLS_KEYCTX_TX_WR_AUTHINSRT 0x7f
#define V_TLS_KEYCTX_TX_WR_AUTHINSRT(x) \
((x) << S_TLS_KEYCTX_TX_WR_AUTHINSRT)
#define G_TLS_KEYCTX_TX_WR_AUTHINSRT(x) \
(((x) >> S_TLS_KEYCTX_TX_WR_AUTHINSRT) & \
M_TLS_KEYCTX_TX_WR_AUTHINSRT)
struct tls_hdr {
__u8 type;
__be16 version;
__be16 length;
} __packed;
struct tlsrx_hdr_pkt {
__u8 type;
__be16 version;
__be16 length;
__be64 tls_seq;
__be16 reserved1;
__u8 res_to_mac_error;
} __packed;
/* res_to_mac_error fields */
#define S_TLSRX_HDR_PKT_INTERNAL_ERROR 4
#define M_TLSRX_HDR_PKT_INTERNAL_ERROR 0x1
#define V_TLSRX_HDR_PKT_INTERNAL_ERROR(x) \
((x) << S_TLSRX_HDR_PKT_INTERNAL_ERROR)
#define G_TLSRX_HDR_PKT_INTERNAL_ERROR(x) \
(((x) >> S_TLSRX_HDR_PKT_INTERNAL_ERROR) & M_TLSRX_HDR_PKT_INTERNAL_ERROR)
#define F_TLSRX_HDR_PKT_INTERNAL_ERROR V_TLSRX_HDR_PKT_INTERNAL_ERROR(1U)
#define S_TLSRX_HDR_PKT_SPP_ERROR 3
#define M_TLSRX_HDR_PKT_SPP_ERROR 0x1
#define V_TLSRX_HDR_PKT_SPP_ERROR(x) ((x) << S_TLSRX_HDR_PKT_SPP_ERROR)
#define G_TLSRX_HDR_PKT_SPP_ERROR(x) \
(((x) >> S_TLSRX_HDR_PKT_SPP_ERROR) & M_TLSRX_HDR_PKT_SPP_ERROR)
#define F_TLSRX_HDR_PKT_SPP_ERROR V_TLSRX_HDR_PKT_SPP_ERROR(1U)
#define S_TLSRX_HDR_PKT_CCDX_ERROR 2
#define M_TLSRX_HDR_PKT_CCDX_ERROR 0x1
#define V_TLSRX_HDR_PKT_CCDX_ERROR(x) ((x) << S_TLSRX_HDR_PKT_CCDX_ERROR)
#define G_TLSRX_HDR_PKT_CCDX_ERROR(x) \
(((x) >> S_TLSRX_HDR_PKT_CCDX_ERROR) & M_TLSRX_HDR_PKT_CCDX_ERROR)
#define F_TLSRX_HDR_PKT_CCDX_ERROR V_TLSRX_HDR_PKT_CCDX_ERROR(1U)
#define S_TLSRX_HDR_PKT_PAD_ERROR 1
#define M_TLSRX_HDR_PKT_PAD_ERROR 0x1
#define V_TLSRX_HDR_PKT_PAD_ERROR(x) ((x) << S_TLSRX_HDR_PKT_PAD_ERROR)
#define G_TLSRX_HDR_PKT_PAD_ERROR(x) \
(((x) >> S_TLSRX_HDR_PKT_PAD_ERROR) & M_TLSRX_HDR_PKT_PAD_ERROR)
#define F_TLSRX_HDR_PKT_PAD_ERROR V_TLSRX_HDR_PKT_PAD_ERROR(1U)
#define S_TLSRX_HDR_PKT_MAC_ERROR 0
#define M_TLSRX_HDR_PKT_MAC_ERROR 0x1
#define V_TLSRX_HDR_PKT_MAC_ERROR(x) ((x) << S_TLSRX_HDR_PKT_MAC_ERROR)
#define G_TLSRX_HDR_PKT_MAC_ERROR(x) \
(((x) >> S_TLSRX_HDR_PKT_MAC_ERROR) & M_TLSRX_HDR_PKT_MAC_ERROR)
#define F_TLSRX_HDR_PKT_MAC_ERROR V_TLSRX_HDR_PKT_MAC_ERROR(1U)
#define M_TLSRX_HDR_PKT_ERROR 0x1F
#endif /* _KERNEL */
#endif /* !__T4_TLS_H__ */

75
sys/dev/cxgbe/tom/t4_tom.c
View File

@ -71,6 +71,7 @@ __FBSDID("$FreeBSD$");
#include "common/t4_tcb.h"
#include "tom/t4_tom_l2t.h"
#include "tom/t4_tom.h"
#include "tom/t4_tls.h"
static struct protosw toe_protosw;
static struct pr_usrreqs toe_usrreqs;
@ -199,6 +200,7 @@ free_toepcb(struct toepcb *toep)
if (toep->ulp_mode == ULP_MODE_TCPDDP)
ddp_uninit_toep(toep);
tls_uninit_toep(toep);
free(toep, M_CXGBE);
}
@ -619,12 +621,48 @@ select_ntuple(struct vi_info *vi, struct l2t_entry *e)
return (htobe64(V_FILTER_TUPLE(ntuple)));
}
void
set_tcpddp_ulp_mode(struct toepcb *toep)
static int
is_tls_sock(struct socket *so, struct adapter *sc)
{
struct inpcb *inp = sotoinpcb(so);
int i, rc;
/* XXX: Eventually add a SO_WANT_TLS socket option perhaps? */
rc = 0;
ADAPTER_LOCK(sc);
for (i = 0; i < sc->tt.num_tls_rx_ports; i++) {
if (inp->inp_lport == htons(sc->tt.tls_rx_ports[i]) ||
inp->inp_fport == htons(sc->tt.tls_rx_ports[i])) {
rc = 1;
break;
}
}
ADAPTER_UNLOCK(sc);
return (rc);
}
int
select_ulp_mode(struct socket *so, struct adapter *sc)
{
toep->ulp_mode = ULP_MODE_TCPDDP;
ddp_init_toep(toep);
if (can_tls_offload(sc) && is_tls_sock(so, sc))
return (ULP_MODE_TLS);
else if (sc->tt.ddp && (so->so_options & SO_NO_DDP) == 0)
return (ULP_MODE_TCPDDP);
else
return (ULP_MODE_NONE);
}
void
set_ulp_mode(struct toepcb *toep, int ulp_mode)
{
CTR4(KTR_CXGBE, "%s: toep %p (tid %d) ulp_mode %d",
__func__, toep, toep->tid, ulp_mode);
toep->ulp_mode = ulp_mode;
tls_init_toep(toep);
if (toep->ulp_mode == ULP_MODE_TCPDDP)
ddp_init_toep(toep);
}
int
@ -959,6 +997,7 @@ free_tom_data(struct adapter *sc, struct tom_data *td)
KASSERT(td->lctx_count == 0,
("%s: lctx hash table is not empty.", __func__));
tls_free_kmap(td);
t4_free_ppod_region(&td->pr);
destroy_clip_table(sc, td);
@ -1063,6 +1102,12 @@ t4_tom_activate(struct adapter *sc)
/* CLIP table for IPv6 offload */
init_clip_table(sc, td);
if (sc->vres.key.size != 0) {
rc = tls_init_kmap(sc, td);
if (rc != 0)
goto done;
}
/* toedev ops */
tod = &td->tod;
init_toedev(tod);
@ -1167,6 +1212,24 @@ t4_aio_queue_tom(struct socket *so, struct kaiocb *job)
return (t4_aio_queue_aiotx(so, job));
}
static int
t4_ctloutput_tom(struct socket *so, struct sockopt *sopt)
{
if (sopt->sopt_level != IPPROTO_TCP)
return (tcp_ctloutput(so, sopt));
switch (sopt->sopt_name) {
case TCP_TLSOM_SET_TLS_CONTEXT:
case TCP_TLSOM_GET_TLS_TOM:
case TCP_TLSOM_CLR_TLS_TOM:
case TCP_TLSOM_CLR_QUIES:
return (t4_ctloutput_tls(so, sopt));
default:
return (tcp_ctloutput(so, sopt));
}
}
static int
t4_tom_mod_load(void)
{
@ -1178,6 +1241,7 @@ t4_tom_mod_load(void)
t4_init_cpl_io_handlers();
t4_ddp_mod_load();
t4_tls_mod_load();
tcp_protosw = pffindproto(PF_INET, IPPROTO_TCP, SOCK_STREAM);
if (tcp_protosw == NULL)
@ -1185,6 +1249,7 @@ t4_tom_mod_load(void)
bcopy(tcp_protosw, &toe_protosw, sizeof(toe_protosw));
bcopy(tcp_protosw->pr_usrreqs, &toe_usrreqs, sizeof(toe_usrreqs));
toe_usrreqs.pru_aio_queue = t4_aio_queue_tom;
toe_protosw.pr_ctloutput = t4_ctloutput_tom;
toe_protosw.pr_usrreqs = &toe_usrreqs;
tcp6_protosw = pffindproto(PF_INET6, IPPROTO_TCP, SOCK_STREAM);
@ -1193,6 +1258,7 @@ t4_tom_mod_load(void)
bcopy(tcp6_protosw, &toe6_protosw, sizeof(toe6_protosw));
bcopy(tcp6_protosw->pr_usrreqs, &toe6_usrreqs, sizeof(toe6_usrreqs));
toe6_usrreqs.pru_aio_queue = t4_aio_queue_tom;
toe6_protosw.pr_ctloutput = t4_ctloutput_tom;
toe6_protosw.pr_usrreqs = &toe6_usrreqs;
TIMEOUT_TASK_INIT(taskqueue_thread, &clip_task, 0, t4_clip_task, NULL);
@ -1228,6 +1294,7 @@ t4_tom_mod_unload(void)
taskqueue_cancel_timeout(taskqueue_thread, &clip_task, NULL);
}
t4_tls_mod_unload();
t4_ddp_mod_unload();
t4_uninit_connect_cpl_handlers();

44
sys/dev/cxgbe/tom/t4_tom.h
View File

@ -33,6 +33,7 @@
#ifndef __T4_TOM_H__
#define __T4_TOM_H__
#include <sys/vmem.h>
#include "tom/t4_tls.h"
#define LISTEN_HASH_SIZE 32
@ -71,6 +72,8 @@ enum {
TPF_SYNQE_TCPDDP = (1 << 10), /* ulp_mode TCPDDP in toepcb */
TPF_SYNQE_EXPANDED = (1 << 11), /* toepcb ready, tid context updated */
TPF_SYNQE_HAS_L2TE = (1 << 12), /* we've replied to PASS_ACCEPT_REQ */
TPF_SYNQE_TLS = (1 << 13), /* ulp_mode TLS in toepcb */
TPF_FORCE_CREDITS = (1 << 14), /* always send credits */
};
enum {
@ -83,9 +86,12 @@ enum {
DDP_DEAD = (1 << 6), /* toepcb is shutting down */
};
struct sockopt;
struct ofld_tx_sdesc {
uint32_t plen; /* payload length */
uint8_t tx_credits; /* firmware tx credits (unit is 16B) */
void *iv_buffer; /* optional buffer holding IVs for TLS */
};
struct ppod_region {
@ -125,6 +131,9 @@ TAILQ_HEAD(pagesetq, pageset);
#define EXT_FLAG_AIOTX EXT_FLAG_VENDOR1
#define IS_AIOTX_MBUF(m) \
((m)->m_flags & M_EXT && (m)->m_ext.ext_flags & EXT_FLAG_AIOTX)
struct ddp_buffer {
struct pageset *ps;
@ -185,6 +194,7 @@ struct toepcb {
struct mbufq ulp_pdu_reclaimq;
struct ddp_pcb ddp;
struct tls_ofld_info tls;
TAILQ_HEAD(, kaiocb) aiotx_jobq;
struct task aiotx_task;
@ -269,6 +279,8 @@ struct tom_data {
struct ppod_region pr;
vmem_t *key_map;
struct mtx clip_table_lock;
struct clip_head clip_table;
int clip_gen;
@ -309,6 +321,18 @@ mbuf_ulp_submode(struct mbuf *m)
return (m->m_pkthdr.PH_per.eight[0]);
}
static inline int
is_tls_offload(struct toepcb *toep)
{
return (toep->ulp_mode == ULP_MODE_TLS);
}
static inline int
can_tls_offload(struct adapter *sc)
{
return (sc->tt.tls && sc->cryptocaps & FW_CAPS_CONFIG_TLSKEYS);
}
/* t4_tom.c */
struct toepcb *alloc_toepcb(struct vi_info *, int, int, int);
struct toepcb *hold_toepcb(struct toepcb *);
@ -327,7 +351,8 @@ int select_rcv_wscale(void);
uint64_t calc_opt0(struct socket *, struct vi_info *, struct l2t_entry *,
int, int, int, int);
uint64_t select_ntuple(struct vi_info *, struct l2t_entry *);
void set_tcpddp_ulp_mode(struct toepcb *);
int select_ulp_mode(struct socket *, struct adapter *);
void set_ulp_mode(struct toepcb *, int);
int negative_advice(int);
struct clip_entry *hold_lip(struct tom_data *, struct in6_addr *,
struct clip_entry *);
@ -362,7 +387,10 @@ void t4_uninit_cpl_io_handlers(void);
void send_abort_rpl(struct adapter *, struct sge_wrq *, int , int);
void send_flowc_wr(struct toepcb *, struct flowc_tx_params *);
void send_reset(struct adapter *, struct toepcb *, uint32_t);
int send_rx_credits(struct adapter *, struct toepcb *, int);
void send_rx_modulate(struct adapter *, struct toepcb *);
void make_established(struct toepcb *, uint32_t, uint32_t, uint16_t);
int t4_close_conn(struct adapter *, struct toepcb *);
void t4_rcvd(struct toedev *, struct tcpcb *);
void t4_rcvd_locked(struct toedev *, struct tcpcb *);
int t4_tod_output(struct toedev *, struct tcpcb *);
@ -401,4 +429,18 @@ void handle_ddp_indicate(struct toepcb *);
void handle_ddp_tcb_rpl(struct toepcb *, const struct cpl_set_tcb_rpl *);
void insert_ddp_data(struct toepcb *, uint32_t);
/* t4_tls.c */
int t4_ctloutput_tls(struct socket *, struct sockopt *);
void t4_push_tls_records(struct adapter *, struct toepcb *, int);
void t4_tls_mod_load(void);
void t4_tls_mod_unload(void);
void tls_establish(struct toepcb *);
void tls_free_kmap(struct tom_data *);
int tls_init_kmap(struct adapter *, struct tom_data *);
void tls_init_toep(struct toepcb *);
int tls_rx_key(struct toepcb *);
void tls_stop_handshake_timer(struct toepcb *);
int tls_tx_key(struct toepcb *);
void tls_uninit_toep(struct toepcb *);
#endif

1
sys/modules/cxgbe/tom/Makefile
View File

@ -16,6 +16,7 @@ SRCS+= t4_connect.c
SRCS+= t4_cpl_io.c
SRCS+= t4_ddp.c
SRCS+= t4_listen.c
SRCS+= t4_tls.c
SRCS+= t4_tom.c
SRCS+= t4_tom_l2t.c