d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Driver for 4x10Gb Ethernet reference NIC FPGA design for NetFPGA SUME

Browse Source 
development board.

Submitted by:	Denis Salopek <denis.salopek AT fer.hr>
Reported by:	zec, bz (src); rgrimes, bcr (manpages)
MFC after:	7 days
Sponsored by:	Google Summer of Code 2020
Differential Revision:	https://reviews.freebsd.org/D26074
This commit is contained in:
Marko Zec 2020-08-30 07:34:32 +00:00
parent 0820428967
commit bd36872867
Notes: svn2git 2020-12-20 02:59:44 +00:00 
svn path=/head/; revision=364973
7 changed files with 1957 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
share/man/man4/Makefile
View File

@ -514,6 +514,7 @@ MAN= aac.4 \
ste.4 \
stf.4 \
stge.4 \
${_sume.4} \
${_superio.4} \
sym.4 \
syncache.4 \
@ -851,6 +852,7 @@ _qlxgbe.4= qlxgbe.4
_qlnxe.4= qlnxe.4
_sfxge.4= sfxge.4
_smartpqi.4= smartpqi.4
_sume.4= sume.4
_vmd.4= vmd.4
MLINKS+=qlxge.4 if_qlxge.4
@ -858,6 +860,7 @@ MLINKS+=qlxgb.4 if_qlxgb.4
MLINKS+=qlxgbe.4 if_qlxgbe.4
MLINKS+=qlnxe.4 if_qlnxe.4
MLINKS+=sfxge.4 if_sfxge.4
MLINKS+=sume.4 if_sume.4
.if ${MK_BHYVE} != "no"
_bhyve.4= bhyve.4

98
share/man/man4/sume.4 Normal file
View File

@ -0,0 +1,98 @@
.\"-
.\" SPDX-License-Identifier: BSD-2-Clause-FreeBSD
.\"
.\" Copyright (c) 2020 Denis Salopek
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
.\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
.\" LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
.\" POSSIBILITY OF SUCH DAMAGE.
.\"
.\" $FreeBSD$
.\"
.Dd August 30, 2020
.Dt SUME 4
.Os
.Sh NAME
.Nm sume
.Nd "NetFPGA SUME 4x10Gb Ethernet driver"
.Sh SYNOPSIS
To compile this driver into the kernel, place the following lines
in your kernel configuration file:
.Bd -ragged -offset indent
.Cd "device sume"
.Ed
.Pp
Alternatively, to load the driver as a module at boot time, place
the following line in
.Xr loader.conf 5 :
.Bd -literal -offset indent
if_sume_load="YES"
.Ed
.Sh DESCRIPTION
The
.Nm
driver provides support for NetFPGA SUME Virtex-7 FPGA Development Board
with the reference NIC bitstream loaded onto it.
The HDL design for the reference NIC project uses the RIFFA based DMA
engine to communicate with the host machine over PCIe.
Every packet is transmitted to / from the board via a single DMA
transaction, taking up to two or three interrupts per one transaction
which yields low performance.
.Pp
There is no support for Jumbo frames as the hardware is capable of
dealing only with frames with maximum size of 1514 bytes.
The hardware does not support multicast filtering, provides no checksums,
and offers no other offloading.
.Sh SEE ALSO
.Xr arp 4 ,
.Xr netgraph 4 ,
.Xr netintro 4 ,
.Xr ng_ether 4 ,
.Xr vlan 4 ,
.Xr ifconfig 8
.Sh AUTHORS
The Linux
.Nm
driver was originally written by
.An -nosplit
.An Bjoern A. Zeeb .
The
.Fx version and this manual page were written by
.An Denis Salopek
as a GSoC project.
More information about the project can be found here:
.Pa https://wiki.freebsd.org/SummerOfCode2020Projects/NetFPGA_SUME_Driver
.Sh BUGS
The reference NIC hardware design provides no mechanism for quiescing
inbound traffic from interfaces configured as DOWN.
All packets from administratively disabled interfaces are transferred to
main memory, leaving the driver with the task of dropping such packets,
thus consuming PCI bandwidth, interrupts and CPU cycles in vain.
.Pp
Pre-built FPGA bitstream from the NetFPGA project may not work correctly.
At higher RX packet rates, the newly incoming packets can overwrite the
ones in an internal FIFO so the packets would arrive in main memory
corrupted, until a physical reset of the board.
.Pp
Occasionally, the driver can get stuck in a non-IDLE TX state due to
a missed interrupt.
The driver includes a watchdog function which monitors for such a
condition and resets the board automatically.
For more details, visit the NetFPGA SUME project site.

1
sys/conf/files.amd64
View File

@ -355,6 +355,7 @@ dev/smartpqi/smartpqi_response.c optional smartpqi
dev/smartpqi/smartpqi_sis.c optional smartpqi
dev/smartpqi/smartpqi_tag.c optional smartpqi
dev/speaker/spkr.c optional speaker
dev/sume/if_sume.c optional sume
dev/superio/superio.c optional superio isa
dev/syscons/apm/apm_saver.c optional apm_saver apm
dev/syscons/scvesactl.c optional sc vga vesa

242
sys/dev/sume/adapter.h Normal file
View File

@ -0,0 +1,242 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
*
* Copyright (c) 2015 Bjoern A. Zeeb
* Copyright (c) 2020 Denis Salopek
*
* This software was developed by SRI International and the University of
* Cambridge Computer Laboratory under DARPA/AFRL contract FA8750-11-C-0249
* ("MRC2"), as part of the DARPA MRC research programme.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/* $FreeBSD$ */
#define DEFAULT_ETHER_ADDRESS "\02SUME\00"
#define SUME_ETH_DEVICE_NAME "sume"
#define MAX_IFC_NAME_LEN 8
#define SUME_NPORTS 4
#define SUME_IOCTL_CMD_WRITE_REG (SIOCGPRIVATE_0)
#define SUME_IOCTL_CMD_READ_REG (SIOCGPRIVATE_1)
#define SUME_LOCK(adapter) mtx_lock(&adapter->lock);
#define SUME_UNLOCK(adapter) mtx_unlock(&adapter->lock);
/* Currently SUME only uses 2 fixed channels for all port traffic and regs. */
#define SUME_RIFFA_CHANNEL_DATA 0
#define SUME_RIFFA_CHANNEL_REG 1
#define SUME_RIFFA_CHANNELS 2
/* RIFFA constants. */
#define RIFFA_MAX_CHNLS 12
#define RIFFA_MAX_BUS_WIDTH_PARAM 4
#define RIFFA_SG_BUF_SIZE (4*1024)
#define RIFFA_SG_ELEMS 200
/* RIFFA register offsets. */
#define RIFFA_RX_SG_LEN_REG_OFF 0x0
#define RIFFA_RX_SG_ADDR_LO_REG_OFF 0x1
#define RIFFA_RX_SG_ADDR_HI_REG_OFF 0x2
#define RIFFA_RX_LEN_REG_OFF 0x3
#define RIFFA_RX_OFFLAST_REG_OFF 0x4
#define RIFFA_TX_SG_LEN_REG_OFF 0x5
#define RIFFA_TX_SG_ADDR_LO_REG_OFF 0x6
#define RIFFA_TX_SG_ADDR_HI_REG_OFF 0x7
#define RIFFA_TX_LEN_REG_OFF 0x8
#define RIFFA_TX_OFFLAST_REG_OFF 0x9
#define RIFFA_INFO_REG_OFF 0xA
#define RIFFA_IRQ_REG0_OFF 0xB
#define RIFFA_IRQ_REG1_OFF 0xC
#define RIFFA_RX_TNFR_LEN_REG_OFF 0xD
#define RIFFA_TX_TNFR_LEN_REG_OFF 0xE
#define RIFFA_CHNL_REG(c, o) ((c << 4) + o)
/*
* RIFFA state machine;
* rather than using complex circular buffers for 1 transaction.
*/
#define SUME_RIFFA_CHAN_STATE_IDLE 0x01
#define SUME_RIFFA_CHAN_STATE_READY 0x02
#define SUME_RIFFA_CHAN_STATE_READ 0x04
#define SUME_RIFFA_CHAN_STATE_LEN 0x08
/* Accessor macros. */
#define SUME_OFFLAST ((0 << 1) | (1 & 0x01))
#define SUME_RIFFA_LAST(offlast) ((offlast) & 0x01)
#define SUME_RIFFA_OFFSET(offlast) ((uint64_t)((offlast) >> 1) << 2)
#define SUME_RIFFA_LEN(len) ((uint64_t)(len) << 2)
#define SUME_RIFFA_LO_ADDR(addr) (addr & 0xFFFFFFFF)
#define SUME_RIFFA_HI_ADDR(addr) ((addr >> 32) & 0xFFFFFFFF)
/* Vector bits. */
#define SUME_MSI_RXQUE (1 << 0)
#define SUME_MSI_RXBUF (1 << 1)
#define SUME_MSI_RXDONE (1 << 2)
#define SUME_MSI_TXBUF (1 << 3)
#define SUME_MSI_TXDONE (1 << 4)
/* Invalid vector. */
#define SUME_INVALID_VECT 0xc0000000
/* Module register data (packet counters, link status...) */
#define SUME_MOD0_REG_BASE 0x44040000
#define SUME_MOD_REG(port) (SUME_MOD0_REG_BASE + 0x10000 * port)
#define SUME_RESET_OFFSET 0x8
#define SUME_PKTIN_OFFSET 0x18
#define SUME_PKTOUT_OFFSET 0x1c
#define SUME_STATUS_OFFSET 0x48
#define SUME_RESET_ADDR(p) (SUME_MOD_REG(p) + SUME_RESET_OFFSET)
#define SUME_STAT_RX_ADDR(p) (SUME_MOD_REG(p) + SUME_PKTIN_OFFSET)
#define SUME_STAT_TX_ADDR(p) (SUME_MOD_REG(p) + SUME_PKTOUT_OFFSET)
#define SUME_STATUS_ADDR(p) (SUME_MOD_REG(p) + SUME_STATUS_OFFSET)
#define SUME_LINK_STATUS(val) ((val >> 12) & 0x1)
/* Various bits and pieces. */
#define SUME_RIFFA_MAGIC 0xcafe
#define SUME_MR_WRITE 0x1f
#define SUME_MR_READ 0x00
#define SUME_INIT_RTAG -3
#define SUME_DPORT_MASK 0xaa
#define SUME_MIN_PKT_SIZE (ETHER_MIN_LEN - ETHER_CRC_LEN)
struct irq {
uint32_t rid;
struct resource *res;
void *tag;
} __aligned(CACHE_LINE_SIZE);
struct nf_stats {
uint64_t hw_rx_packets;
uint64_t hw_tx_packets;
uint64_t ifc_down_bytes;
uint64_t ifc_down_packets;
uint64_t rx_bytes;
uint64_t rx_dropped;
uint64_t rx_packets;
uint64_t tx_bytes;
uint64_t tx_dropped;
uint64_t tx_packets;
};
struct riffa_chnl_dir {
uint32_t state;
bus_dma_tag_t ch_tag;
bus_dmamap_t ch_map;
char *buf_addr; /* bouncebuf addresses+len. */
bus_addr_t buf_hw_addr; /* -- " -- mapped. */
uint32_t num_sg;
uint32_t event; /* Used for modreg r/w */
uint32_t len; /* words */
uint32_t offlast;
uint32_t recovery;
uint32_t rtag;
};
struct sume_ifreq {
uint32_t addr;
uint32_t val;
};
struct nf_priv {
struct sume_adapter *adapter;
struct ifmedia media;
struct nf_stats stats;
uint32_t unit;
uint32_t port;
uint32_t link_up;
};
struct sume_adapter {
struct mtx lock;
uint32_t running;
uint32_t rid;
struct riffa_chnl_dir **recv;
struct riffa_chnl_dir **send;
device_t dev;
struct ifnet *ifp[SUME_NPORTS];
struct resource *bar0_addr;
bus_space_tag_t bt;
bus_space_handle_t bh;
bus_size_t bar0_len;
struct irq irq;
struct callout timer;
struct task stat_task;
struct taskqueue *tq;
uint64_t bytes_err;
uint64_t packets_err;
uint32_t last_ifc;
uint32_t num_sg;
uint32_t sg_buf_size;
uint32_t sume_debug;
uint32_t wd_counter;
};
/* SUME metadata:
* sport - not used for RX. For TX, set to 0x02, 0x08, 0x20, 0x80, depending on
* the sending interface (nf0, nf1, nf2 or nf3).
* dport - For RX, is set to 0x02, 0x08, 0x20, 0x80, depending on the receiving
* interface (nf0, nf1, nf2 or nf3). For TX, set to 0x01, 0x04, 0x10, 0x40,
* depending on the sending HW interface (nf0, nf1, nf2 or nf3).
* plen - length of the send/receive packet data (in bytes)
* magic - SUME hardcoded magic number which should be 0xcafe
* t1, t1 - could be used for timestamping by SUME
*/
struct nf_metadata {
uint16_t sport;
uint16_t dport;
uint16_t plen;
uint16_t magic;
uint32_t t1;
uint32_t t2;
};
/* Used for ioctl communication with the rwaxi program used to read/write SUME
* internally defined register data.
* addr - address of the SUME module register to read/write
* val - value to write/read to/from the register
* rtag - returned on read: transaction tag, for syncronization
* optype - 0x1f when writing, 0x00 for reading
*/
struct nf_regop_data {
uint32_t addr;
uint32_t val;
uint32_t rtag;
uint32_t optype;
};
/* Our bouncebuffer "descriptor". This holds our physical address (lower and
* upper values) of the beginning of the DMA data to RX/TX. The len is number
* of words to transmit.
*/
struct nf_bb_desc {
uint32_t lower;
uint32_t upper;
uint32_t len;
};

1602
sys/dev/sume/if_sume.c Normal file
View File

File diff suppressed because it is too large Load Diff

2
sys/modules/Makefile
View File

@ -345,6 +345,7 @@ SUBDIR= \
${_sppp} \
ste \
stge \
${_sume} \
${_superio} \
${_sym} \
${_syscons} \
@ -719,6 +720,7 @@ _nvdimm= nvdimm
_pms= pms
_qlxge= qlxge
_qlxgb= qlxgb
_sume= sume
_vmd= vmd
.if ${MK_SOURCELESS_UCODE} != "no"
_qlxgbe= qlxgbe

9
sys/modules/sume/Makefile Normal file
View File

@ -0,0 +1,9 @@
# $FreeBSD$
.PATH: ${SRCTOP}/sys/dev/sume
KMOD= if_sume
SRCS= if_sume.c
SRCS+= device_if.h bus_if.h pci_if.h
.include <bsd.kmod.mk>