Commit Graph

194 Commits

Author SHA1 Message Date
Matt Macy
d7c5a620e2 ifnet: Replace if_addr_lock rwlock with epoch + mutex
Run on LLNW canaries and tested by pho@

gallatin:
Using a 14-core, 28-HTT single socket E5-2697 v3 with a 40GbE MLX5
based ConnectX 4-LX NIC, I see an almost 12% improvement in received
packet rate, and a larger improvement in bytes delivered all the way
to userspace.

When the host receiving 64 streams of netperf -H $DUT -t UDP_STREAM -- -m 1,
I see, using nstat -I mce0 1 before the patch:

InMpps OMpps  InGbs  OGbs err TCP Est %CPU syscalls csw     irq GBfree
4.98   0.00   4.42   0.00 4235592     33   83.80 4720653 2149771   1235 247.32
4.73   0.00   4.20   0.00 4025260     33   82.99 4724900 2139833   1204 247.32
4.72   0.00   4.20   0.00 4035252     33   82.14 4719162 2132023   1264 247.32
4.71   0.00   4.21   0.00 4073206     33   83.68 4744973 2123317   1347 247.32
4.72   0.00   4.21   0.00 4061118     33   80.82 4713615 2188091   1490 247.32
4.72   0.00   4.21   0.00 4051675     33   85.29 4727399 2109011   1205 247.32
4.73   0.00   4.21   0.00 4039056     33   84.65 4724735 2102603   1053 247.32

After the patch

InMpps OMpps  InGbs  OGbs err TCP Est %CPU syscalls csw     irq GBfree
5.43   0.00   4.20   0.00 3313143     33   84.96 5434214 1900162   2656 245.51
5.43   0.00   4.20   0.00 3308527     33   85.24 5439695 1809382   2521 245.51
5.42   0.00   4.19   0.00 3316778     33   87.54 5416028 1805835   2256 245.51
5.42   0.00   4.19   0.00 3317673     33   90.44 5426044 1763056   2332 245.51
5.42   0.00   4.19   0.00 3314839     33   88.11 5435732 1792218   2499 245.52
5.44   0.00   4.19   0.00 3293228     33   91.84 5426301 1668597   2121 245.52

Similarly, netperf reports 230Mb/s before the patch, and 270Mb/s after the patch

Reviewed by:	gallatin
Sponsored by:	Limelight Networks
Differential Revision:	https://reviews.freebsd.org/D15366
2018-05-18 20:13:34 +00:00
Pedro F. Giffuni
df57947f08 spdx: initial adoption of licensing ID tags.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

Initially, only tag files that use BSD 4-Clause "Original" license.

RelNotes:	yes
Differential Revision:	https://reviews.freebsd.org/D13133
2017-11-18 14:26:50 +00:00
Alan Cox
92993af469 Since the page "frame" doesn't belong to a vm object, it can't be paged
out.  Since it can't be paged out, it is never actually enqueued in a
paging queue.  Nonetheless, passing PQ_INACTIVE to vm_page_unwire()
creates the appearance that the page "frame" is being enqueued in the
inactive queue.  As of r288122, we can avoid this false impression by
passing PQ_NONE.

MFC after:	1 week
2017-09-24 02:50:59 +00:00
Pedro F. Giffuni
453130d9bf sys/dev: minor spelling fixes.
Most affect comments, very few have user-visible effects.
2016-05-03 03:41:25 +00:00
Pedro F. Giffuni
d9c9c81c08 sys: use our roundup2/rounddown2() macros when param.h is available.
rounddown2 tends to produce longer lines than the original code
and when the code has a high indentation level it was not really
advantageous to do the replacement.

This tries to strike a balance between readability using the macros
and flexibility of having the expressions, so not everything is
converted.
2016-04-21 19:57:40 +00:00
Pedro F. Giffuni
74b8d63dcc Cleanup unnecessary semicolons from the kernel.
Found with devel/coccinelle.
2016-04-10 23:07:00 +00:00
Gleb Smirnoff
2bab0c5535 New sendfile(2) syscall. A joint effort of NGINX and Netflix from 2013 and
up to now.

The new sendfile is the code that Netflix uses to send their multiple tens
of gigabits of data per second. The new implementation features asynchronous
I/O, when I/O operations are launched, but not awaited to be complete. An
explanation of why such behavior is beneficial compared to old one is
going to be too long for a commit message, so we will skip it here.

Additional features of new syscall are extra flags, which provide an
application more control over data sent. The SF_NOCACHE flag tells
kernel that data shouldn't be cached after it was sent. The SF_READAHEAD()
macro allows to specify readahead size in pages.

The new syscalls is a drop in replacement. No modifications are required
to applications. One can take nginx binary for stable/10 and run it
successfully on head. Although SF_NODISKIO lost its original sense, as now
sendfile doesn't block, and now means something completely different (tm),
using the new sendfile the old way is absolutely safe.

Celebrates:	Netflix global launch!
Sponsored by:	Nginx, Inc.
Sponsored by:	Netflix
Relnotes:	yes
2016-01-08 20:34:57 +00:00
Dimitry Andric
71d827559b Fix r284722, by making it actually compile.
Pointy hat to:	dim
2015-06-23 06:59:46 +00:00
Dimitry Andric
42e2071b63 Fix endless recursion in ti(4)'s ti_ifmedia_upd(), found by clang 3.7.0. 2015-06-23 06:48:02 +00:00
Robert Watson
2a8c860fe3 In order to reduce use of M_EXT outside of the mbuf allocator and
socket-buffer implementations, introduce a return value for MCLGET()
(and m_cljget() that underlies it) to allow the caller to avoid testing
M_EXT itself.  Update all callers to use the return value.

With this change, very few network device drivers remain aware of
M_EXT; the primary exceptions lie in mbuf-chain pretty printers for
debugging, and in a few cases, custom mbuf and cluster allocation
implementations.

NB: This is a difficult-to-test change as it touches many drivers for
which I don't have physical devices.  Instead we've gone for intensive
review, but further post-commit review would definitely be appreciated
to spot errors where changes could not easily be made mechanically,
but were largely mechanical in nature.

Differential Revision:	https://reviews.freebsd.org/D1440
Reviewed by:	adrian, bz, gnn
Sponsored by:	EMC / Isilon Storage Division
2015-01-06 12:59:37 +00:00
Gleb Smirnoff
0353415eb6 - Mechanically convert to if_inc_counter() the rest of counters.
- Do not set if_collisions on interrupt, read them in ti_get_counter().
- Add missing bus_dmamap_sync(BUS_DMASYNC_PREREAD) in ti_ioctl2(). [1]

Submitted by:	mav [1]
2014-09-26 10:47:57 +00:00
John Baldwin
c34f1a08c6 Fix teardown of static DMA allocations in various NIC drivers:
- Add missing calls to bus_dmamap_unload() in et(4).
- Check the bus address against 0 to decide when to call
  bus_dmamap_unload() instead of comparing the bus_dma map against NULL.
- Check the virtual address against NULL to decide when to call
  bus_dmamem_free() instead of comparing the bus_dma map against NULL.
- Don't clear bus_dma map pointers to NULL for static allocations.
  Instead, treat the value as completely opaque.
- Pass the correct virtual address to bus_dmamem_free() in wpi(4) instead
  of trying to free a pointer to the virtual address.

Reviewed by:	yongari
2014-06-17 14:47:49 +00:00
Attilio Rao
3ae10f7477 - Modify vm_page_unwire() and vm_page_enqueue() to directly accept
the queue where to enqueue pages that are going to be unwired.
- Add stronger checks to the enqueue/dequeue for the pagequeues when
  adding and removing pages to them.

Of course, for unmanaged pages the queue parameter of vm_page_unwire() will
be ignored, just as the active parameter today.
This makes adding new pagequeues quicker.

This change effectively modifies the KPI.  __FreeBSD_version will be,
however, bumped just when the full cache of free pages will be
evicted.

Sponsored by:	EMC / Isilon storage division
Reviewed by:	alc
Tested by:	pho
2014-06-16 18:15:27 +00:00
Gleb Smirnoff
76039bc84f The r48589 promised to remove implicit inclusion of if_var.h soon. Prepare
to this event, adding if_var.h to files that do need it. Also, include
all includes that now are included due to implicit pollution via if_var.h

Sponsored by:	Netflix
Sponsored by:	Nginx, Inc.
2013-10-26 17:58:36 +00:00
Andre Oppermann
9d1c4ca773 Remove unused and incomplete support for delayed fragment checksums
from bce(4), bxe(4), mge(4) and ti(4) drivers.
2013-08-19 10:20:20 +00:00
Gleb Smirnoff
c6499eccad Mechanically substitute flags from historic mbuf allocator with
malloc(9) flags in sys/dev.
2012-12-04 09:32:43 +00:00
Dimitry Andric
29658c96ce Remove duplicate const specifiers in many drivers (I hope I got all of
them, please let me know if not).  Most of these are of the form:

static const struct bzzt_type {
	[...list of members...]
} const bzzt_devs[] = {
	[...list of initializers...]
};

The second const is unnecessary, as arrays cannot be modified anyway,
and if the elements are const, the whole thing is const automatically
(e.g. it is placed in .rodata).

I have verified this does not change the binary output of a full kernel
build (except for build timestamps embedded in the object files).

Reviewed by:	yongari, marius
MFC after:	1 week
2012-11-05 19:16:27 +00:00
Pyun YongHyeon
a8eeb50c81 Remove TCP/UDP checksum offloading feature for IP fragmented
datagrams.  Traditionally upper stack fragmented packets without
computing TCP/UDP checksum and these datagrams were passed to
driver.  But there are chances that other packets slip into the
interface queue in SMP world. If this happens firmware running on
MIPS 4000 processor in the controller would see mixed packets and
it shall send out corrupted packets.
While I'm here simplify checksum offloading setup.

MFC After:	1 week
2012-11-01 05:39:21 +00:00
Alan Cox
3b03ca3bbe Eliminate vestiges of page coloring. 2011-12-15 05:07:16 +00:00
Pyun YongHyeon
b7c2632d6a Overhaul bus_dma(9) usage in driver:
- Don't use a single big DMA block for all rings. Create separate
   DMA area for each ring instead.  Currently the following DMA
   areas are created:
	Event ring, standard RX ring, jumbo RX ring, RX return ring,
	hardware MAC statistics and producer/consumer status area.
   For Tigon II, mini RX ring and TX ring are additionally created.
 - Added missing bus_dmamap_sync(9) in various TX/RX paths.
 - TX ring is no longer created for Tigon 1 such that it saves more
   resources on Tigon 1.
 - Data sheet is not clear about alignment requirement of each ring
   so use 32 bytes alignment for normal DMA area but use 64 bytes
   alignment for jumbo RX ring where the extended RX descriptor
   size is 64 bytes.
 - For each TX/RX buffers use separate DMA tag(e.g. the size of a
   DMA segment, total size of DMA segments etc).
 - Tigon allows separate DMA area for event producer, RX return
   producer and TX consumer which is really cool feature.  This
   means TX and RX path could be independently run in parallel.
   However ti(4) uses a single driver lock so it's meaningless
   to have separate DMA area for these producer/consumer such that
   this change creates a single status DMA area.
 - It seems Tigon has no limits on DMA address space and I also
   don't see any problem with that but old comments in driver
   indicates there could be issues on descriptors being located in
   64bit region.  Introduce a tunable, dev.ti.%d.dac, to disable
   using 64bit DMA in driver. The default is 0 which means it would
   use full 64bit DMA.  If there are DMA issues, users can disable
   it by setting the tunable to 0.
 - Do not increase watchdog timer in ti_txeof(). Previously driver
   increased the watchdog timer whenever there are queued TX frames.
 - When stat ticks is set to 0, skip processing ti_stats_update(),
   avoiding bus_dmamap_sync(9) and updating if_collisions counter.
 - MTU does not include FCS bytes, replace it with
   ETHER_VLAN_ENCAP_LEN.

With these changes, ti(4) should work on PAE environments.
Many thanks to Jay Borkenhagen for remote hardware access.
2011-11-14 20:38:14 +00:00
Pyun YongHyeon
e79b243289 Export sysctl node for various interrupt moderation parameters and
have administrators control them.  ti(4) provides a character
device to control various other features of driver via ioctls but
users had to write their own code to manipulate these parameters.
It seems some default values for these parameters are not optimal
on today's system but leave it as it was and let administrators
change them.  The following parameters could be changed:

dev.ti.%d.rx_coal_ticks
dev.ti.%d.rx_max_coal_bds
dev.ti.%d.tx_coal_ticks
dev.ti.%d.tx_max_coal_bds
dev.ti.%d.tx_buf_ratio
dev.ti.%d.stat_ticks

The interface has to be brought down and up again before a change
takes effect.

ti(4) controller supports hardware MAC counters with additional
DMA statistics.  So it's doable to export these counters via
sysctl interface.  Unfortunately, these counters are cumulative
such that driver have to either send an explicit clear command to
controller after extracting them or have to maintain internal
counters to get actual changes.  Neither look good to me so
counters were not exported via sysctl.
2011-11-14 19:10:20 +00:00
Pyun YongHyeon
9b81d5e37f It's bad idea to allocate large memory, 4KB, from stack.
Pre-allocate the memory in device attach time. While I'm here
remove unnecessary reassignment of error variable as it was already
initialized. Also added a missing driver lock in TIIOCSETTRACE
handler.
2011-11-14 18:40:04 +00:00
Pyun YongHyeon
ecf3f60fed Remove dead ifdef. Driver should always check raised interrupt is
for the device.
2011-11-10 23:14:04 +00:00
Pyun YongHyeon
8a9710aa8c style.
No functional changes.
2011-11-10 22:15:11 +00:00
Pyun YongHyeon
10a4360c80 Retire 'options TI_PRIVATE_JUMBOS' and replace local jumbo
allocator with UMA backed jumbo allocator by default. Previously
ti(4) used sf_buf(9) interface for jumbo buffers but it was broken
at this moment such that enabling jumbo frame caused instant panic.
Due to the nature of sf_buf(9) it heavily relies on VM changes but
it seems ti(4) was not received much blessing from VM gurus.  I
don't understand VM magic and implications used in driver either.
Switching to UMA backed jumbo allocator like other network drivers
will make jumbo frame work on ti(4).
While I'm here, fully allocate all RX buffers. This means ti(4) now
uses 512 RX buffer and 1024 mini RX buffers.

To use sf_buf(9) interface for jumbo buffers, introduce a new
'options TI_SF_BUF_JUMBO'. If it is proven that sf_buf(9) is better
for jumbo buffers, interesting developers can fix the issue in
future.

ti(4) still needs more bus_dma(9) cleanups and should use separate
DMA tag/map for each ring(standard, jumbo, mini, command, event
etc) but it should work on all platforms except PAE.

Special thanks to Jay[1] who provided complete remote debugging
access.

Tested by:	Jay Borkenhagen <jayb <> braeburn dot org > [1]
2011-11-08 18:23:02 +00:00
Pyun YongHyeon
f9ea040e0c Do not allow changing MTU to be less than the minimum. 2011-11-07 22:58:49 +00:00
Pyun YongHyeon
aeeee7eec1 If ti_chipinit() fails in ti_stop(), ignore the error and release
all allocated TX/RX buffer resources. If the interface is brought
to up again after the error, we will leak allocated TX/RX buffers.
2011-11-07 22:53:06 +00:00
Pyun YongHyeon
4fce1b52ed Show RX buffer allocation failure and do not blindly send alive
message to firmware. Probably the correct way for this error is to
send a TI_CMD_CODE_STACK_DOWN message to firmware and let firmware
handle the rest.
2011-11-07 22:47:25 +00:00
Pyun YongHyeon
6280f2fb34 Mini ring is not available on Tigon 1 so do not create DMA maps for
mini ring on Tigon 1 to save resources.
2011-11-07 22:17:44 +00:00
Pyun YongHyeon
7beaf112a5 Track which ring was updated in RX handler and update only modified
ring. This should reduce unnecessary register accesses.
2011-11-07 22:11:38 +00:00
Pyun YongHyeon
18707510b7 o Remove unnecessary controller reinitialization.
o Do not blindly UP controller when MTU is changed. Reinitialize
   controller only if driver is running.
 o Remove useless ti_stop() in ti_watchdog() since ti_init_locked()
   always invokes ti_stop().
2011-11-07 18:26:38 +00:00
Pyun YongHyeon
bb6ebb6af8 Remove ti_unit member variable in softc.
While I'm here use PCIR_BAR macro.
2011-11-07 18:19:20 +00:00
Pyun YongHyeon
129f7efda9 Implement altq(4) support.
While I'm here fix a logic error in r227098 where it didn't
re-enable interrupts when TX queue is empty.
2011-11-04 23:34:54 +00:00
Pyun YongHyeon
9c6960c61c Because ti(4) drops a driver lock in RX handler, check whether
driver is still running before re-enabling interrupts.
2011-11-04 23:09:57 +00:00
Pyun YongHyeon
85d21e66be Don't abuse if_hwassist and make sure enabling corresponding TX/RX
checksum offloading and VLAN hardware tag insertion/stripping from
the currently enabled hardware offloading capabilities.
Previously if_hwassist, which was initialized to TX/RX checksum
offloading, was blindly used to enable both TX and RX checksum
offloading such that disabling either TX or RX checksum offloading
was not possible.

ti(4) controllers support TX/RX checksum offloading with VLAN
tagging so announce TX/RX checksum offloading capability over VLAN
to vlan(4).

Make VLAN hardware tag insertion/stripping honors currently enabled
interface capability instead of blindly enabling VLAN hardware
tagging. This change allows disabling hardware support of VLAN tag.

Because ti(4) supports VLAN oversized frames, make network stack
know the capability by setting if_hdrlen.

While I'm here, rewrite SIOCSIFCAP handler and make sure to
reinitialize controller whenever TX/RX checksum offloading and VLAN
hardware tagging option is changed.  The requirement of controller
reinitialization comes from the limitation of Tigon I/II firmware.
Tigon I/II firmware requires all related RCBs should be
reinitialized whenever any of its hardware offloading capabilities
change.

vlan(4) is also notified whenever the parent interface's capability
changes such that it can correctly handle TX/RX checksum offloading
based on parent interface's enabled offloading capabilities.

RX checksum offloading handler was changed to make upper stack use
controller computed partial checksum value.  Previously, ti(4) just
set the computed value for any frames(IPv4, IPv6) and the value was
not used in upper stack because driver didn't set CSUM_DATA_VALID
such that upper network stack had to recompute checksum of TCP/UDP
packets. I have no idea how this was not noticed for a long time.
With this change, upper network stack does not have to fully
recompute the checksum such that calculating pseudo checksum based
on partial checksum is sufficient to know whether received packet's
checksum is correct or not. However, I don't know why ti(4) does
not have controller compute pseudo checksum as controller has
ability to do it. I'm just guessing enabling that feature could
trigger a firmware bug or could be slower than computing it on host
side so just leave it as it was.

In order not to produce false positives, ti(4) now checks whether
controller actually computed IP or TCP/UDP checksum by checking
ti_flags field.
2011-11-04 22:53:52 +00:00
Pyun YongHyeon
9645f9eb0b Don't clear upper 4bits from VLAN tag information. It's
responsibility of vlan(4) to extract VLAN id from the tag
information and vlan(4) correctly handles it.
2011-11-04 21:42:13 +00:00
Pyun YongHyeon
148386d9aa Introduce ti_ifmedia_upd_locked() to use in driver initialization
and add missing driver lock for both ti_ifmedia_upd() and
ti_ifmedia_sts().
2011-11-04 21:30:46 +00:00
Pyun YongHyeon
945c212659 Announce IFCAP_LINKSTATE capability and let network stack know link
state changes.  Hide superfluous link up/down message under
bootverbose since if_link_state_change(9) shows that information.
While I'm here, change baudrate with the resolved speed of the
established link instead of blindly setting it 1G. Unfortunately,
it seems there is no way to differentiate 10/100Mbps from
non-gigabit link so just assume we established a 100Mbps link if
current link is not a gigabit link.
2011-11-04 20:43:37 +00:00
Pyun YongHyeon
5948254c2a Make sure to unload loaded DMA area(descriptor, command, event ring). 2011-11-04 20:25:30 +00:00
Pyun YongHyeon
5ddfea8c43 s/u_intXX_t/uintXX_t/g 2011-11-04 19:12:07 +00:00
Pyun YongHyeon
ee715c5ff4 Make ti(4) build with 'options TI_PRIVATE_JUMBOS'.
This was broken in r175872.

We have a UMA backed jumbo allocator and that is much better
implementation than having a local jumbo buffer allocator in
driver. This local allocator would be removed in near future but
fixing build before removal wouldn't be a bad idea.
2011-11-04 18:39:39 +00:00
Pyun YongHyeon
504dc87bf4 style.
No functional changes.
2011-11-04 18:28:10 +00:00
Pyun YongHyeon
7034f3ad36 Use ANSI function definations. 2011-11-04 17:07:53 +00:00
Marius Strobl
a1d090d428 - Allocate the DMA memory shared between the host and the controller as
coherent.
- Constify the ti_devs table.
- Don't bother to set if_mtu to ETHERMTU, ether_ifattach() does that.

MFC after:	2 weeks
2011-03-11 22:32:17 +00:00
Alan Cox
492d01762f Neither the page lock nor the page queues lock is required to unwire and
free a VM_ALLOC_NOOBJ page.  (Such pages are unmanaged.)
2010-05-03 15:51:59 +00:00
John Baldwin
7cf545d0a1 - Add a private timer to drive the transmit watchdog instead of using
if_watchdog and if_timer.
- Fix some issues in detach for sn(4), ste(4), and ti(4).  Primarily this
  means calling ether_ifdetach() before anything else.
2009-11-19 22:06:40 +00:00
Robert Watson
eb956cd041 Use if_maddr_rlock()/if_maddr_runlock() rather than IF_ADDR_LOCK()/
IF_ADDR_UNLOCK() across network device drivers when accessing the
per-interface multicast address list, if_multiaddrs.  This will
allow us to change the locking strategy without affecting our driver
programming interface or binary interface.

For two wireless drivers, remove unnecessary locking, since they
don't actually access the multicast address list.

Approved by:	re (kib)
MFC after:	6 weeks
2009-06-26 11:45:06 +00:00
Marius Strobl
44f8f2fc05 Remove some remnant alpha hacks.
Approved by:	PCI-maintainers (imp, jhb)
2008-04-26 14:13:48 +00:00
Christian Brueffer
662cac9f23 Fix some "in in" typos in comments.
PR:		121490
Submitted by:	Anatoly Borodin <anatoly.borodin@gmail.com>
Approved by:	rwatson (mentor), jkoshy
MFC after:	3 days
2008-03-26 07:32:08 +00:00
Remko Lodder
f17edc7494 Set the baudrate for if_ti.
PR:		kern/40516
Submitted by:	"Jin Guojun[VFF]" <jin at adsl-63-198-35-122 dot dsl dot snfc21 dot pacbell dot net>
Approved by:	imp (mentor, implicit for minor changes)
MFC After:	1 week
2008-02-19 20:54:42 +00:00