b2f095aaa6
a) Front load as much work as possible in if_transmit, before any driver lock or software queue has to get involved. b) Replace buf_ring with a brand new mp_ring (multiproducer ring). This is specifically for the tx multiqueue model where one of the if_transmit producer threads becomes the consumer and other producers carry on as usual. mp_ring is implemented as standalone code and it should be possible to use it in any driver with tx multiqueue. It also has: - the ability to enqueue/dequeue multiple items. This might become significant if packet batching is ever implemented. - an abdication mechanism to allow a thread to give up writing tx descriptors and have another if_transmit thread take over. A thread that's writing tx descriptors can end up doing so for an unbounded time period if a) there are other if_transmit threads continuously feeding the sofware queue, and b) the chip keeps up with whatever the thread is throwing at it. - accurate statistics about interesting events even when the stats come at the expense of additional branches/conditional code. The NIC txq lock is uncontested on the fast path at this point. I've left it there for synchronization with the control events (interface up/down, modload/unload). c) Add support for "type 1" coalescing work request in the normal NIC tx path. This work request is optimized for frames with a single item in the DMA gather list. These are very common when forwarding packets. Note that netmap tx in cxgbe already uses these "type 1" work requests. d) Do not request automatic cidx updates every 32 descriptors. Instead, request updates via bits in individual work requests (still every 32 descriptors approximately). Also, request an automatic final update when the queue idles after activity. This means NIC tx reclaim is still performed lazily but it will catch up quickly as soon as the queue idles. This seems to be the best middle ground and I'll probably do something similar for netmap tx as well. e) Implement a faster tx path for WRQs (used by TOE tx and control queues, _not_ by the normal NIC tx). Allow work requests to be written directly to the hardware descriptor ring if room is available. I will convert t4_tom and iw_cxgbe modules to this faster style gradually. MFC after: 2 months
69 lines
2.5 KiB
C
69 lines
2.5 KiB
C
/*-
|
|
* Copyright (c) 2014 Chelsio Communications, Inc.
|
|
* All rights reserved.
|
|
* Written by: Navdeep Parhar <np@FreeBSD.org>
|
|
*
|
|
* Redistribution and use in source and binary forms, with or without
|
|
* modification, are permitted provided that the following conditions
|
|
* are met:
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
* notice, this list of conditions and the following disclaimer.
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
* documentation and/or other materials provided with the distribution.
|
|
*
|
|
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
* SUCH DAMAGE.
|
|
*
|
|
* $FreeBSD$
|
|
*
|
|
*/
|
|
|
|
#ifndef __CXGBE_MP_RING_H
|
|
#define __CXGBE_MP_RING_H
|
|
|
|
#ifndef _KERNEL
|
|
#error "no user-serviceable parts inside"
|
|
#endif
|
|
|
|
struct mp_ring;
|
|
typedef u_int (*ring_drain_t)(struct mp_ring *, u_int, u_int);
|
|
typedef u_int (*ring_can_drain_t)(struct mp_ring *);
|
|
|
|
struct mp_ring {
|
|
volatile uint64_t state __aligned(CACHE_LINE_SIZE);
|
|
|
|
int size __aligned(CACHE_LINE_SIZE);
|
|
void * cookie;
|
|
struct malloc_type * mt;
|
|
ring_drain_t drain;
|
|
ring_can_drain_t can_drain; /* cheap, may be unreliable */
|
|
counter_u64_t enqueues;
|
|
counter_u64_t drops;
|
|
counter_u64_t starts;
|
|
counter_u64_t stalls;
|
|
counter_u64_t restarts; /* recovered after stalling */
|
|
counter_u64_t abdications;
|
|
|
|
void * volatile items[] __aligned(CACHE_LINE_SIZE);
|
|
};
|
|
|
|
int mp_ring_alloc(struct mp_ring **, int, void *, ring_drain_t,
|
|
ring_can_drain_t, struct malloc_type *, int);
|
|
void mp_ring_free(struct mp_ring *);
|
|
int mp_ring_enqueue(struct mp_ring *, void **, int, int);
|
|
void mp_ring_check_drainage(struct mp_ring *, int);
|
|
void mp_ring_reset_stats(struct mp_ring *);
|
|
int mp_ring_is_idle(struct mp_ring *);
|
|
|
|
#endif
|