freebsd-dev/sys/netinet/in_rss.c

358 lines
10 KiB
C
Raw Normal View History

Several years after initial development, merge prototype support for linking NIC Receive Side Scaling (RSS) to the network stack's connection-group implementation. This prototype (and derived patches) are in use at Juniper and several other FreeBSD-using companies, so despite some reservations about its maturity, merge the patch to the base tree so that it can be iteratively refined in collaboration rather than maintained as a set of gradually diverging patch sets. (1) Merge a software implementation of the Toeplitz hash specified in RSS implemented by David Malone. This is used to allow suitable pcbgroup placement of connections before the first packet is received from the NIC. Software hashing is generally avoided, however, due to high cost of the hash on general-purpose CPUs. (2) In in_rss.c, maintain authoritative versions of RSS state intended to be pushed to each NIC, including keying material, hash algorithm/ configuration, and buckets. Provide software-facing interfaces to hash 2- and 4-tuples for IPv4 and IPv6 using both the RSS standardised Toeplitz and a 'naive' variation with a hash efficient in software but with poor distribution properties. Implement rss_m2cpuid()to be used by netisr and other load balancing code to look up the CPU on which an mbuf should be processed. (3) In the Ethernet link layer, allow netisr distribution using RSS as a source of policy as an alternative to source ordering; continue to default to direct dispatch (i.e., don't try and requeue packets for processing on the 'right' CPU if they arrive in a directly dispatchable context). (4) Allow RSS to control tuning of connection groups in order to align groups with RSS buckets. If a packet arrives on a protocol using connection groups, and contains a suitable hardware-generated hash, use that hash value to select the connection group for pcb lookup for both IPv4 and IPv6. If no hardware-generated Toeplitz hash is available, we fall back on regular PCB lookup risking contention rather than pay the cost of Toeplitz in software -- this is a less scalable but, at my last measurement, faster approach. As core counts go up, we may want to revise this strategy despite CPU overhead. Where device drivers suitably configure NICs, and connection groups / RSS are enabled, this should avoid both lock and line contention during connection lookup for TCP. This commit does not modify any device drivers to tune device RSS configuration to the global RSS configuration; patches are in circulation to do this for at least Chelsio T3 and Intel 1G/10G drivers. Currently, the KPI for device drivers is not particularly robust, nor aware of more advanced features such as runtime reconfiguration/rebalancing. This will hopefully prove a useful starting point for refinement. No MFC is scheduled as we will first want to nail down a more mature and maintainable KPI/KBI for device drivers. Sponsored by: Juniper Networks (original work) Sponsored by: EMC/Isilon (patch update and merge)
2014-03-15 00:57:50 +00:00
/*-
* Copyright (c) 2010-2011 Juniper Networks, Inc.
* All rights reserved.
*
* This software was developed by Robert N. M. Watson under contract
* to Juniper Networks, Inc.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
#include "opt_inet6.h"
#include "opt_pcbgroup.h"
#ifndef PCBGROUP
#error "options RSS depends on options PCBGROUP"
#endif
#include <sys/param.h>
#include <sys/mbuf.h>
#include <sys/socket.h>
#include <sys/priv.h>
#include <sys/kernel.h>
#include <sys/smp.h>
#include <sys/sysctl.h>
#include <sys/sbuf.h>
Several years after initial development, merge prototype support for linking NIC Receive Side Scaling (RSS) to the network stack's connection-group implementation. This prototype (and derived patches) are in use at Juniper and several other FreeBSD-using companies, so despite some reservations about its maturity, merge the patch to the base tree so that it can be iteratively refined in collaboration rather than maintained as a set of gradually diverging patch sets. (1) Merge a software implementation of the Toeplitz hash specified in RSS implemented by David Malone. This is used to allow suitable pcbgroup placement of connections before the first packet is received from the NIC. Software hashing is generally avoided, however, due to high cost of the hash on general-purpose CPUs. (2) In in_rss.c, maintain authoritative versions of RSS state intended to be pushed to each NIC, including keying material, hash algorithm/ configuration, and buckets. Provide software-facing interfaces to hash 2- and 4-tuples for IPv4 and IPv6 using both the RSS standardised Toeplitz and a 'naive' variation with a hash efficient in software but with poor distribution properties. Implement rss_m2cpuid()to be used by netisr and other load balancing code to look up the CPU on which an mbuf should be processed. (3) In the Ethernet link layer, allow netisr distribution using RSS as a source of policy as an alternative to source ordering; continue to default to direct dispatch (i.e., don't try and requeue packets for processing on the 'right' CPU if they arrive in a directly dispatchable context). (4) Allow RSS to control tuning of connection groups in order to align groups with RSS buckets. If a packet arrives on a protocol using connection groups, and contains a suitable hardware-generated hash, use that hash value to select the connection group for pcb lookup for both IPv4 and IPv6. If no hardware-generated Toeplitz hash is available, we fall back on regular PCB lookup risking contention rather than pay the cost of Toeplitz in software -- this is a less scalable but, at my last measurement, faster approach. As core counts go up, we may want to revise this strategy despite CPU overhead. Where device drivers suitably configure NICs, and connection groups / RSS are enabled, this should avoid both lock and line contention during connection lookup for TCP. This commit does not modify any device drivers to tune device RSS configuration to the global RSS configuration; patches are in circulation to do this for at least Chelsio T3 and Intel 1G/10G drivers. Currently, the KPI for device drivers is not particularly robust, nor aware of more advanced features such as runtime reconfiguration/rebalancing. This will hopefully prove a useful starting point for refinement. No MFC is scheduled as we will first want to nail down a more mature and maintainable KPI/KBI for device drivers. Sponsored by: Juniper Networks (original work) Sponsored by: EMC/Isilon (patch update and merge)
2014-03-15 00:57:50 +00:00
#include <net/if.h>
#include <net/if_var.h>
#include <net/netisr.h>
#include <net/rss_config.h>
Several years after initial development, merge prototype support for linking NIC Receive Side Scaling (RSS) to the network stack's connection-group implementation. This prototype (and derived patches) are in use at Juniper and several other FreeBSD-using companies, so despite some reservations about its maturity, merge the patch to the base tree so that it can be iteratively refined in collaboration rather than maintained as a set of gradually diverging patch sets. (1) Merge a software implementation of the Toeplitz hash specified in RSS implemented by David Malone. This is used to allow suitable pcbgroup placement of connections before the first packet is received from the NIC. Software hashing is generally avoided, however, due to high cost of the hash on general-purpose CPUs. (2) In in_rss.c, maintain authoritative versions of RSS state intended to be pushed to each NIC, including keying material, hash algorithm/ configuration, and buckets. Provide software-facing interfaces to hash 2- and 4-tuples for IPv4 and IPv6 using both the RSS standardised Toeplitz and a 'naive' variation with a hash efficient in software but with poor distribution properties. Implement rss_m2cpuid()to be used by netisr and other load balancing code to look up the CPU on which an mbuf should be processed. (3) In the Ethernet link layer, allow netisr distribution using RSS as a source of policy as an alternative to source ordering; continue to default to direct dispatch (i.e., don't try and requeue packets for processing on the 'right' CPU if they arrive in a directly dispatchable context). (4) Allow RSS to control tuning of connection groups in order to align groups with RSS buckets. If a packet arrives on a protocol using connection groups, and contains a suitable hardware-generated hash, use that hash value to select the connection group for pcb lookup for both IPv4 and IPv6. If no hardware-generated Toeplitz hash is available, we fall back on regular PCB lookup risking contention rather than pay the cost of Toeplitz in software -- this is a less scalable but, at my last measurement, faster approach. As core counts go up, we may want to revise this strategy despite CPU overhead. Where device drivers suitably configure NICs, and connection groups / RSS are enabled, this should avoid both lock and line contention during connection lookup for TCP. This commit does not modify any device drivers to tune device RSS configuration to the global RSS configuration; patches are in circulation to do this for at least Chelsio T3 and Intel 1G/10G drivers. Currently, the KPI for device drivers is not particularly robust, nor aware of more advanced features such as runtime reconfiguration/rebalancing. This will hopefully prove a useful starting point for refinement. No MFC is scheduled as we will first want to nail down a more mature and maintainable KPI/KBI for device drivers. Sponsored by: Juniper Networks (original work) Sponsored by: EMC/Isilon (patch update and merge)
2014-03-15 00:57:50 +00:00
#include <netinet/in.h>
#include <netinet/in_pcb.h>
#include <netinet/in_rss.h>
#include <netinet/in_var.h>
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
/* for software rss hash support */
#include <netinet/ip.h>
#include <netinet/tcp.h>
#include <netinet/udp.h>
Several years after initial development, merge prototype support for linking NIC Receive Side Scaling (RSS) to the network stack's connection-group implementation. This prototype (and derived patches) are in use at Juniper and several other FreeBSD-using companies, so despite some reservations about its maturity, merge the patch to the base tree so that it can be iteratively refined in collaboration rather than maintained as a set of gradually diverging patch sets. (1) Merge a software implementation of the Toeplitz hash specified in RSS implemented by David Malone. This is used to allow suitable pcbgroup placement of connections before the first packet is received from the NIC. Software hashing is generally avoided, however, due to high cost of the hash on general-purpose CPUs. (2) In in_rss.c, maintain authoritative versions of RSS state intended to be pushed to each NIC, including keying material, hash algorithm/ configuration, and buckets. Provide software-facing interfaces to hash 2- and 4-tuples for IPv4 and IPv6 using both the RSS standardised Toeplitz and a 'naive' variation with a hash efficient in software but with poor distribution properties. Implement rss_m2cpuid()to be used by netisr and other load balancing code to look up the CPU on which an mbuf should be processed. (3) In the Ethernet link layer, allow netisr distribution using RSS as a source of policy as an alternative to source ordering; continue to default to direct dispatch (i.e., don't try and requeue packets for processing on the 'right' CPU if they arrive in a directly dispatchable context). (4) Allow RSS to control tuning of connection groups in order to align groups with RSS buckets. If a packet arrives on a protocol using connection groups, and contains a suitable hardware-generated hash, use that hash value to select the connection group for pcb lookup for both IPv4 and IPv6. If no hardware-generated Toeplitz hash is available, we fall back on regular PCB lookup risking contention rather than pay the cost of Toeplitz in software -- this is a less scalable but, at my last measurement, faster approach. As core counts go up, we may want to revise this strategy despite CPU overhead. Where device drivers suitably configure NICs, and connection groups / RSS are enabled, this should avoid both lock and line contention during connection lookup for TCP. This commit does not modify any device drivers to tune device RSS configuration to the global RSS configuration; patches are in circulation to do this for at least Chelsio T3 and Intel 1G/10G drivers. Currently, the KPI for device drivers is not particularly robust, nor aware of more advanced features such as runtime reconfiguration/rebalancing. This will hopefully prove a useful starting point for refinement. No MFC is scheduled as we will first want to nail down a more mature and maintainable KPI/KBI for device drivers. Sponsored by: Juniper Networks (original work) Sponsored by: EMC/Isilon (patch update and merge)
2014-03-15 00:57:50 +00:00
/*
* Hash an IPv4 2-tuple.
*/
uint32_t
rss_hash_ip4_2tuple(struct in_addr src, struct in_addr dst)
{
uint8_t data[sizeof(src) + sizeof(dst)];
u_int datalen;
datalen = 0;
bcopy(&src, &data[datalen], sizeof(src));
datalen += sizeof(src);
bcopy(&dst, &data[datalen], sizeof(dst));
datalen += sizeof(dst);
return (rss_hash(datalen, data));
}
/*
* Hash an IPv4 4-tuple.
*/
uint32_t
rss_hash_ip4_4tuple(struct in_addr src, u_short srcport, struct in_addr dst,
u_short dstport)
{
uint8_t data[sizeof(src) + sizeof(dst) + sizeof(srcport) +
sizeof(dstport)];
u_int datalen;
datalen = 0;
bcopy(&src, &data[datalen], sizeof(src));
datalen += sizeof(src);
bcopy(&dst, &data[datalen], sizeof(dst));
datalen += sizeof(dst);
bcopy(&srcport, &data[datalen], sizeof(srcport));
datalen += sizeof(srcport);
bcopy(&dstport, &data[datalen], sizeof(dstport));
datalen += sizeof(dstport);
return (rss_hash(datalen, data));
}
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
/*
* Calculate an appropriate ipv4 2-tuple or 4-tuple given the given
* IPv4 source/destination address, UDP or TCP source/destination ports
* and the protocol type.
*
* The protocol code may wish to do a software hash of the given
* tuple. This depends upon the currently configured RSS hash types.
*
* This assumes that the packet in question isn't a fragment.
*
* It also assumes the packet source/destination address
* are in "incoming" packet order (ie, source is "far" address.)
*/
int
rss_proto_software_hash_v4(struct in_addr s, struct in_addr d,
u_short sp, u_short dp, int proto,
uint32_t *hashval, uint32_t *hashtype)
{
uint32_t hash;
/*
* Next, choose the hash type depending upon the protocol
* identifier.
*/
if ((proto == IPPROTO_TCP) &&
(rss_gethashconfig() & RSS_HASHTYPE_RSS_TCP_IPV4)) {
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
hash = rss_hash_ip4_4tuple(s, sp, d, dp);
*hashval = hash;
*hashtype = M_HASHTYPE_RSS_TCP_IPV4;
return (0);
} else if ((proto == IPPROTO_UDP) &&
(rss_gethashconfig() & RSS_HASHTYPE_RSS_UDP_IPV4)) {
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
hash = rss_hash_ip4_4tuple(s, sp, d, dp);
*hashval = hash;
*hashtype = M_HASHTYPE_RSS_UDP_IPV4;
return (0);
} else if (rss_gethashconfig() & RSS_HASHTYPE_RSS_IPV4) {
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
/* RSS doesn't hash on other protocols like SCTP; so 2-tuple */
hash = rss_hash_ip4_2tuple(s, d);
*hashval = hash;
*hashtype = M_HASHTYPE_RSS_IPV4;
return (0);
}
/* No configured available hashtypes! */
RSS_DEBUG("no available hashtypes!\n");
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
return (-1);
}
/*
* Do a software calculation of the RSS for the given mbuf.
*
* This is typically used by the input path to recalculate the RSS after
* some form of packet processing (eg de-capsulation, IP fragment reassembly.)
*
* dir is the packet direction - RSS_HASH_PKT_INGRESS for incoming and
* RSS_HASH_PKT_EGRESS for outgoing.
*
* Returns 0 if a hash was done, -1 if no hash was done, +1 if
* the mbuf already had a valid RSS flowid.
*
* This function doesn't modify the mbuf. It's up to the caller to
* assign flowid/flowtype as appropriate.
*/
int
rss_mbuf_software_hash_v4(const struct mbuf *m, int dir, uint32_t *hashval,
uint32_t *hashtype)
{
const struct ip *ip;
const struct tcphdr *th;
const struct udphdr *uh;
uint32_t flowid;
uint32_t flowtype;
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
uint8_t proto;
int iphlen;
int is_frag = 0;
/*
* XXX For now this only handles hashing on incoming mbufs.
*/
if (dir != RSS_HASH_PKT_INGRESS) {
RSS_DEBUG("called on EGRESS packet!\n");
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
return (-1);
}
/*
* First, validate that the mbuf we have is long enough
* to have an IPv4 header in it.
*/
if (m->m_pkthdr.len < (sizeof(struct ip))) {
RSS_DEBUG("short mbuf pkthdr\n");
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
return (-1);
}
if (m->m_len < (sizeof(struct ip))) {
RSS_DEBUG("short mbuf len\n");
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
return (-1);
}
/* Ok, let's dereference that */
ip = mtod(m, struct ip *);
proto = ip->ip_p;
iphlen = ip->ip_hl << 2;
/*
* If this is a fragment then it shouldn't be four-tuple
* hashed just yet. Once it's reassembled into a full
* frame it should be re-hashed.
*/
if (ip->ip_off & htons(IP_MF | IP_OFFMASK))
is_frag = 1;
/*
* If the mbuf flowid/flowtype matches the packet type,
* and we don't support the 4-tuple version of the given protocol,
* then signal to the owner that it can trust the flowid/flowtype
* details.
*
* This is a little picky - eg, if TCPv4 / UDPv4 hashing
* is supported but we got a TCP/UDP frame only 2-tuple hashed,
* then we shouldn't just "trust" the 2-tuple hash. We need
* a 4-tuple hash.
*/
flowid = m->m_pkthdr.flowid;
flowtype = M_HASHTYPE_GET(m);
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
if (flowtype != M_HASHTYPE_NONE) {
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
switch (proto) {
case IPPROTO_UDP:
if ((rss_gethashconfig() & RSS_HASHTYPE_RSS_UDP_IPV4) &&
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
(flowtype == M_HASHTYPE_RSS_UDP_IPV4) &&
(is_frag == 0)) {
return (1);
}
/*
* Only allow 2-tuple for UDP frames if we don't also
* support 4-tuple for UDP.
*/
if ((rss_gethashconfig() & RSS_HASHTYPE_RSS_IPV4) &&
((rss_gethashconfig() & RSS_HASHTYPE_RSS_UDP_IPV4) == 0) &&
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
flowtype == M_HASHTYPE_RSS_IPV4) {
return (1);
}
break;
case IPPROTO_TCP:
if ((rss_gethashconfig() & RSS_HASHTYPE_RSS_TCP_IPV4) &&
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
(flowtype == M_HASHTYPE_RSS_TCP_IPV4) &&
(is_frag == 0)) {
return (1);
}
/*
* Only allow 2-tuple for TCP frames if we don't also
* support 2-tuple for TCP.
*/
if ((rss_gethashconfig() & RSS_HASHTYPE_RSS_IPV4) &&
((rss_gethashconfig() & RSS_HASHTYPE_RSS_TCP_IPV4) == 0) &&
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
flowtype == M_HASHTYPE_RSS_IPV4) {
return (1);
}
break;
default:
if ((rss_gethashconfig() & RSS_HASHTYPE_RSS_IPV4) &&
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
flowtype == M_HASHTYPE_RSS_IPV4) {
return (1);
}
break;
}
}
/*
* Decode enough information to make a hash decision.
*
* XXX TODO: does the hardware hash on 4-tuple if IP
* options are present?
*/
if ((rss_gethashconfig() & RSS_HASHTYPE_RSS_TCP_IPV4) &&
(proto == IPPROTO_TCP) &&
(is_frag == 0)) {
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
if (m->m_len < iphlen + sizeof(struct tcphdr)) {
RSS_DEBUG("short TCP frame?\n");
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
return (-1);
}
th = (const struct tcphdr *)((c_caddr_t)ip + iphlen);
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
return rss_proto_software_hash_v4(ip->ip_src, ip->ip_dst,
th->th_sport,
th->th_dport,
proto,
hashval,
hashtype);
} else if ((rss_gethashconfig() & RSS_HASHTYPE_RSS_UDP_IPV4) &&
(proto == IPPROTO_UDP) &&
(is_frag == 0)) {
uh = (const struct udphdr *)((c_caddr_t)ip + iphlen);
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
if (m->m_len < iphlen + sizeof(struct udphdr)) {
RSS_DEBUG("short UDP frame?\n");
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
return (-1);
}
return rss_proto_software_hash_v4(ip->ip_src, ip->ip_dst,
uh->uh_sport,
uh->uh_dport,
proto,
hashval,
hashtype);
} else if (rss_gethashconfig() & RSS_HASHTYPE_RSS_IPV4) {
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
/* Default to 2-tuple hash */
return rss_proto_software_hash_v4(ip->ip_src, ip->ip_dst,
0, /* source port */
0, /* destination port */
0, /* IPPROTO_IP */
hashval,
hashtype);
} else {
RSS_DEBUG("no available hashtypes!\n");
return (-1);
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
}
}
/*
* Similar to rss_m2cpuid, but designed to be used by the IP NETISR
* on incoming frames.
*
* If an existing RSS hash exists and it matches what the configured
* hashing is, then use it.
*
* If there's an existing RSS hash but the desired hash is different,
* or if there's no useful RSS hash, then calculate it via
* the software path.
*
* XXX TODO: definitely want statistics here!
*/
struct mbuf *
rss_soft_m2cpuid_v4(struct mbuf *m, uintptr_t source, u_int *cpuid)
Implement IPv4 RSS software hash functions to use during packet ingress and egress. * rss_mbuf_software_hash_v4 - look at the IPv4 mbuf to fetch the IPv4 details + direction to calculate a hash. * rss_proto_software_hash_v4 - hash the given source/destination IPv4 address, port and direction. * rss_soft_m2cpuid - map the given mbuf to an RSS CPU ("bucket" for now) These functions are intended to be used by the stack to support the following: * Not all NICs do RSS hashing, so we should support some way of doing a hash in software; * The NIC / driver may not hash frames the way we want (eg UDP 4-tuple hashing when the stack is only doing 2-tuple hashing for UDP); so we may need to re-hash frames; * .. same with IPv4 fragments - they will need to be re-hashed after reassembly; * .. and same with things like IP tunneling and such; * The transmit path for things like UDP, RAW and ICMP don't currently have any RSS information attached to them - so they'll need an RSS calculation performed before transmit. TODO: * Counters! Everywhere! * Add a debug mode that software hashes received frames and compares them to the hardware hash provided by the hardware to ensure they match. The IPv6 part of this is missing - I'm going to do some re-juggling of where various parts of the RSS framework live before I add the IPv6 code (read: the IPv6 code is going to go into netinet6/in6_rss.[ch], rather than living here.) Note: This API is still fluid. Please keep that in mind. Differential Revision: https://reviews.freebsd.org/D527 Reviewed by: grehan
2014-09-09 03:10:21 +00:00
{
uint32_t hash_val, hash_type;
int ret;
M_ASSERTPKTHDR(m);
ret = rss_mbuf_software_hash_v4(m, RSS_HASH_PKT_INGRESS,
&hash_val, &hash_type);
if (ret > 0) {
/* mbuf has a valid hash already; don't need to modify it */
*cpuid = rss_hash2cpuid(m->m_pkthdr.flowid, M_HASHTYPE_GET(m));
} else if (ret == 0) {
/* hash was done; update */
m->m_pkthdr.flowid = hash_val;
M_HASHTYPE_SET(m, hash_type);
*cpuid = rss_hash2cpuid(m->m_pkthdr.flowid, M_HASHTYPE_GET(m));
} else { /* ret < 0 */
/* no hash was done */
*cpuid = NETISR_CPUID_NONE;
}
return (m);
}