2005-01-07 01:45:51 +00:00
|
|
|
/*-
|
1995-10-03 16:54:17 +00:00
|
|
|
* Copyright (c) 1982, 1986, 1988, 1990, 1993, 1994, 1995
|
1994-05-24 10:09:53 +00:00
|
|
|
* The Regents of the University of California. All rights reserved.
|
2010-11-12 06:41:55 +00:00
|
|
|
* Copyright (c) 2007-2008,2010
|
|
|
|
* Swinburne University of Technology, Melbourne, Australia.
|
|
|
|
* Copyright (c) 2009-2010 Lawrence Stewart <lstewart@freebsd.org>
|
|
|
|
* Copyright (c) 2010 The FreeBSD Foundation
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
* Copyright (c) 2010-2011 Juniper Networks, Inc.
|
2010-11-12 06:41:55 +00:00
|
|
|
* All rights reserved.
|
|
|
|
*
|
|
|
|
* Portions of this software were developed at the Centre for Advanced Internet
|
2011-04-12 08:13:18 +00:00
|
|
|
* Architectures, Swinburne University of Technology, by Lawrence Stewart,
|
|
|
|
* James Healy and David Hayes, made possible in part by a grant from the Cisco
|
|
|
|
* University Research Program Fund at Community Foundation Silicon Valley.
|
2010-11-12 06:41:55 +00:00
|
|
|
*
|
|
|
|
* Portions of this software were developed at the Centre for Advanced
|
|
|
|
* Internet Architectures, Swinburne University of Technology, Melbourne,
|
|
|
|
* Australia by David Hayes under sponsorship from the FreeBSD Foundation.
|
1994-05-24 10:09:53 +00:00
|
|
|
*
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
* Portions of this software were developed by Robert N. M. Watson under
|
|
|
|
* contract to Juniper Networks, Inc.
|
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
* are met:
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
|
|
|
* 4. Neither the name of the University nor the names of its contributors
|
|
|
|
* may be used to endorse or promote products derived from this software
|
|
|
|
* without specific prior written permission.
|
|
|
|
*
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
*
|
1995-10-03 16:54:17 +00:00
|
|
|
* @(#)tcp_input.c 8.12 (Berkeley) 5/24/95
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
|
|
|
|
2007-10-07 20:44:24 +00:00
|
|
|
#include <sys/cdefs.h>
|
|
|
|
__FBSDID("$FreeBSD$");
|
|
|
|
|
2007-03-21 18:56:03 +00:00
|
|
|
#include "opt_ipfw.h" /* for ipfw_fwd */
|
Initial import of RFC 2385 (TCP-MD5) digest support.
This is the first of two commits; bringing in the kernel support first.
This can be enabled by compiling a kernel with options TCP_SIGNATURE
and FAST_IPSEC.
For the uninitiated, this is a TCP option which provides for a means of
authenticating TCP sessions which came into being before IPSEC. It is
still relevant today, however, as it is used by many commercial router
vendors, particularly with BGP, and as such has become a requirement for
interconnect at many major Internet points of presence.
Several parts of the TCP and IP headers, including the segment payload,
are digested with MD5, including a shared secret. The PF_KEY interface
is used to manage the secrets using security associations in the SADB.
There is a limitation here in that as there is no way to map a TCP flow
per-port back to an SPI without polluting tcpcb or using the SPD; the
code to do the latter is unstable at this time. Therefore this code only
supports per-host keying granularity.
Whilst FAST_IPSEC is mutually exclusive with KAME IPSEC (and thus IPv6),
TCP_SIGNATURE applies only to IPv4. For the vast majority of prospective
users of this feature, this will not pose any problem.
This implementation is output-only; that is, the option is honoured when
responding to a host initiating a TCP session, but no effort is made
[yet] to authenticate inbound traffic. This is, however, sufficient to
interwork with Cisco equipment.
Tested with a Cisco 2501 running IOS 12.0(27), and Quagga 0.96.4 with
local patches. Patches for tcpdump to validate TCP-MD5 sessions are also
available from me upon request.
Sponsored by: sentex.net
2004-02-11 04:26:04 +00:00
|
|
|
#include "opt_inet.h"
|
2000-01-09 19:17:30 +00:00
|
|
|
#include "opt_inet6.h"
|
2000-01-15 14:56:38 +00:00
|
|
|
#include "opt_ipsec.h"
|
1997-09-16 18:36:06 +00:00
|
|
|
#include "opt_tcpdebug.h"
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/param.h>
|
1995-11-09 20:23:09 +00:00
|
|
|
#include <sys/kernel.h>
|
2010-12-28 12:13:30 +00:00
|
|
|
#include <sys/hhook.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/malloc.h>
|
|
|
|
#include <sys/mbuf.h>
|
1997-04-27 20:01:29 +00:00
|
|
|
#include <sys/proc.h> /* for proc0 declaration */
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/protosw.h>
|
2002-04-30 01:54:54 +00:00
|
|
|
#include <sys/signalvar.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/socket.h>
|
|
|
|
#include <sys/socketvar.h>
|
2002-04-30 01:54:54 +00:00
|
|
|
#include <sys/sysctl.h>
|
1996-04-04 10:46:44 +00:00
|
|
|
#include <sys/syslog.h>
|
2002-04-30 01:54:54 +00:00
|
|
|
#include <sys/systm.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
|
1995-10-03 16:54:17 +00:00
|
|
|
#include <machine/cpu.h> /* before tcp_seq.h, for tcp_random18() */
|
|
|
|
|
2004-02-24 15:27:41 +00:00
|
|
|
#include <vm/uma.h>
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <net/if.h>
|
|
|
|
#include <net/route.h>
|
2009-08-01 19:26:27 +00:00
|
|
|
#include <net/vnet.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2007-07-28 12:20:39 +00:00
|
|
|
#define TCPSTATES /* for logging */
|
|
|
|
|
2010-11-12 06:41:55 +00:00
|
|
|
#include <netinet/cc.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <netinet/in.h>
|
2002-04-30 01:54:54 +00:00
|
|
|
#include <netinet/in_pcb.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <netinet/in_systm.h>
|
2002-04-30 01:54:54 +00:00
|
|
|
#include <netinet/in_var.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <netinet/ip.h>
|
2006-02-18 17:05:00 +00:00
|
|
|
#include <netinet/ip_icmp.h> /* required for icmp_var.h */
|
|
|
|
#include <netinet/icmp_var.h> /* for ICMP_BANDLIM */
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <netinet/ip_var.h>
|
2005-11-18 20:12:40 +00:00
|
|
|
#include <netinet/ip_options.h>
|
2000-07-04 16:35:15 +00:00
|
|
|
#include <netinet/ip6.h>
|
|
|
|
#include <netinet/icmp6.h>
|
|
|
|
#include <netinet6/in6_pcb.h>
|
2002-04-30 01:54:54 +00:00
|
|
|
#include <netinet6/ip6_var.h>
|
|
|
|
#include <netinet6/nd6.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <netinet/tcp_fsm.h>
|
|
|
|
#include <netinet/tcp_seq.h>
|
|
|
|
#include <netinet/tcp_timer.h>
|
|
|
|
#include <netinet/tcp_var.h>
|
2000-01-09 19:17:30 +00:00
|
|
|
#include <netinet6/tcp6_var.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <netinet/tcpip.h>
|
2007-07-27 00:57:06 +00:00
|
|
|
#include <netinet/tcp_syncache.h>
|
1994-09-15 10:36:56 +00:00
|
|
|
#ifdef TCPDEBUG
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <netinet/tcp_debug.h>
|
2000-01-09 19:17:30 +00:00
|
|
|
#endif /* TCPDEBUG */
|
2012-06-19 07:34:13 +00:00
|
|
|
#ifdef TCP_OFFLOAD
|
|
|
|
#include <netinet/tcp_offload.h>
|
|
|
|
#endif
|
2000-01-09 19:17:30 +00:00
|
|
|
|
2007-07-03 12:13:45 +00:00
|
|
|
#ifdef IPSEC
|
2002-10-16 02:25:05 +00:00
|
|
|
#include <netipsec/ipsec.h>
|
|
|
|
#include <netipsec/ipsec6.h>
|
2007-07-03 12:13:45 +00:00
|
|
|
#endif /*IPSEC*/
|
2002-10-16 02:25:05 +00:00
|
|
|
|
2000-03-27 19:14:27 +00:00
|
|
|
#include <machine/in_cksum.h>
|
|
|
|
|
2006-10-22 11:52:19 +00:00
|
|
|
#include <security/mac/mac_framework.h>
|
|
|
|
|
2010-11-12 06:41:55 +00:00
|
|
|
const int tcprexmtthresh = 3;
|
1995-11-14 20:34:56 +00:00
|
|
|
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
VNET_DEFINE(struct tcpstat, tcpstat);
|
|
|
|
SYSCTL_VNET_STRUCT(_net_inet_tcp, TCPCTL_STATS, stats, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcpstat), tcpstat,
|
Step 1.5 of importing the network stack virtualization infrastructure
from the vimage project, as per plan established at devsummit 08/08:
http://wiki.freebsd.org/Image/Notes200808DevSummit
Introduce INIT_VNET_*() initializer macros, VNET_FOREACH() iterator
macros, and CURVNET_SET() context setting macros, all currently
resolving to NOPs.
Prepare for virtualization of selected SYSCTL objects by introducing a
family of SYSCTL_V_*() macros, currently resolving to their global
counterparts, i.e. SYSCTL_V_INT() == SYSCTL_INT().
Move selected #defines from sys/sys/vimage.h to newly introduced header
files specific to virtualized subsystems (sys/net/vnet.h,
sys/netinet/vinet.h etc.).
All the changes are verified to have zero functional impact at this
point in time by doing MD5 comparision between pre- and post-change
object files(*).
(*) netipsec/keysock.c did not validate depending on compile time options.
Implemented by: julian, bz, brooks, zec
Reviewed by: julian, bz, brooks, kris, rwatson, ...
Approved by: julian (mentor)
Obtained from: //depot/projects/vimage-commit2/...
X-MFC after: never
Sponsored by: NLnet Foundation, The FreeBSD Foundation
2008-10-02 15:37:58 +00:00
|
|
|
"TCP statistics (struct tcpstat, netinet/tcp_var.h)");
|
1995-11-14 20:34:56 +00:00
|
|
|
|
2007-07-28 12:20:39 +00:00
|
|
|
int tcp_log_in_vain = 0;
|
2004-08-16 18:32:07 +00:00
|
|
|
SYSCTL_INT(_net_inet_tcp, OID_AUTO, log_in_vain, CTLFLAG_RW,
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
&tcp_log_in_vain, 0,
|
|
|
|
"Log all incoming TCP segments to closed ports");
|
1996-04-04 10:46:44 +00:00
|
|
|
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(int, blackhole) = 0;
|
|
|
|
#define V_blackhole VNET(blackhole)
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, blackhole, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(blackhole), 0,
|
|
|
|
"Do not send RST on segments to closed ports");
|
1999-08-17 12:17:53 +00:00
|
|
|
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(int, tcp_delack_enabled) = 1;
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, delayed_ack, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcp_delack_enabled), 0,
|
1999-05-03 23:57:32 +00:00
|
|
|
"Delay ACK to try and piggyback it onto a data packet");
|
1998-02-26 05:25:39 +00:00
|
|
|
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(int, drop_synfin) = 0;
|
|
|
|
#define V_drop_synfin VNET(drop_synfin)
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, drop_synfin, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(drop_synfin), 0,
|
|
|
|
"Drop TCP packets with SYN+FIN set");
|
1999-09-14 16:14:05 +00:00
|
|
|
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(int, tcp_do_rfc3042) = 1;
|
|
|
|
#define V_tcp_do_rfc3042 VNET(tcp_do_rfc3042)
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, rfc3042, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcp_do_rfc3042), 0,
|
|
|
|
"Enable RFC 3042 (Limited Transmit)");
|
2003-03-12 20:27:28 +00:00
|
|
|
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(int, tcp_do_rfc3390) = 1;
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, rfc3390, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcp_do_rfc3390), 0,
|
2003-03-13 01:43:45 +00:00
|
|
|
"Enable RFC 3390 (Increasing TCP's Initial Congestion Window)");
|
|
|
|
|
2012-10-28 19:47:46 +00:00
|
|
|
SYSCTL_NODE(_net_inet_tcp, OID_AUTO, experimental, CTLFLAG_RW, 0,
|
|
|
|
"Experimental TCP extensions");
|
|
|
|
|
|
|
|
VNET_DEFINE(int, tcp_do_initcwnd10) = 1;
|
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp_experimental, OID_AUTO, initcwnd10, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcp_do_initcwnd10), 0,
|
|
|
|
"Enable draft-ietf-tcpm-initcwnd-05 (Increasing initial CWND to 10)");
|
|
|
|
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(int, tcp_do_rfc3465) = 1;
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, rfc3465, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcp_do_rfc3465), 0,
|
2009-01-15 06:44:22 +00:00
|
|
|
"Enable RFC 3465 (Appropriate Byte Counting)");
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(int, tcp_abc_l_var) = 2;
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, abc_l_var, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcp_abc_l_var), 2,
|
2009-01-15 06:44:22 +00:00
|
|
|
"Cap the max cwnd increment during slow-start to this number of segments");
|
|
|
|
|
2011-11-07 15:43:11 +00:00
|
|
|
static SYSCTL_NODE(_net_inet_tcp, OID_AUTO, ecn, CTLFLAG_RW, 0, "TCP ECN");
|
2008-07-31 15:10:09 +00:00
|
|
|
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(int, tcp_do_ecn) = 0;
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp_ecn, OID_AUTO, enable, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcp_do_ecn), 0,
|
|
|
|
"TCP ECN support");
|
|
|
|
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(int, tcp_ecn_maxretries) = 1;
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp_ecn, OID_AUTO, maxretries, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcp_ecn_maxretries), 0,
|
|
|
|
"Max retries before giving up on ECN");
|
|
|
|
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(int, tcp_insecure_rst) = 0;
|
|
|
|
#define V_tcp_insecure_rst VNET(tcp_insecure_rst)
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, insecure_rst, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcp_insecure_rst), 0,
|
2007-03-19 19:00:51 +00:00
|
|
|
"Follow the old (insecure) criteria for accepting RST packets");
|
2005-01-03 07:08:37 +00:00
|
|
|
|
2011-10-17 00:05:31 +00:00
|
|
|
VNET_DEFINE(int, tcp_recvspace) = 1024*64;
|
2011-10-16 20:18:39 +00:00
|
|
|
#define V_tcp_recvspace VNET(tcp_recvspace)
|
2011-11-02 20:58:47 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp, TCPCTL_RECVSPACE, recvspace, CTLFLAG_RW,
|
2011-10-16 20:18:39 +00:00
|
|
|
&VNET_NAME(tcp_recvspace), 0, "Initial receive socket buffer size");
|
|
|
|
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(int, tcp_do_autorcvbuf) = 1;
|
|
|
|
#define V_tcp_do_autorcvbuf VNET(tcp_do_autorcvbuf)
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, recvbuf_auto, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcp_do_autorcvbuf), 0,
|
Step 1.5 of importing the network stack virtualization infrastructure
from the vimage project, as per plan established at devsummit 08/08:
http://wiki.freebsd.org/Image/Notes200808DevSummit
Introduce INIT_VNET_*() initializer macros, VNET_FOREACH() iterator
macros, and CURVNET_SET() context setting macros, all currently
resolving to NOPs.
Prepare for virtualization of selected SYSCTL objects by introducing a
family of SYSCTL_V_*() macros, currently resolving to their global
counterparts, i.e. SYSCTL_V_INT() == SYSCTL_INT().
Move selected #defines from sys/sys/vimage.h to newly introduced header
files specific to virtualized subsystems (sys/net/vnet.h,
sys/netinet/vinet.h etc.).
All the changes are verified to have zero functional impact at this
point in time by doing MD5 comparision between pre- and post-change
object files(*).
(*) netipsec/keysock.c did not validate depending on compile time options.
Implemented by: julian, bz, brooks, zec
Reviewed by: julian, bz, brooks, kris, rwatson, ...
Approved by: julian (mentor)
Obtained from: //depot/projects/vimage-commit2/...
X-MFC after: never
Sponsored by: NLnet Foundation, The FreeBSD Foundation
2008-10-02 15:37:58 +00:00
|
|
|
"Enable automatic receive buffer sizing");
|
2007-02-01 18:32:13 +00:00
|
|
|
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(int, tcp_autorcvbuf_inc) = 16*1024;
|
|
|
|
#define V_tcp_autorcvbuf_inc VNET(tcp_autorcvbuf_inc)
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, recvbuf_inc, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcp_autorcvbuf_inc), 0,
|
2007-03-19 19:00:51 +00:00
|
|
|
"Incrementor step size of automatic receive buffer");
|
2007-02-01 18:32:13 +00:00
|
|
|
|
2011-08-25 09:20:13 +00:00
|
|
|
VNET_DEFINE(int, tcp_autorcvbuf_max) = 2*1024*1024;
|
2010-04-29 11:52:42 +00:00
|
|
|
#define V_tcp_autorcvbuf_max VNET(tcp_autorcvbuf_max)
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, recvbuf_max, CTLFLAG_RW,
|
|
|
|
&VNET_NAME(tcp_autorcvbuf_max), 0,
|
Step 1.5 of importing the network stack virtualization infrastructure
from the vimage project, as per plan established at devsummit 08/08:
http://wiki.freebsd.org/Image/Notes200808DevSummit
Introduce INIT_VNET_*() initializer macros, VNET_FOREACH() iterator
macros, and CURVNET_SET() context setting macros, all currently
resolving to NOPs.
Prepare for virtualization of selected SYSCTL objects by introducing a
family of SYSCTL_V_*() macros, currently resolving to their global
counterparts, i.e. SYSCTL_V_INT() == SYSCTL_INT().
Move selected #defines from sys/sys/vimage.h to newly introduced header
files specific to virtualized subsystems (sys/net/vnet.h,
sys/netinet/vinet.h etc.).
All the changes are verified to have zero functional impact at this
point in time by doing MD5 comparision between pre- and post-change
object files(*).
(*) netipsec/keysock.c did not validate depending on compile time options.
Implemented by: julian, bz, brooks, zec
Reviewed by: julian, bz, brooks, kris, rwatson, ...
Approved by: julian (mentor)
Obtained from: //depot/projects/vimage-commit2/...
X-MFC after: never
Sponsored by: NLnet Foundation, The FreeBSD Foundation
2008-10-02 15:37:58 +00:00
|
|
|
"Max size of automatic receive buffer");
|
2007-02-01 18:32:13 +00:00
|
|
|
|
Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator. Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...). This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.
Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack. Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory. Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.
Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy. Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address. When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.
This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.
Bump __FreeBSD_version and update UPDATING.
Portions submitted by: bz
Reviewed by: bz, zec
Discussed with: gnn, jamie, jeff, jhb, julian, sam
Suggested by: peter
Approved by: re (kensmith)
2009-07-14 22:48:30 +00:00
|
|
|
VNET_DEFINE(struct inpcbhead, tcb);
|
2008-11-19 09:39:34 +00:00
|
|
|
#define tcb6 tcb /* for KAME src sync over BSD*'s */
|
2010-04-29 11:52:42 +00:00
|
|
|
VNET_DEFINE(struct inpcbinfo, tcbinfo);
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2005-06-27 22:27:42 +00:00
|
|
|
static void tcp_dooptions(struct tcpopt *, u_char *, int, int);
|
2007-05-06 15:16:05 +00:00
|
|
|
static void tcp_do_segment(struct mbuf *, struct tcphdr *,
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
struct socket *, struct tcpcb *, int, int, uint8_t,
|
|
|
|
int);
|
2007-03-23 20:16:50 +00:00
|
|
|
static void tcp_dropwithreset(struct mbuf *, struct tcphdr *,
|
|
|
|
struct tcpcb *, int, int);
|
2002-03-19 21:25:46 +00:00
|
|
|
static void tcp_pulloutofband(struct socket *,
|
|
|
|
struct tcphdr *, struct mbuf *, int);
|
|
|
|
static void tcp_xmit_timer(struct tcpcb *, int);
|
2002-08-17 02:05:25 +00:00
|
|
|
static void tcp_newreno_partial_ack(struct tcpcb *, struct tcphdr *);
|
2011-04-25 17:13:40 +00:00
|
|
|
static void inline tcp_fields_to_host(struct tcphdr *);
|
|
|
|
#ifdef TCP_SIGNATURE
|
|
|
|
static void inline tcp_fields_to_net(struct tcphdr *);
|
|
|
|
static int inline tcp_signature_verify_input(struct mbuf *, int, int,
|
|
|
|
int, struct tcpopt *, struct tcphdr *, u_int);
|
|
|
|
#endif
|
2010-11-12 06:41:55 +00:00
|
|
|
static void inline cc_ack_received(struct tcpcb *tp, struct tcphdr *th,
|
|
|
|
uint16_t type);
|
|
|
|
static void inline cc_conn_init(struct tcpcb *tp);
|
|
|
|
static void inline cc_post_recovery(struct tcpcb *tp, struct tcphdr *th);
|
2010-12-28 12:13:30 +00:00
|
|
|
static void inline hhook_run_tcp_est_in(struct tcpcb *tp,
|
|
|
|
struct tcphdr *th, struct tcpopt *to);
|
2008-07-31 15:10:09 +00:00
|
|
|
|
2009-08-02 19:43:32 +00:00
|
|
|
/*
|
|
|
|
* Kernel module interface for updating tcpstat. The argument is an index
|
|
|
|
* into tcpstat treated as an array of u_long. While this encodes the
|
|
|
|
* general layout of tcpstat into the caller, it doesn't encode its location,
|
|
|
|
* so that future changes to add, for example, per-CPU stats support won't
|
|
|
|
* cause binary compatibility problems for kernel modules.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
kmod_tcpstat_inc(int statnum)
|
|
|
|
{
|
|
|
|
|
|
|
|
(*((u_long *)&V_tcpstat + statnum))++;
|
|
|
|
}
|
|
|
|
|
2010-12-28 12:13:30 +00:00
|
|
|
/*
|
|
|
|
* Wrapper for the TCP established input helper hook.
|
|
|
|
*/
|
|
|
|
static void inline
|
|
|
|
hhook_run_tcp_est_in(struct tcpcb *tp, struct tcphdr *th, struct tcpopt *to)
|
|
|
|
{
|
|
|
|
struct tcp_hhook_data hhook_data;
|
|
|
|
|
|
|
|
if (V_tcp_hhh[HHOOK_TCP_EST_IN]->hhh_nhooks > 0) {
|
|
|
|
hhook_data.tp = tp;
|
|
|
|
hhook_data.th = th;
|
|
|
|
hhook_data.to = to;
|
|
|
|
|
|
|
|
hhook_run_hooks(V_tcp_hhh[HHOOK_TCP_EST_IN], &hhook_data,
|
|
|
|
tp->osd);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2010-11-12 06:41:55 +00:00
|
|
|
/*
|
|
|
|
* CC wrapper hook functions
|
|
|
|
*/
|
2008-07-31 15:10:09 +00:00
|
|
|
static void inline
|
2010-11-12 06:41:55 +00:00
|
|
|
cc_ack_received(struct tcpcb *tp, struct tcphdr *th, uint16_t type)
|
2008-07-31 15:10:09 +00:00
|
|
|
{
|
2010-11-12 06:41:55 +00:00
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
|
|
|
|
|
|
|
tp->ccv->bytes_this_ack = BYTES_THIS_ACK(tp, th);
|
|
|
|
if (tp->snd_cwnd == min(tp->snd_cwnd, tp->snd_wnd))
|
|
|
|
tp->ccv->flags |= CCF_CWND_LIMITED;
|
|
|
|
else
|
|
|
|
tp->ccv->flags &= ~CCF_CWND_LIMITED;
|
|
|
|
|
|
|
|
if (type == CC_ACK) {
|
|
|
|
if (tp->snd_cwnd > tp->snd_ssthresh) {
|
|
|
|
tp->t_bytes_acked += min(tp->ccv->bytes_this_ack,
|
|
|
|
V_tcp_abc_l_var * tp->t_maxseg);
|
|
|
|
if (tp->t_bytes_acked >= tp->snd_cwnd) {
|
|
|
|
tp->t_bytes_acked -= tp->snd_cwnd;
|
|
|
|
tp->ccv->flags |= CCF_ABC_SENTAWND;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
tp->ccv->flags &= ~CCF_ABC_SENTAWND;
|
|
|
|
tp->t_bytes_acked = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (CC_ALGO(tp)->ack_received != NULL) {
|
|
|
|
/* XXXLAS: Find a way to live without this */
|
|
|
|
tp->ccv->curack = th->th_ack;
|
|
|
|
CC_ALGO(tp)->ack_received(tp->ccv, type);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void inline
|
|
|
|
cc_conn_init(struct tcpcb *tp)
|
|
|
|
{
|
|
|
|
struct hc_metrics_lite metrics;
|
|
|
|
struct inpcb *inp = tp->t_inpcb;
|
|
|
|
int rtt;
|
|
|
|
|
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
|
|
|
|
|
|
|
tcp_hc_get(&inp->inp_inc, &metrics);
|
|
|
|
|
|
|
|
if (tp->t_srtt == 0 && (rtt = metrics.rmx_rtt)) {
|
|
|
|
tp->t_srtt = rtt;
|
|
|
|
tp->t_rttbest = tp->t_srtt + TCP_RTT_SCALE;
|
|
|
|
TCPSTAT_INC(tcps_usedrtt);
|
|
|
|
if (metrics.rmx_rttvar) {
|
|
|
|
tp->t_rttvar = metrics.rmx_rttvar;
|
|
|
|
TCPSTAT_INC(tcps_usedrttvar);
|
|
|
|
} else {
|
|
|
|
/* default variation is +- 1 rtt */
|
|
|
|
tp->t_rttvar =
|
|
|
|
tp->t_srtt * TCP_RTTVAR_SCALE / TCP_RTT_SCALE;
|
|
|
|
}
|
|
|
|
TCPT_RANGESET(tp->t_rxtcur,
|
|
|
|
((tp->t_srtt >> 2) + tp->t_rttvar) >> 1,
|
|
|
|
tp->t_rttmin, TCPTV_REXMTMAX);
|
|
|
|
}
|
|
|
|
if (metrics.rmx_ssthresh) {
|
|
|
|
/*
|
|
|
|
* There's some sort of gateway or interface
|
|
|
|
* buffer limit on the path. Use this to set
|
|
|
|
* the slow start threshhold, but set the
|
|
|
|
* threshold to no less than 2*mss.
|
|
|
|
*/
|
|
|
|
tp->snd_ssthresh = max(2 * tp->t_maxseg, metrics.rmx_ssthresh);
|
|
|
|
TCPSTAT_INC(tcps_usedssthresh);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2011-10-16 20:06:44 +00:00
|
|
|
* Set the initial slow-start flight size.
|
2010-11-12 06:41:55 +00:00
|
|
|
*
|
2012-10-28 17:25:08 +00:00
|
|
|
* RFC5681 Section 3.1 specifies the default conservative values.
|
|
|
|
* RFC3390 specifies slightly more aggressive values.
|
2012-10-28 19:47:46 +00:00
|
|
|
* Draft-ietf-tcpm-initcwnd-05 increases it to ten segments.
|
2012-10-28 17:25:08 +00:00
|
|
|
*
|
|
|
|
* If a SYN or SYN/ACK was lost and retransmitted, we have to
|
|
|
|
* reduce the initial CWND to one segment as congestion is likely
|
|
|
|
* requiring us to be cautious.
|
2010-11-12 06:41:55 +00:00
|
|
|
*/
|
2012-10-28 17:25:08 +00:00
|
|
|
if (tp->snd_cwnd == 1)
|
|
|
|
tp->snd_cwnd = tp->t_maxseg; /* SYN(-ACK) lost */
|
2012-10-28 19:47:46 +00:00
|
|
|
else if (V_tcp_do_initcwnd10)
|
|
|
|
tp->snd_cwnd = min(10 * tp->t_maxseg,
|
|
|
|
max(2 * tp->t_maxseg, 14600));
|
2012-10-28 17:25:08 +00:00
|
|
|
else if (V_tcp_do_rfc3390)
|
2010-11-12 06:41:55 +00:00
|
|
|
tp->snd_cwnd = min(4 * tp->t_maxseg,
|
|
|
|
max(2 * tp->t_maxseg, 4380));
|
2012-10-28 17:16:09 +00:00
|
|
|
else {
|
|
|
|
/* Per RFC5681 Section 3.1 */
|
|
|
|
if (tp->t_maxseg > 2190)
|
|
|
|
tp->snd_cwnd = 2 * tp->t_maxseg;
|
|
|
|
else if (tp->t_maxseg > 1095)
|
|
|
|
tp->snd_cwnd = 3 * tp->t_maxseg;
|
|
|
|
else
|
|
|
|
tp->snd_cwnd = 4 * tp->t_maxseg;
|
|
|
|
}
|
2010-11-12 06:41:55 +00:00
|
|
|
|
|
|
|
if (CC_ALGO(tp)->conn_init != NULL)
|
|
|
|
CC_ALGO(tp)->conn_init(tp->ccv);
|
|
|
|
}
|
|
|
|
|
|
|
|
void inline
|
|
|
|
cc_cong_signal(struct tcpcb *tp, struct tcphdr *th, uint32_t type)
|
|
|
|
{
|
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
|
|
|
|
|
|
|
switch(type) {
|
|
|
|
case CC_NDUPACK:
|
|
|
|
if (!IN_FASTRECOVERY(tp->t_flags)) {
|
|
|
|
tp->snd_recover = tp->snd_max;
|
|
|
|
if (tp->t_flags & TF_ECN_PERMIT)
|
|
|
|
tp->t_flags |= TF_ECN_SND_CWR;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case CC_ECN:
|
|
|
|
if (!IN_CONGRECOVERY(tp->t_flags)) {
|
|
|
|
TCPSTAT_INC(tcps_ecn_rcwnd);
|
|
|
|
tp->snd_recover = tp->snd_max;
|
|
|
|
if (tp->t_flags & TF_ECN_PERMIT)
|
|
|
|
tp->t_flags |= TF_ECN_SND_CWR;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case CC_RTO:
|
|
|
|
tp->t_dupacks = 0;
|
|
|
|
tp->t_bytes_acked = 0;
|
|
|
|
EXIT_RECOVERY(tp->t_flags);
|
2010-12-02 01:01:37 +00:00
|
|
|
tp->snd_ssthresh = max(2, min(tp->snd_wnd, tp->snd_cwnd) / 2 /
|
|
|
|
tp->t_maxseg) * tp->t_maxseg;
|
2010-11-12 06:41:55 +00:00
|
|
|
tp->snd_cwnd = tp->t_maxseg;
|
|
|
|
break;
|
|
|
|
case CC_RTO_ERR:
|
|
|
|
TCPSTAT_INC(tcps_sndrexmitbad);
|
|
|
|
/* RTO was unnecessary, so reset everything. */
|
|
|
|
tp->snd_cwnd = tp->snd_cwnd_prev;
|
|
|
|
tp->snd_ssthresh = tp->snd_ssthresh_prev;
|
|
|
|
tp->snd_recover = tp->snd_recover_prev;
|
|
|
|
if (tp->t_flags & TF_WASFRECOVERY)
|
|
|
|
ENTER_FASTRECOVERY(tp->t_flags);
|
|
|
|
if (tp->t_flags & TF_WASCRECOVERY)
|
|
|
|
ENTER_CONGRECOVERY(tp->t_flags);
|
|
|
|
tp->snd_nxt = tp->snd_max;
|
2011-04-29 15:40:12 +00:00
|
|
|
tp->t_flags &= ~TF_PREVVALID;
|
2010-11-12 06:41:55 +00:00
|
|
|
tp->t_badrxtwin = 0;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (CC_ALGO(tp)->cong_signal != NULL) {
|
|
|
|
if (th != NULL)
|
|
|
|
tp->ccv->curack = th->th_ack;
|
|
|
|
CC_ALGO(tp)->cong_signal(tp->ccv, type);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void inline
|
|
|
|
cc_post_recovery(struct tcpcb *tp, struct tcphdr *th)
|
|
|
|
{
|
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
|
|
|
|
|
|
|
/* XXXLAS: KASSERT that we're in recovery? */
|
|
|
|
|
|
|
|
if (CC_ALGO(tp)->post_recovery != NULL) {
|
|
|
|
tp->ccv->curack = th->th_ack;
|
|
|
|
CC_ALGO(tp)->post_recovery(tp->ccv);
|
|
|
|
}
|
|
|
|
/* XXXLAS: EXIT_RECOVERY ? */
|
|
|
|
tp->t_bytes_acked = 0;
|
2008-07-31 15:10:09 +00:00
|
|
|
}
|
1995-11-14 20:34:56 +00:00
|
|
|
|
2011-04-25 17:13:40 +00:00
|
|
|
static inline void
|
|
|
|
tcp_fields_to_host(struct tcphdr *th)
|
|
|
|
{
|
|
|
|
|
|
|
|
th->th_seq = ntohl(th->th_seq);
|
|
|
|
th->th_ack = ntohl(th->th_ack);
|
|
|
|
th->th_win = ntohs(th->th_win);
|
|
|
|
th->th_urp = ntohs(th->th_urp);
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef TCP_SIGNATURE
|
|
|
|
static inline void
|
|
|
|
tcp_fields_to_net(struct tcphdr *th)
|
|
|
|
{
|
|
|
|
|
|
|
|
th->th_seq = htonl(th->th_seq);
|
|
|
|
th->th_ack = htonl(th->th_ack);
|
|
|
|
th->th_win = htons(th->th_win);
|
|
|
|
th->th_urp = htons(th->th_urp);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline int
|
|
|
|
tcp_signature_verify_input(struct mbuf *m, int off0, int tlen, int optlen,
|
|
|
|
struct tcpopt *to, struct tcphdr *th, u_int tcpbflag)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
tcp_fields_to_net(th);
|
|
|
|
ret = tcp_signature_verify(m, off0, tlen, optlen, to, th, tcpbflag);
|
|
|
|
tcp_fields_to_host(th);
|
|
|
|
return (ret);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2000-01-09 19:17:30 +00:00
|
|
|
/* Neighbor Discovery, Neighbor Unreachability Detection Upper layer hint. */
|
|
|
|
#ifdef INET6
|
|
|
|
#define ND6_HINT(tp) \
|
|
|
|
do { \
|
|
|
|
if ((tp) && (tp)->t_inpcb && \
|
2003-11-20 20:07:39 +00:00
|
|
|
((tp)->t_inpcb->inp_vflag & INP_IPV6) != 0) \
|
|
|
|
nd6_nud_hint(NULL, NULL, 0); \
|
2000-01-09 19:17:30 +00:00
|
|
|
} while (0)
|
|
|
|
#else
|
|
|
|
#define ND6_HINT(tp)
|
|
|
|
#endif
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2001-02-25 15:17:24 +00:00
|
|
|
/*
|
2001-12-02 08:49:29 +00:00
|
|
|
* Indicate whether this ack should be delayed. We can delay the ack if
|
2001-12-13 04:02:09 +00:00
|
|
|
* - there is no delayed ack timer in progress and
|
|
|
|
* - our last ack wasn't a 0-sized window. We never want to delay
|
2003-02-19 21:18:23 +00:00
|
|
|
* the ack that opens up a 0-sized window and
|
|
|
|
* - delayed acks are enabled or
|
|
|
|
* - this is a half-synchronized T/TCP connection.
|
2001-02-25 15:17:24 +00:00
|
|
|
*/
|
2003-02-19 21:18:23 +00:00
|
|
|
#define DELAY_ACK(tp) \
|
2007-04-11 09:45:16 +00:00
|
|
|
((!tcp_timer_active(tp, TT_DELACK) && \
|
2003-02-22 21:54:57 +00:00
|
|
|
(tp->t_flags & TF_RXWIN0SENT) == 0) && \
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
(V_tcp_delack_enabled || (tp->t_flags & TF_NEEDSYN)))
|
2001-02-25 15:17:24 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
2007-05-28 23:27:44 +00:00
|
|
|
* TCP input handling is split into multiple parts:
|
|
|
|
* tcp6_input is a thin wrapper around tcp_input for the extended
|
|
|
|
* ip6_protox[] call format in ip6_input
|
|
|
|
* tcp_input handles primary segment validation, inpcb lookup and
|
|
|
|
* SYN processing on listen sockets
|
|
|
|
* tcp_do_segment processes the ACK and text of the segment for
|
|
|
|
* establishing, established and closing connections
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
#ifdef INET6
|
|
|
|
int
|
2007-03-21 19:37:55 +00:00
|
|
|
tcp6_input(struct mbuf **mp, int *offp, int proto)
|
2000-01-09 19:17:30 +00:00
|
|
|
{
|
2007-03-21 19:37:55 +00:00
|
|
|
struct mbuf *m = *mp;
|
2001-06-11 12:39:29 +00:00
|
|
|
struct in6_ifaddr *ia6;
|
2000-01-09 19:17:30 +00:00
|
|
|
|
|
|
|
IP6_EXTHDR_CHECK(m, *offp, sizeof(struct tcphdr), IPPROTO_DONE);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* draft-itojun-ipv6-tcp-to-anycast
|
|
|
|
* better place to put this in?
|
|
|
|
*/
|
2001-06-11 12:39:29 +00:00
|
|
|
ia6 = ip6_getdstifaddr(m);
|
2004-08-16 18:32:07 +00:00
|
|
|
if (ia6 && (ia6->ia6_flags & IN6_IFF_ANYCAST)) {
|
2000-01-09 19:17:30 +00:00
|
|
|
struct ip6_hdr *ip6;
|
|
|
|
|
2009-06-23 20:19:09 +00:00
|
|
|
ifa_free(&ia6->ia_ifa);
|
2000-01-09 19:17:30 +00:00
|
|
|
ip6 = mtod(m, struct ip6_hdr *);
|
|
|
|
icmp6_error(m, ICMP6_DST_UNREACH, ICMP6_DST_UNREACH_ADDR,
|
|
|
|
(caddr_t)&ip6->ip6_dst - (caddr_t)ip6);
|
|
|
|
return IPPROTO_DONE;
|
|
|
|
}
|
2012-06-04 18:43:51 +00:00
|
|
|
if (ia6)
|
|
|
|
ifa_free(&ia6->ia_ifa);
|
2000-01-09 19:17:30 +00:00
|
|
|
|
2001-09-03 20:03:55 +00:00
|
|
|
tcp_input(m, *offp);
|
2000-01-09 19:17:30 +00:00
|
|
|
return IPPROTO_DONE;
|
|
|
|
}
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif /* INET6 */
|
2000-01-09 19:17:30 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
void
|
2007-03-21 19:37:55 +00:00
|
|
|
tcp_input(struct mbuf *m, int off0)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2011-04-30 11:21:29 +00:00
|
|
|
struct tcphdr *th = NULL;
|
2007-03-21 19:37:55 +00:00
|
|
|
struct ip *ip = NULL;
|
|
|
|
struct inpcb *inp = NULL;
|
2007-03-23 20:16:50 +00:00
|
|
|
struct tcpcb *tp = NULL;
|
|
|
|
struct socket *so = NULL;
|
1995-10-03 16:54:17 +00:00
|
|
|
u_char *optp = NULL;
|
1994-05-25 09:21:21 +00:00
|
|
|
int optlen = 0;
|
2011-04-30 11:21:29 +00:00
|
|
|
#ifdef INET
|
|
|
|
int len;
|
|
|
|
#endif
|
|
|
|
int tlen = 0, off;
|
2000-01-09 19:17:30 +00:00
|
|
|
int drop_hdrlen;
|
2007-03-21 19:37:55 +00:00
|
|
|
int thflags;
|
2007-03-23 20:16:50 +00:00
|
|
|
int rstreason = 0; /* For badport_bandlim accounting purposes */
|
2011-04-25 17:13:40 +00:00
|
|
|
#ifdef TCP_SIGNATURE
|
|
|
|
uint8_t sig_checked = 0;
|
|
|
|
#endif
|
2011-04-30 11:21:29 +00:00
|
|
|
uint8_t iptos = 0;
|
2012-10-25 09:39:14 +00:00
|
|
|
struct m_tag *fwd_tag = NULL;
|
2002-08-17 02:05:25 +00:00
|
|
|
#ifdef INET6
|
2007-03-23 20:16:50 +00:00
|
|
|
struct ip6_hdr *ip6 = NULL;
|
2002-08-17 02:05:25 +00:00
|
|
|
int isipv6;
|
|
|
|
#else
|
2007-05-28 11:03:53 +00:00
|
|
|
const void *ip6 = NULL;
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif /* INET6 */
|
2007-03-23 20:16:50 +00:00
|
|
|
struct tcpopt to; /* options in this segment */
|
2007-05-28 11:03:53 +00:00
|
|
|
char *s = NULL; /* address and port logging */
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
int ti_locked;
|
|
|
|
#define TI_UNLOCKED 1
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
#define TI_WLOCKED 2
|
2002-06-10 20:05:46 +00:00
|
|
|
|
1994-09-15 10:36:56 +00:00
|
|
|
#ifdef TCPDEBUG
|
2002-08-17 02:05:25 +00:00
|
|
|
/*
|
|
|
|
* The size of tcp_saveipgen must be the size of the max ip header,
|
|
|
|
* now IPv6.
|
|
|
|
*/
|
2007-03-24 22:15:02 +00:00
|
|
|
u_char tcp_saveipgen[IP6_HDR_LEN];
|
2002-06-23 21:22:56 +00:00
|
|
|
struct tcphdr tcp_savetcp;
|
1994-09-15 10:36:56 +00:00
|
|
|
short ostate = 0;
|
|
|
|
#endif
|
2002-08-17 02:05:25 +00:00
|
|
|
|
2000-01-09 19:17:30 +00:00
|
|
|
#ifdef INET6
|
|
|
|
isipv6 = (mtod(m, struct ip *)->ip_v == 6) ? 1 : 0;
|
|
|
|
#endif
|
1995-02-09 23:13:27 +00:00
|
|
|
|
2007-03-23 20:16:50 +00:00
|
|
|
to.to_flags = 0;
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvtotal);
|
2000-01-09 19:17:30 +00:00
|
|
|
|
2004-03-01 19:10:31 +00:00
|
|
|
#ifdef INET6
|
2011-04-30 11:21:29 +00:00
|
|
|
if (isipv6) {
|
2007-05-28 23:27:44 +00:00
|
|
|
/* IP6_EXTHDR_CHECK() is already done at tcp6_input(). */
|
2012-05-25 02:23:26 +00:00
|
|
|
|
|
|
|
if (m->m_len < (sizeof(*ip6) + sizeof(*th))) {
|
|
|
|
m = m_pullup(m, sizeof(*ip6) + sizeof(*th));
|
|
|
|
if (m == NULL) {
|
|
|
|
TCPSTAT_INC(tcps_rcvshort);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2000-01-09 19:17:30 +00:00
|
|
|
ip6 = mtod(m, struct ip6_hdr *);
|
2012-05-25 02:23:26 +00:00
|
|
|
th = (struct tcphdr *)((caddr_t)ip6 + off0);
|
2000-01-09 19:17:30 +00:00
|
|
|
tlen = sizeof(*ip6) + ntohs(ip6->ip6_plen) - off0;
|
It turns out that too many drivers are not only parsing the L2/3/4
headers for TSO but also for generic checksum offloading. Ideally we
would only have one common function shared amongst all drivers, and
perhaps when updating them for IPv6 we should introduce that.
Eventually we should provide the meta information along with mbufs to
avoid (re-)parsing entirely.
To not break IPv6 (checksums and offload) and to be able to MFC the
changes without risking to hurt 3rd party drivers, duplicate the v4
framework, as other OSes have done as well.
Introduce interface capability flags for TX/RX checksum offload with
IPv6, to allow independent toggling (where possible). Add CSUM_*_IPV6
flags for UDP/TCP over IPv6, and reserve further for SCTP, and IPv6
fragmentation. Define CSUM_DELAY_DATA_IPV6 as we do for legacy IP and
add an alias for CSUM_DATA_VALID_IPV6.
This pretty much brings IPv6 handling in line with IPv4.
TSO is still handled in a different way and not via if_hwassist.
Update ifconfig to allow (un)setting of the new capability flags.
Update loopback to announce the new capabilities and if_hwassist flags.
Individual driver updates will have to follow, as will SCTP.
Reported by: gallatin, dim, ..
Reviewed by: gallatin (glanced at?)
MFC after: 3 days
X-MFC with: r235961,235959,235958
2012-05-28 09:30:13 +00:00
|
|
|
if (m->m_pkthdr.csum_flags & CSUM_DATA_VALID_IPV6) {
|
2012-05-25 02:23:26 +00:00
|
|
|
if (m->m_pkthdr.csum_flags & CSUM_PSEUDO_HDR)
|
|
|
|
th->th_sum = m->m_pkthdr.csum_data;
|
|
|
|
else
|
|
|
|
th->th_sum = in6_cksum_pseudo(ip6, tlen,
|
|
|
|
IPPROTO_TCP, m->m_pkthdr.csum_data);
|
|
|
|
th->th_sum ^= 0xffff;
|
|
|
|
} else
|
|
|
|
th->th_sum = in6_cksum(m, IPPROTO_TCP, off0, tlen);
|
|
|
|
if (th->th_sum) {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvbadsum);
|
2000-01-09 19:17:30 +00:00
|
|
|
goto drop;
|
|
|
|
}
|
2001-06-11 12:39:29 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Be proactive about unspecified IPv6 address in source.
|
|
|
|
* As we use all-zero to indicate unbounded/unconnected pcb,
|
|
|
|
* unspecified IPv6 address can be used to confuse us.
|
|
|
|
*
|
|
|
|
* Note that packets with unspecified IPv6 destination is
|
|
|
|
* already dropped in ip6_input.
|
|
|
|
*/
|
|
|
|
if (IN6_IS_ADDR_UNSPECIFIED(&ip6->ip6_src)) {
|
|
|
|
/* XXX stat */
|
|
|
|
goto drop;
|
|
|
|
}
|
2011-04-30 11:21:29 +00:00
|
|
|
}
|
2004-03-01 19:10:31 +00:00
|
|
|
#endif
|
2011-04-30 11:21:29 +00:00
|
|
|
#if defined(INET) && defined(INET6)
|
|
|
|
else
|
|
|
|
#endif
|
|
|
|
#ifdef INET
|
|
|
|
{
|
2000-03-27 19:14:27 +00:00
|
|
|
/*
|
2002-08-17 02:05:25 +00:00
|
|
|
* Get IP and TCP header together in first mbuf.
|
|
|
|
* Note: IP leaves IP header in first mbuf.
|
2000-03-27 19:14:27 +00:00
|
|
|
*/
|
2002-08-17 02:05:25 +00:00
|
|
|
if (off0 > sizeof (struct ip)) {
|
2012-10-12 09:24:24 +00:00
|
|
|
ip_stripoptions(m);
|
2002-08-17 02:05:25 +00:00
|
|
|
off0 = sizeof(struct ip);
|
|
|
|
}
|
|
|
|
if (m->m_len < sizeof (struct tcpiphdr)) {
|
2007-03-23 19:11:22 +00:00
|
|
|
if ((m = m_pullup(m, sizeof (struct tcpiphdr)))
|
|
|
|
== NULL) {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvshort);
|
2002-08-17 02:05:25 +00:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
ip = mtod(m, struct ip *);
|
|
|
|
th = (struct tcphdr *)((caddr_t)ip + off0);
|
2012-10-23 08:33:13 +00:00
|
|
|
tlen = ntohs(ip->ip_len) - off0;
|
2002-08-17 02:05:25 +00:00
|
|
|
|
|
|
|
if (m->m_pkthdr.csum_flags & CSUM_DATA_VALID) {
|
|
|
|
if (m->m_pkthdr.csum_flags & CSUM_PSEUDO_HDR)
|
|
|
|
th->th_sum = m->m_pkthdr.csum_data;
|
|
|
|
else
|
|
|
|
th->th_sum = in_pseudo(ip->ip_src.s_addr,
|
2012-10-22 21:09:03 +00:00
|
|
|
ip->ip_dst.s_addr,
|
|
|
|
htonl(m->m_pkthdr.csum_data + tlen +
|
|
|
|
IPPROTO_TCP));
|
2002-08-17 02:05:25 +00:00
|
|
|
th->th_sum ^= 0xffff;
|
|
|
|
} else {
|
2012-10-22 21:09:03 +00:00
|
|
|
struct ipovly *ipov = (struct ipovly *)ip;
|
|
|
|
|
2002-08-17 02:05:25 +00:00
|
|
|
/*
|
|
|
|
* Checksum extended TCP header and data.
|
|
|
|
*/
|
2012-10-22 21:09:03 +00:00
|
|
|
len = off0 + tlen;
|
2002-08-17 02:05:25 +00:00
|
|
|
bzero(ipov->ih_x1, sizeof(ipov->ih_x1));
|
2012-10-22 21:09:03 +00:00
|
|
|
ipov->ih_len = htons(tlen);
|
2002-08-17 02:05:25 +00:00
|
|
|
th->th_sum = in_cksum(m, len);
|
|
|
|
}
|
|
|
|
if (th->th_sum) {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvbadsum);
|
2002-08-17 02:05:25 +00:00
|
|
|
goto drop;
|
|
|
|
}
|
|
|
|
/* Re-initialization for later version check */
|
|
|
|
ip->ip_v = IPVERSION;
|
|
|
|
}
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif /* INET */
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2008-07-31 15:10:09 +00:00
|
|
|
#ifdef INET6
|
|
|
|
if (isipv6)
|
|
|
|
iptos = (ntohl(ip6->ip6_flow) >> 20) & 0xff;
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif
|
|
|
|
#if defined(INET) && defined(INET6)
|
2008-07-31 15:10:09 +00:00
|
|
|
else
|
|
|
|
#endif
|
2011-04-30 11:21:29 +00:00
|
|
|
#ifdef INET
|
2008-07-31 15:10:09 +00:00
|
|
|
iptos = ip->ip_tos;
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif
|
2008-07-31 15:10:09 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* Check that TCP offset makes sense,
|
|
|
|
* pull out TCP options and adjust length. XXX
|
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
off = th->th_off << 2;
|
1994-05-24 10:09:53 +00:00
|
|
|
if (off < sizeof (struct tcphdr) || off > tlen) {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvbadoff);
|
1994-05-24 10:09:53 +00:00
|
|
|
goto drop;
|
|
|
|
}
|
2000-01-09 19:17:30 +00:00
|
|
|
tlen -= off; /* tlen is used instead of ti->ti_len */
|
1994-05-24 10:09:53 +00:00
|
|
|
if (off > sizeof (struct tcphdr)) {
|
2004-03-01 19:10:31 +00:00
|
|
|
#ifdef INET6
|
2011-04-30 11:21:29 +00:00
|
|
|
if (isipv6) {
|
2000-01-09 19:17:30 +00:00
|
|
|
IP6_EXTHDR_CHECK(m, off0, off, );
|
|
|
|
ip6 = mtod(m, struct ip6_hdr *);
|
|
|
|
th = (struct tcphdr *)((caddr_t)ip6 + off0);
|
2011-04-30 11:21:29 +00:00
|
|
|
}
|
2004-03-01 19:10:31 +00:00
|
|
|
#endif
|
2011-04-30 11:21:29 +00:00
|
|
|
#if defined(INET) && defined(INET6)
|
|
|
|
else
|
|
|
|
#endif
|
|
|
|
#ifdef INET
|
|
|
|
{
|
2002-08-17 02:05:25 +00:00
|
|
|
if (m->m_len < sizeof(struct ip) + off) {
|
|
|
|
if ((m = m_pullup(m, sizeof (struct ip) + off))
|
2007-03-23 19:11:22 +00:00
|
|
|
== NULL) {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvshort);
|
2002-08-17 02:05:25 +00:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
ip = mtod(m, struct ip *);
|
|
|
|
th = (struct tcphdr *)((caddr_t)ip + off0);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
}
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif
|
1994-05-24 10:09:53 +00:00
|
|
|
optlen = off - sizeof (struct tcphdr);
|
2000-01-09 19:17:30 +00:00
|
|
|
optp = (u_char *)(th + 1);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
2000-01-09 19:17:30 +00:00
|
|
|
thflags = th->th_flags;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Convert TCP protocol specific fields to host format.
|
|
|
|
*/
|
2011-04-25 17:13:40 +00:00
|
|
|
tcp_fields_to_host(th);
|
1994-05-24 10:09:53 +00:00
|
|
|
|
1995-04-05 10:32:14 +00:00
|
|
|
/*
|
2007-03-21 18:52:58 +00:00
|
|
|
* Delay dropping TCP, IP headers, IPv6 ext headers, and TCP options.
|
1995-04-05 10:32:14 +00:00
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
drop_hdrlen = off0 + off;
|
1995-04-05 10:32:14 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
* Locate pcb for segment; if we're likely to add or remove a
|
|
|
|
* connection then first acquire pcbinfo lock. There are two cases
|
|
|
|
* where we might discover later we need a write lock despite the
|
|
|
|
* flags: ACKs moving a connection out of the syncache, and ACKs for
|
|
|
|
* a connection in TIMEWAIT.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if ((thflags & (TH_SYN | TH_FIN | TH_RST)) != 0) {
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WLOCK(&V_tcbinfo);
|
|
|
|
ti_locked = TI_WLOCKED;
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
} else
|
|
|
|
ti_locked = TI_UNLOCKED;
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
findpcb:
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
#ifdef INVARIANTS
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (ti_locked == TI_WLOCKED) {
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
} else {
|
|
|
|
INP_INFO_UNLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
}
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
#endif
|
|
|
|
|
2007-05-28 23:27:44 +00:00
|
|
|
/*
|
|
|
|
* Grab info from PACKET_TAG_IPFORWARD tag prepended to the chain.
|
|
|
|
*/
|
2012-11-02 01:20:55 +00:00
|
|
|
if (m->m_flags & M_IP_NEXTHOP)
|
2012-10-25 09:39:14 +00:00
|
|
|
fwd_tag = m_tag_find(m, PACKET_TAG_IPFORWARD, NULL);
|
2004-08-17 22:05:54 +00:00
|
|
|
|
2011-08-20 17:05:11 +00:00
|
|
|
#ifdef INET6
|
|
|
|
if (isipv6 && fwd_tag != NULL) {
|
|
|
|
struct sockaddr_in6 *next_hop6;
|
|
|
|
|
|
|
|
next_hop6 = (struct sockaddr_in6 *)(fwd_tag + 1);
|
|
|
|
/*
|
|
|
|
* Transparently forwarded. Pretend to be the destination.
|
|
|
|
* Already got one like this?
|
|
|
|
*/
|
|
|
|
inp = in6_pcblookup_mbuf(&V_tcbinfo,
|
|
|
|
&ip6->ip6_src, th->th_sport, &ip6->ip6_dst, th->th_dport,
|
|
|
|
INPLOOKUP_WLOCKPCB, m->m_pkthdr.rcvif, m);
|
|
|
|
if (!inp) {
|
|
|
|
/*
|
|
|
|
* It's new. Try to find the ambushing socket.
|
|
|
|
* Because we've rewritten the destination address,
|
|
|
|
* any hardware-generated hash is ignored.
|
|
|
|
*/
|
|
|
|
inp = in6_pcblookup(&V_tcbinfo, &ip6->ip6_src,
|
|
|
|
th->th_sport, &next_hop6->sin6_addr,
|
|
|
|
next_hop6->sin6_port ? ntohs(next_hop6->sin6_port) :
|
|
|
|
th->th_dport, INPLOOKUP_WILDCARD |
|
|
|
|
INPLOOKUP_WLOCKPCB, m->m_pkthdr.rcvif);
|
|
|
|
}
|
|
|
|
/* Remove the tag from the packet. We don't need it anymore. */
|
|
|
|
m_tag_delete(m, fwd_tag);
|
2012-11-02 01:20:55 +00:00
|
|
|
m->m_flags &= ~M_IP_NEXTHOP;
|
2012-10-25 09:39:14 +00:00
|
|
|
} else if (isipv6) {
|
2011-08-20 17:05:11 +00:00
|
|
|
inp = in6_pcblookup_mbuf(&V_tcbinfo, &ip6->ip6_src,
|
|
|
|
th->th_sport, &ip6->ip6_dst, th->th_dport,
|
|
|
|
INPLOOKUP_WILDCARD | INPLOOKUP_WLOCKPCB,
|
|
|
|
m->m_pkthdr.rcvif, m);
|
|
|
|
}
|
|
|
|
#endif /* INET6 */
|
|
|
|
#if defined(INET6) && defined(INET)
|
|
|
|
else
|
|
|
|
#endif
|
|
|
|
#ifdef INET
|
|
|
|
if (fwd_tag != NULL) {
|
2004-08-17 22:05:54 +00:00
|
|
|
struct sockaddr_in *next_hop;
|
|
|
|
|
|
|
|
next_hop = (struct sockaddr_in *)(fwd_tag+1);
|
1998-07-06 03:20:19 +00:00
|
|
|
/*
|
Remove (almost all) global variables that were used to hold
packet forwarding state ("annotations") during ip processing.
The code is considerably cleaner now.
The variables removed by this change are:
ip_divert_cookie used by divert sockets
ip_fw_fwd_addr used for transparent ip redirection
last_pkt used by dynamic pipes in dummynet
Removal of the first two has been done by carrying the annotations
into volatile structs prepended to the mbuf chains, and adding
appropriate code to add/remove annotations in the routines which
make use of them, i.e. ip_input(), ip_output(), tcp_input(),
bdg_forward(), ether_demux(), ether_output_frame(), div_output().
On passing, remove a bug in divert handling of fragmented packet.
Now it is the fragment at offset 0 which sets the divert status of
the whole packet, whereas formerly it was the last incoming fragment
to decide.
Removal of last_pkt required a change in the interface of ip_fw_chk()
and dummynet_io(). On passing, use the same mechanism for dummynet
annotations and for divert/forward annotations.
option IPFIREWALL_FORWARD is effectively useless, the code to
implement it is very small and is now in by default to avoid the
obfuscation of conditionally compiled code.
NOTES:
* there is at least one global variable left, sro_fwd, in ip_output().
I am not sure if/how this can be removed.
* I have deliberately avoided gratuitous style changes in this commit
to avoid cluttering the diffs. Minor stule cleanup will likely be
necessary
* this commit only focused on the IP layer. I am sure there is a
number of global variables used in the TCP and maybe UDP stack.
* despite the number of files touched, there are absolutely no API's
or data structures changed by this commit (except the interfaces of
ip_fw_chk() and dummynet_io(), which are internal anyways), so
an MFC is quite safe and unintrusive (and desirable, given the
improved readability of the code).
MFC after: 10 days
2002-06-22 11:51:02 +00:00
|
|
|
* Transparently forwarded. Pretend to be the destination.
|
2004-08-16 18:32:07 +00:00
|
|
|
* already got one like this?
|
1998-07-06 03:20:19 +00:00
|
|
|
*/
|
Add _mbuf() variants of various inpcb-related interfaces, including lookup,
hash install, etc. For now, these are arguments are unused, but as we add
RSS support, we will want to use hashes extracted from mbufs, rather than
manually calculated hashes of header fields, due to the expensive of the
software version of Toeplitz (and similar hashes).
Add notes that it would be nice to be able to pass mbufs into lookup
routines in pf(4), optimising firewall lookup in the same way, but the
code structure there doesn't facilitate that currently.
(In principle there is no reason this couldn't be MFCed -- the change
extends rather than modifies the KBI. However, it won't be useful without
other previous possibly less MFCable changes.)
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-06-04 16:33:06 +00:00
|
|
|
inp = in_pcblookup_mbuf(&V_tcbinfo, ip->ip_src, th->th_sport,
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
ip->ip_dst, th->th_dport, INPLOOKUP_WLOCKPCB,
|
Add _mbuf() variants of various inpcb-related interfaces, including lookup,
hash install, etc. For now, these are arguments are unused, but as we add
RSS support, we will want to use hashes extracted from mbufs, rather than
manually calculated hashes of header fields, due to the expensive of the
software version of Toeplitz (and similar hashes).
Add notes that it would be nice to be able to pass mbufs into lookup
routines in pf(4), optimising firewall lookup in the same way, but the
code structure there doesn't facilitate that currently.
(In principle there is no reason this couldn't be MFCed -- the change
extends rather than modifies the KBI. However, it won't be useful without
other previous possibly less MFCable changes.)
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-06-04 16:33:06 +00:00
|
|
|
m->m_pkthdr.rcvif, m);
|
1998-07-06 03:20:19 +00:00
|
|
|
if (!inp) {
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
/*
|
|
|
|
* It's new. Try to find the ambushing socket.
|
Add _mbuf() variants of various inpcb-related interfaces, including lookup,
hash install, etc. For now, these are arguments are unused, but as we add
RSS support, we will want to use hashes extracted from mbufs, rather than
manually calculated hashes of header fields, due to the expensive of the
software version of Toeplitz (and similar hashes).
Add notes that it would be nice to be able to pass mbufs into lookup
routines in pf(4), optimising firewall lookup in the same way, but the
code structure there doesn't facilitate that currently.
(In principle there is no reason this couldn't be MFCed -- the change
extends rather than modifies the KBI. However, it won't be useful without
other previous possibly less MFCable changes.)
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-06-04 16:33:06 +00:00
|
|
|
* Because we've rewritten the destination address,
|
|
|
|
* any hardware-generated hash is ignored.
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
*/
|
|
|
|
inp = in_pcblookup(&V_tcbinfo, ip->ip_src,
|
|
|
|
th->th_sport, next_hop->sin_addr,
|
|
|
|
next_hop->sin_port ? ntohs(next_hop->sin_port) :
|
|
|
|
th->th_dport, INPLOOKUP_WILDCARD |
|
|
|
|
INPLOOKUP_WLOCKPCB, m->m_pkthdr.rcvif);
|
1998-07-06 03:20:19 +00:00
|
|
|
}
|
2004-08-17 22:05:54 +00:00
|
|
|
/* Remove the tag from the packet. We don't need it anymore. */
|
|
|
|
m_tag_delete(m, fwd_tag);
|
2012-11-02 01:20:55 +00:00
|
|
|
m->m_flags &= ~M_IP_NEXTHOP;
|
2007-03-21 18:56:03 +00:00
|
|
|
} else
|
2011-08-20 17:05:11 +00:00
|
|
|
inp = in_pcblookup_mbuf(&V_tcbinfo, ip->ip_src,
|
|
|
|
th->th_sport, ip->ip_dst, th->th_dport,
|
|
|
|
INPLOOKUP_WILDCARD | INPLOOKUP_WLOCKPCB,
|
|
|
|
m->m_pkthdr.rcvif, m);
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif /* INET */
|
2000-01-09 19:17:30 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
2007-03-21 18:36:49 +00:00
|
|
|
* If the INPCB does not exist then all data in the incoming
|
|
|
|
* segment is discarded and an appropriate RST is sent back.
|
Add code to allow the system to handle multiple routing tables.
This particular implementation is designed to be fully backwards compatible
and to be MFC-able to 7.x (and 6.x)
Currently the only protocol that can make use of the multiple tables is IPv4
Similar functionality exists in OpenBSD and Linux.
From my notes:
-----
One thing where FreeBSD has been falling behind, and which by chance I
have some time to work on is "policy based routing", which allows
different
packet streams to be routed by more than just the destination address.
Constraints:
------------
I want to make some form of this available in the 6.x tree
(and by extension 7.x) , but FreeBSD in general needs it so I might as
well do it in -current and back port the portions I need.
One of the ways that this can be done is to have the ability to
instantiate multiple kernel routing tables (which I will now
refer to as "Forwarding Information Bases" or "FIBs" for political
correctness reasons). Which FIB a particular packet uses to make
the next hop decision can be decided by a number of mechanisms.
The policies these mechanisms implement are the "Policies" referred
to in "Policy based routing".
One of the constraints I have if I try to back port this work to
6.x is that it must be implemented as a EXTENSION to the existing
ABIs in 6.x so that third party applications do not need to be
recompiled in timespan of the branch.
This first version will not have some of the bells and whistles that
will come with later versions. It will, for example, be limited to 16
tables in the first commit.
Implementation method, Compatible version. (part 1)
-------------------------------
For this reason I have implemented a "sufficient subset" of a
multiple routing table solution in Perforce, and back-ported it
to 6.x. (also in Perforce though not always caught up with what I
have done in -current/P4). The subset allows a number of FIBs
to be defined at compile time (8 is sufficient for my purposes in 6.x)
and implements the changes needed to allow IPV4 to use them. I have not
done the changes for ipv6 simply because I do not need it, and I do not
have enough knowledge of ipv6 (e.g. neighbor discovery) needed to do it.
Other protocol families are left untouched and should there be
users with proprietary protocol families, they should continue to work
and be oblivious to the existence of the extra FIBs.
To understand how this is done, one must know that the current FIB
code starts everything off with a single dimensional array of
pointers to FIB head structures (One per protocol family), each of
which in turn points to the trie of routes available to that family.
The basic change in the ABI compatible version of the change is to
extent that array to be a 2 dimensional array, so that
instead of protocol family X looking at rt_tables[X] for the
table it needs, it looks at rt_tables[Y][X] when for all
protocol families except ipv4 Y is always 0.
Code that is unaware of the change always just sees the first row
of the table, which of course looks just like the one dimensional
array that existed before.
The entry points rtrequest(), rtalloc(), rtalloc1(), rtalloc_ign()
are all maintained, but refer only to the first row of the array,
so that existing callers in proprietary protocols can continue to
do the "right thing".
Some new entry points are added, for the exclusive use of ipv4 code
called in_rtrequest(), in_rtalloc(), in_rtalloc1() and in_rtalloc_ign(),
which have an extra argument which refers the code to the correct row.
In addition, there are some new entry points (currently called
rtalloc_fib() and friends) that check the Address family being
looked up and call either rtalloc() (and friends) if the protocol
is not IPv4 forcing the action to row 0 or to the appropriate row
if it IS IPv4 (and that info is available). These are for calling
from code that is not specific to any particular protocol. The way
these are implemented would change in the non ABI preserving code
to be added later.
One feature of the first version of the code is that for ipv4,
the interface routes show up automatically on all the FIBs, so
that no matter what FIB you select you always have the basic
direct attached hosts available to you. (rtinit() does this
automatically).
You CAN delete an interface route from one FIB should you want
to but by default it's there. ARP information is also available
in each FIB. It's assumed that the same machine would have the
same MAC address, regardless of which FIB you are using to get
to it.
This brings us as to how the correct FIB is selected for an outgoing
IPV4 packet.
Firstly, all packets have a FIB associated with them. if nothing
has been done to change it, it will be FIB 0. The FIB is changed
in the following ways.
Packets fall into one of a number of classes.
1/ locally generated packets, coming from a socket/PCB.
Such packets select a FIB from a number associated with the
socket/PCB. This in turn is inherited from the process,
but can be changed by a socket option. The process in turn
inherits it on fork. I have written a utility call setfib
that acts a bit like nice..
setfib -3 ping target.example.com # will use fib 3 for ping.
It is an obvious extension to make it a property of a jail
but I have not done so. It can be achieved by combining the setfib and
jail commands.
2/ packets received on an interface for forwarding.
By default these packets would use table 0,
(or possibly a number settable in a sysctl(not yet)).
but prior to routing the firewall can inspect them (see below).
(possibly in the future you may be able to associate a FIB
with packets received on an interface.. An ifconfig arg, but not yet.)
3/ packets inspected by a packet classifier, which can arbitrarily
associate a fib with it on a packet by packet basis.
A fib assigned to a packet by a packet classifier
(such as ipfw) would over-ride a fib associated by
a more default source. (such as cases 1 or 2).
4/ a tcp listen socket associated with a fib will generate
accept sockets that are associated with that same fib.
5/ Packets generated in response to some other packet (e.g. reset
or icmp packets). These should use the FIB associated with the
packet being reponded to.
6/ Packets generated during encapsulation.
gif, tun and other tunnel interfaces will encapsulate using the FIB
that was in effect withthe proces that set up the tunnel.
thus setfib 1 ifconfig gif0 [tunnel instructions]
will set the fib for the tunnel to use to be fib 1.
Routing messages would be associated with their
process, and thus select one FIB or another.
messages from the kernel would be associated with the fib they
refer to and would only be received by a routing socket associated
with that fib. (not yet implemented)
In addition Netstat has been edited to be able to cope with the
fact that the array is now 2 dimensional. (It looks in system
memory using libkvm (!)). Old versions of netstat see only the first FIB.
In addition two sysctls are added to give:
a) the number of FIBs compiled in (active)
b) the default FIB of the calling process.
Early testing experience:
-------------------------
Basically our (IronPort's) appliance does this functionality already
using ipfw fwd but that method has some drawbacks.
For example,
It can't fully simulate a routing table because it can't influence the
socket's choice of local address when a connect() is done.
Testing during the generating of these changes has been
remarkably smooth so far. Multiple tables have co-existed
with no notable side effects, and packets have been routes
accordingly.
ipfw has grown 2 new keywords:
setfib N ip from anay to any
count ip from any to any fib N
In pf there seems to be a requirement to be able to give symbolic names to the
fibs but I do not have that capacity. I am not sure if it is required.
SCTP has interestingly enough built in support for this, called VRFs
in Cisco parlance. it will be interesting to see how that handles it
when it suddenly actually does something.
Where to next:
--------------------
After committing the ABI compatible version and MFCing it, I'd
like to proceed in a forward direction in -current. this will
result in some roto-tilling in the routing code.
Firstly: the current code's idea of having a separate tree per
protocol family, all of the same format, and pointed to by the
1 dimensional array is a bit silly. Especially when one considers that
there is code that makes assumptions about every protocol having the
same internal structures there. Some protocols don't WANT that
sort of structure. (for example the whole idea of a netmask is foreign
to appletalk). This needs to be made opaque to the external code.
My suggested first change is to add routing method pointers to the
'domain' structure, along with information pointing the data.
instead of having an array of pointers to uniform structures,
there would be an array pointing to the 'domain' structures
for each protocol address domain (protocol family),
and the methods this reached would be called. The methods would have
an argument that gives FIB number, but the protocol would be free
to ignore it.
When the ABI can be changed it raises the possibilty of the
addition of a fib entry into the "struct route". Currently,
the structure contains the sockaddr of the desination, and the resulting
fib entry. To make this work fully, one could add a fib number
so that given an address and a fib, one can find the third element, the
fib entry.
Interaction with the ARP layer/ LL layer would need to be
revisited as well. Qing Li has been working on this already.
This work was sponsored by Ironport Systems/Cisco
Reviewed by: several including rwatson, bz and mlair (parts each)
Obtained from: Ironport systems/Cisco
2008-05-09 23:03:00 +00:00
|
|
|
* XXX MRT Send RST using which routing table?
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1996-04-04 10:46:44 +00:00
|
|
|
if (inp == NULL) {
|
2007-03-21 18:36:49 +00:00
|
|
|
/*
|
|
|
|
* Log communication attempts to ports that are not
|
|
|
|
* in use.
|
|
|
|
*/
|
|
|
|
if ((tcp_log_in_vain == 1 && (thflags & TH_SYN)) ||
|
|
|
|
tcp_log_in_vain == 2) {
|
2010-08-18 17:39:47 +00:00
|
|
|
if ((s = tcp_log_vain(NULL, th, (void *)ip, ip6)))
|
Add tcp_log_addrs() function to generate and standardized TCP log line
for use thoughout the tcp subsystem.
It is IPv4 and IPv6 aware creates a line in the following format:
"TCP: [1.2.3.4]:50332 to [1.2.3.4]:80 tcpflags <RST>"
A "\n" is not included at the end. The caller is supposed to add
further information after the standard tcp log header.
The function returns a NUL terminated string which the caller has
to free(s, M_TCPLOG) after use. All memory allocation is done
with M_NOWAIT and the return value may be NULL in memory shortage
situations.
Either struct in_conninfo || (struct tcphdr && (struct ip || struct
ip6_hdr) have to be supplied.
Due to ip[6].h header inclusion limitations and ordering issues the
struct ip and struct ip6_hdr parameters have to be casted and passed
as void * pointers.
tcp_log_addrs(struct in_conninfo *inc, struct tcphdr *th, void *ip4hdr,
void *ip6hdr)
Usage example:
struct ip *ip;
char *tcplog;
if (tcplog = tcp_log_addrs(NULL, th, (void *)ip, NULL)) {
log(LOG_DEBUG, "%s; %s: Connection attempt to closed port\n",
tcplog, __func__);
free(s, M_TCPLOG);
}
2007-05-18 19:58:37 +00:00
|
|
|
log(LOG_INFO, "%s; %s: Connection attempt "
|
|
|
|
"to closed port\n", s, __func__);
|
1999-08-19 05:22:12 +00:00
|
|
|
}
|
2007-03-21 18:36:49 +00:00
|
|
|
/*
|
|
|
|
* When blackholing do not respond with a RST but
|
|
|
|
* completely ignore the segment and drop it.
|
|
|
|
*/
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
if ((V_blackhole == 1 && (thflags & TH_SYN)) ||
|
|
|
|
V_blackhole == 2)
|
2007-03-28 12:58:13 +00:00
|
|
|
goto dropunlock;
|
2007-03-21 18:36:49 +00:00
|
|
|
|
2001-02-11 07:39:51 +00:00
|
|
|
rstreason = BANDLIM_RST_CLOSEDPORT;
|
|
|
|
goto dropwithreset;
|
1996-04-04 10:46:44 +00:00
|
|
|
}
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
INP_WLOCK_ASSERT(inp);
|
2009-04-10 06:16:14 +00:00
|
|
|
if (!(inp->inp_flags & INP_HW_FLOWID)
|
|
|
|
&& (m->m_flags & M_FLOWID)
|
|
|
|
&& ((inp->inp_socket == NULL)
|
|
|
|
|| !(inp->inp_socket->so_options & SO_ACCEPTCONN))) {
|
|
|
|
inp->inp_flags |= INP_HW_FLOWID;
|
|
|
|
inp->inp_flags &= ~INP_SW_FLOWID;
|
|
|
|
inp->inp_flowid = m->m_pkthdr.flowid;
|
|
|
|
}
|
2007-09-10 14:49:32 +00:00
|
|
|
#ifdef IPSEC
|
|
|
|
#ifdef INET6
|
|
|
|
if (isipv6 && ipsec6_in_reject(m, inp)) {
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
V_ipsec6stat.in_polvio++;
|
2007-09-10 14:49:32 +00:00
|
|
|
goto dropunlock;
|
|
|
|
} else
|
|
|
|
#endif /* INET6 */
|
|
|
|
if (ipsec4_in_reject(m, inp) != 0) {
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
V_ipsec4stat.in_polvio++;
|
2007-09-10 14:49:32 +00:00
|
|
|
goto dropunlock;
|
|
|
|
}
|
|
|
|
#endif /* IPSEC */
|
|
|
|
|
2007-05-28 23:27:44 +00:00
|
|
|
/*
|
|
|
|
* Check the minimum TTL for socket.
|
|
|
|
*/
|
2006-01-14 16:39:31 +00:00
|
|
|
if (inp->inp_ip_minttl != 0) {
|
|
|
|
#ifdef INET6
|
|
|
|
if (isipv6 && inp->inp_ip_minttl > ip6->ip6_hlim)
|
2007-03-23 20:16:50 +00:00
|
|
|
goto dropunlock;
|
2006-01-18 18:59:30 +00:00
|
|
|
else
|
2006-01-14 16:39:31 +00:00
|
|
|
#endif
|
|
|
|
if (inp->inp_ip_minttl > ip->ip_ttl)
|
2007-03-23 20:16:50 +00:00
|
|
|
goto dropunlock;
|
2006-01-14 16:39:31 +00:00
|
|
|
}
|
2005-08-22 16:13:08 +00:00
|
|
|
|
2007-03-21 18:52:58 +00:00
|
|
|
/*
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
* A previous connection in TIMEWAIT state is supposed to catch stray
|
|
|
|
* or duplicate segments arriving late. If this segment was a
|
2012-07-22 17:31:36 +00:00
|
|
|
* legitimate new connection attempt, the old INPCB gets removed and
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
* we can try again to find a listening socket.
|
|
|
|
*
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
* At this point, due to earlier optimism, we may hold only an inpcb
|
|
|
|
* lock, and not the inpcbinfo write lock. If so, we need to try to
|
|
|
|
* acquire it, or if that fails, acquire a reference on the inpcb,
|
|
|
|
* drop all locks, acquire a global write lock, and then re-acquire
|
|
|
|
* the inpcb lock. We may at that point discover that another thread
|
|
|
|
* has tried to free the inpcb, in which case we need to loop back
|
|
|
|
* and try to find a new inpcb to deliver to.
|
|
|
|
*
|
|
|
|
* XXXRW: It may be time to rethink timewait locking.
|
2007-03-21 18:52:58 +00:00
|
|
|
*/
|
2009-10-05 22:24:13 +00:00
|
|
|
relocked:
|
2009-03-15 09:58:31 +00:00
|
|
|
if (inp->inp_flags & INP_TIMEWAIT) {
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (ti_locked == TI_UNLOCKED) {
|
|
|
|
if (INP_INFO_TRY_WLOCK(&V_tcbinfo) == 0) {
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
in_pcbref(inp);
|
|
|
|
INP_WUNLOCK(inp);
|
|
|
|
INP_INFO_WLOCK(&V_tcbinfo);
|
|
|
|
ti_locked = TI_WLOCKED;
|
|
|
|
INP_WLOCK(inp);
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (in_pcbrele_wlocked(inp)) {
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
inp = NULL;
|
|
|
|
goto findpcb;
|
|
|
|
}
|
2009-10-06 20:35:41 +00:00
|
|
|
} else
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
ti_locked = TI_WLOCKED;
|
|
|
|
}
|
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
|
2003-02-19 22:32:43 +00:00
|
|
|
if (thflags & TH_SYN)
|
2006-06-26 15:35:25 +00:00
|
|
|
tcp_dooptions(&to, optp, optlen, TO_SYN);
|
2007-05-28 23:27:44 +00:00
|
|
|
/*
|
|
|
|
* NB: tcp_twcheck unlocks the INP and frees the mbuf.
|
|
|
|
*/
|
2007-05-16 17:14:25 +00:00
|
|
|
if (tcp_twcheck(inp, &to, th, m, tlen))
|
2003-02-19 22:32:43 +00:00
|
|
|
goto findpcb;
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
INP_INFO_WUNLOCK(&V_tcbinfo);
|
2003-02-19 22:32:43 +00:00
|
|
|
return;
|
|
|
|
}
|
2007-03-21 18:52:58 +00:00
|
|
|
/*
|
|
|
|
* The TCPCB may no longer exist if the connection is winding
|
|
|
|
* down or it is in the CLOSED state. Either way we drop the
|
|
|
|
* segment and send an appropriate response.
|
|
|
|
*/
|
1994-05-24 10:09:53 +00:00
|
|
|
tp = intotcpcb(inp);
|
2007-05-28 11:03:53 +00:00
|
|
|
if (tp == NULL || tp->t_state == TCPS_CLOSED) {
|
2001-02-11 07:39:51 +00:00
|
|
|
rstreason = BANDLIM_RST_CLOSEDPORT;
|
|
|
|
goto dropwithreset;
|
2000-12-15 21:45:49 +00:00
|
|
|
}
|
1995-05-30 08:16:23 +00:00
|
|
|
|
2012-06-19 07:34:13 +00:00
|
|
|
#ifdef TCP_OFFLOAD
|
|
|
|
if (tp->t_flags & TF_TOE) {
|
|
|
|
tcp_offload_input(tp, m);
|
|
|
|
m = NULL; /* consumed by the TOE driver */
|
|
|
|
goto dropunlock;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
/*
|
|
|
|
* We've identified a valid inpcb, but it could be that we need an
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
* inpcbinfo write lock but don't hold it. In this case, attempt to
|
|
|
|
* acquire using the same strategy as the TIMEWAIT case above. If we
|
|
|
|
* relock, we have to jump back to 'relocked' as the connection might
|
|
|
|
* now be in TIMEWAIT.
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
*/
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
#ifdef INVARIANTS
|
|
|
|
if ((thflags & (TH_SYN | TH_FIN | TH_RST)) != 0)
|
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
#endif
|
|
|
|
if (tp->t_state != TCPS_ESTABLISHED) {
|
|
|
|
if (ti_locked == TI_UNLOCKED) {
|
|
|
|
if (INP_INFO_TRY_WLOCK(&V_tcbinfo) == 0) {
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
in_pcbref(inp);
|
|
|
|
INP_WUNLOCK(inp);
|
|
|
|
INP_INFO_WLOCK(&V_tcbinfo);
|
|
|
|
ti_locked = TI_WLOCKED;
|
|
|
|
INP_WLOCK(inp);
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (in_pcbrele_wlocked(inp)) {
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
inp = NULL;
|
|
|
|
goto findpcb;
|
|
|
|
}
|
2009-10-05 22:24:13 +00:00
|
|
|
goto relocked;
|
2009-10-06 20:35:41 +00:00
|
|
|
} else
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
ti_locked = TI_WLOCKED;
|
|
|
|
}
|
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
}
|
|
|
|
|
2002-07-31 19:06:49 +00:00
|
|
|
#ifdef MAC
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK_ASSERT(inp);
|
2007-10-24 19:04:04 +00:00
|
|
|
if (mac_inpcb_check_deliver(inp, m))
|
2007-03-23 20:16:50 +00:00
|
|
|
goto dropunlock;
|
2002-07-31 19:06:49 +00:00
|
|
|
#endif
|
Introduce a MAC label reference in 'struct inpcb', which caches
the MAC label referenced from 'struct socket' in the IPv4 and
IPv6-based protocols. This permits MAC labels to be checked during
network delivery operations without dereferencing inp->inp_socket
to get to so->so_label, which will eventually avoid our having to
grab the socket lock during delivery at the network layer.
This change introduces 'struct inpcb' as a labeled object to the
MAC Framework, along with the normal circus of entry points:
initialization, creation from socket, destruction, as well as a
delivery access control check.
For most policies, the inpcb label will simply be a cache of the
socket label, so a new protocol switch method is introduced,
pr_sosetlabel() to notify protocols that the socket layer label
has been updated so that the cache can be updated while holding
appropriate locks. Most protocols implement this using
pru_sosetlabel_null(), but IPv4/IPv6 protocols using inpcbs use
the the worker function in_pcbsosetlabel(), which calls into the
MAC Framework to perform a cache update.
Biba, LOMAC, and MLS implement these entry points, as do the stub
policy, and test policy.
Reviewed by: sam, bms
Obtained from: TrustedBSD Project
Sponsored by: DARPA, Network Associates Laboratories
2003-11-18 00:39:07 +00:00
|
|
|
so = inp->inp_socket;
|
2007-03-23 20:16:50 +00:00
|
|
|
KASSERT(so != NULL, ("%s: so == NULL", __func__));
|
1994-09-15 10:36:56 +00:00
|
|
|
#ifdef TCPDEBUG
|
2002-12-20 11:16:52 +00:00
|
|
|
if (so->so_options & SO_DEBUG) {
|
|
|
|
ostate = tp->t_state;
|
2007-05-09 06:09:40 +00:00
|
|
|
#ifdef INET6
|
2011-08-20 18:45:38 +00:00
|
|
|
if (isipv6) {
|
2002-12-20 11:16:52 +00:00
|
|
|
bcopy((char *)ip6, (char *)tcp_saveipgen, sizeof(*ip6));
|
2007-05-09 11:39:46 +00:00
|
|
|
} else
|
2011-08-20 18:45:38 +00:00
|
|
|
#endif
|
2002-12-20 11:16:52 +00:00
|
|
|
bcopy((char *)ip, (char *)tcp_saveipgen, sizeof(*ip));
|
|
|
|
tcp_savetcp = *th;
|
|
|
|
}
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif /* TCPDEBUG */
|
2007-03-21 18:49:43 +00:00
|
|
|
/*
|
|
|
|
* When the socket is accepting connections (the INPCB is in LISTEN
|
|
|
|
* state) we look into the SYN cache if this is a new connection
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
* attempt or the completion of a previous one. Because listen
|
|
|
|
* sockets are never in TCPS_ESTABLISHED, the V_tcbinfo lock will be
|
|
|
|
* held in this case.
|
2007-03-21 18:49:43 +00:00
|
|
|
*/
|
2002-12-20 11:16:52 +00:00
|
|
|
if (so->so_options & SO_ACCEPTCONN) {
|
|
|
|
struct in_conninfo inc;
|
|
|
|
|
2007-04-20 15:21:29 +00:00
|
|
|
KASSERT(tp->t_state == TCPS_LISTEN, ("%s: so accepting but "
|
|
|
|
"tp not listening", __func__));
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
2007-04-20 15:21:29 +00:00
|
|
|
|
2006-06-26 16:14:19 +00:00
|
|
|
bzero(&inc, sizeof(inc));
|
2007-03-23 20:16:50 +00:00
|
|
|
#ifdef INET6
|
2001-11-22 04:50:44 +00:00
|
|
|
if (isipv6) {
|
2008-12-17 12:52:34 +00:00
|
|
|
inc.inc_flags |= INC_ISIPV6;
|
2001-11-22 04:50:44 +00:00
|
|
|
inc.inc6_faddr = ip6->ip6_src;
|
|
|
|
inc.inc6_laddr = ip6->ip6_dst;
|
2007-03-23 20:16:50 +00:00
|
|
|
} else
|
|
|
|
#endif
|
|
|
|
{
|
2001-11-22 04:50:44 +00:00
|
|
|
inc.inc_faddr = ip->ip_src;
|
|
|
|
inc.inc_laddr = ip->ip_dst;
|
|
|
|
}
|
|
|
|
inc.inc_fport = th->th_sport;
|
|
|
|
inc.inc_lport = th->th_dport;
|
2009-07-28 19:43:27 +00:00
|
|
|
inc.inc_fibnum = so->so_fibnum;
|
2001-11-22 04:50:44 +00:00
|
|
|
|
2007-03-23 19:11:22 +00:00
|
|
|
/*
|
2007-05-28 23:27:44 +00:00
|
|
|
* Check for an existing connection attempt in syncache if
|
|
|
|
* the flag is only ACK. A successful lookup creates a new
|
|
|
|
* socket appended to the listen queue in SYN_RECEIVED state.
|
2001-11-22 04:50:44 +00:00
|
|
|
*/
|
2007-05-28 11:03:53 +00:00
|
|
|
if ((thflags & (TH_RST|TH_ACK|TH_SYN)) == TH_ACK) {
|
2007-05-28 11:35:40 +00:00
|
|
|
/*
|
|
|
|
* Parse the TCP options here because
|
|
|
|
* syncookies need access to the reflected
|
|
|
|
* timestamp.
|
|
|
|
*/
|
|
|
|
tcp_dooptions(&to, optp, optlen, 0);
|
|
|
|
/*
|
|
|
|
* NB: syncache_expand() doesn't unlock
|
|
|
|
* inp and tcpinfo locks.
|
|
|
|
*/
|
|
|
|
if (!syncache_expand(&inc, &to, th, &so, m)) {
|
2006-09-13 13:08:27 +00:00
|
|
|
/*
|
2007-05-28 11:35:40 +00:00
|
|
|
* No syncache entry or ACK was not
|
|
|
|
* for our SYN/ACK. Send a RST.
|
2007-05-28 23:27:44 +00:00
|
|
|
* NB: syncache did its own logging
|
|
|
|
* of the failure cause.
|
2006-09-13 13:08:27 +00:00
|
|
|
*/
|
2007-05-28 11:35:40 +00:00
|
|
|
rstreason = BANDLIM_RST_OPENPORT;
|
|
|
|
goto dropwithreset;
|
|
|
|
}
|
|
|
|
if (so == NULL) {
|
2007-04-23 19:41:47 +00:00
|
|
|
/*
|
2007-05-28 11:35:40 +00:00
|
|
|
* We completed the 3-way handshake
|
|
|
|
* but could not allocate a socket
|
|
|
|
* either due to memory shortage,
|
|
|
|
* listen queue length limits or
|
|
|
|
* global socket limits. Send RST
|
|
|
|
* or wait and have the remote end
|
|
|
|
* retransmit the ACK for another
|
|
|
|
* try.
|
2007-04-23 19:41:47 +00:00
|
|
|
*/
|
2007-05-28 11:35:40 +00:00
|
|
|
if ((s = tcp_log_addrs(&inc, th, NULL, NULL)))
|
|
|
|
log(LOG_DEBUG, "%s; %s: Listen socket: "
|
2007-05-28 23:27:44 +00:00
|
|
|
"Socket allocation failed due to "
|
|
|
|
"limits or memory shortage, %s\n",
|
2008-08-20 01:05:56 +00:00
|
|
|
s, __func__,
|
|
|
|
V_tcp_sc_rst_sock_fail ?
|
|
|
|
"sending RST" : "try again");
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
if (V_tcp_sc_rst_sock_fail) {
|
2007-05-28 11:35:40 +00:00
|
|
|
rstreason = BANDLIM_UNLIMITED;
|
2001-02-11 07:39:51 +00:00
|
|
|
goto dropwithreset;
|
2007-05-28 11:35:40 +00:00
|
|
|
} else
|
|
|
|
goto dropunlock;
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* Socket is created in state SYN_RECEIVED.
|
2007-05-28 23:27:44 +00:00
|
|
|
* Unlock the listen socket, lock the newly
|
|
|
|
* created socket and update the tp variable.
|
2007-05-28 11:35:40 +00:00
|
|
|
*/
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WUNLOCK(inp); /* listen socket */
|
2007-05-28 11:35:40 +00:00
|
|
|
inp = sotoinpcb(so);
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK(inp); /* new connection */
|
2007-05-28 11:35:40 +00:00
|
|
|
tp = intotcpcb(inp);
|
2007-05-28 23:27:44 +00:00
|
|
|
KASSERT(tp->t_state == TCPS_SYN_RECEIVED,
|
|
|
|
("%s: ", __func__));
|
2011-04-25 17:13:40 +00:00
|
|
|
#ifdef TCP_SIGNATURE
|
|
|
|
if (sig_checked == 0) {
|
|
|
|
tcp_dooptions(&to, optp, optlen,
|
|
|
|
(thflags & TH_SYN) ? TO_SYN : 0);
|
|
|
|
if (!tcp_signature_verify_input(m, off0, tlen,
|
|
|
|
optlen, &to, th, tp->t_flags)) {
|
|
|
|
|
|
|
|
/*
|
|
|
|
* In SYN_SENT state if it receives an
|
|
|
|
* RST, it is allowed for further
|
|
|
|
* processing.
|
|
|
|
*/
|
|
|
|
if ((thflags & TH_RST) == 0 ||
|
|
|
|
(tp->t_state == TCPS_SYN_SENT) == 0)
|
|
|
|
goto dropunlock;
|
|
|
|
}
|
|
|
|
sig_checked = 1;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2007-05-28 11:35:40 +00:00
|
|
|
/*
|
|
|
|
* Process the segment and the data it
|
|
|
|
* contains. tcp_do_segment() consumes
|
|
|
|
* the mbuf chain and unlocks the inpcb.
|
|
|
|
*/
|
2008-07-31 15:10:09 +00:00
|
|
|
tcp_do_segment(m, th, so, tp, drop_hdrlen, tlen,
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
iptos, ti_locked);
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
INP_INFO_UNLOCK_ASSERT(&V_tcbinfo);
|
2007-05-28 11:35:40 +00:00
|
|
|
return;
|
2007-05-28 11:03:53 +00:00
|
|
|
}
|
|
|
|
/*
|
2007-05-28 23:27:44 +00:00
|
|
|
* Segment flag validation for new connection attempts:
|
|
|
|
*
|
2007-05-28 11:03:53 +00:00
|
|
|
* Our (SYN|ACK) response was rejected.
|
|
|
|
* Check with syncache and remove entry to prevent
|
|
|
|
* retransmits.
|
2007-07-28 11:51:44 +00:00
|
|
|
*
|
|
|
|
* NB: syncache_chkrst does its own logging of failure
|
|
|
|
* causes.
|
2007-05-28 11:03:53 +00:00
|
|
|
*/
|
|
|
|
if (thflags & TH_RST) {
|
2007-07-28 11:51:44 +00:00
|
|
|
syncache_chkrst(&inc, th);
|
2007-05-28 11:03:53 +00:00
|
|
|
goto dropunlock;
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* We can't do anything without SYN.
|
|
|
|
*/
|
|
|
|
if ((thflags & TH_SYN) == 0) {
|
|
|
|
if ((s = tcp_log_addrs(&inc, th, NULL, NULL)))
|
|
|
|
log(LOG_DEBUG, "%s; %s: Listen socket: "
|
2007-07-28 12:20:39 +00:00
|
|
|
"SYN is missing, segment ignored\n",
|
2007-05-28 23:27:44 +00:00
|
|
|
s, __func__);
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_badsyn);
|
2007-05-28 11:03:53 +00:00
|
|
|
goto dropunlock;
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* (SYN|ACK) is bogus on a listen socket.
|
|
|
|
*/
|
|
|
|
if (thflags & TH_ACK) {
|
|
|
|
if ((s = tcp_log_addrs(&inc, th, NULL, NULL)))
|
|
|
|
log(LOG_DEBUG, "%s; %s: Listen socket: "
|
2007-05-28 23:27:44 +00:00
|
|
|
"SYN|ACK invalid, segment rejected\n",
|
|
|
|
s, __func__);
|
2007-05-28 11:03:53 +00:00
|
|
|
syncache_badack(&inc); /* XXX: Not needed! */
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_badsyn);
|
2007-05-28 11:03:53 +00:00
|
|
|
rstreason = BANDLIM_RST_OPENPORT;
|
|
|
|
goto dropwithreset;
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* If the drop_synfin option is enabled, drop all
|
|
|
|
* segments with both the SYN and FIN bits set.
|
|
|
|
* This prevents e.g. nmap from identifying the
|
|
|
|
* TCP/IP stack.
|
|
|
|
* XXX: Poor reasoning. nmap has other methods
|
|
|
|
* and is constantly refining its stack detection
|
|
|
|
* strategies.
|
2007-05-28 23:27:44 +00:00
|
|
|
* XXX: This is a violation of the TCP specification
|
2007-05-28 11:03:53 +00:00
|
|
|
* and was used by RFC1644.
|
|
|
|
*/
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
if ((thflags & TH_FIN) && V_drop_synfin) {
|
2007-05-28 11:03:53 +00:00
|
|
|
if ((s = tcp_log_addrs(&inc, th, NULL, NULL)))
|
|
|
|
log(LOG_DEBUG, "%s; %s: Listen socket: "
|
2007-07-28 12:20:39 +00:00
|
|
|
"SYN|FIN segment ignored (based on "
|
2007-05-28 23:27:44 +00:00
|
|
|
"sysctl setting)\n", s, __func__);
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_badsyn);
|
2011-01-07 21:40:34 +00:00
|
|
|
goto dropunlock;
|
2007-05-28 11:03:53 +00:00
|
|
|
}
|
2001-11-22 04:50:44 +00:00
|
|
|
/*
|
|
|
|
* Segment's flags are (SYN) or (SYN|FIN).
|
2007-05-28 11:03:53 +00:00
|
|
|
*
|
|
|
|
* TH_PUSH, TH_URG, TH_ECE, TH_CWR are ignored
|
|
|
|
* as they do not affect the state of the TCP FSM.
|
|
|
|
* The data pointed to by TH_URG and th_urp is ignored.
|
2001-11-22 04:50:44 +00:00
|
|
|
*/
|
2007-05-28 11:03:53 +00:00
|
|
|
KASSERT((thflags & (TH_RST|TH_ACK)) == 0,
|
|
|
|
("%s: Listen socket: TH_RST or TH_ACK set", __func__));
|
|
|
|
KASSERT(thflags & (TH_SYN),
|
|
|
|
("%s: Listen socket: TH_SYN not set", __func__));
|
2001-06-11 12:39:29 +00:00
|
|
|
#ifdef INET6
|
2001-11-22 04:50:44 +00:00
|
|
|
/*
|
|
|
|
* If deprecated address is forbidden,
|
|
|
|
* we do not accept SYN to deprecated interface
|
|
|
|
* address to prevent any new inbound connection from
|
|
|
|
* getting established.
|
|
|
|
* When we do not accept SYN, we send a TCP RST,
|
|
|
|
* with deprecated source address (instead of dropping
|
|
|
|
* it). We compromise it as it is much better for peer
|
|
|
|
* to send a RST, and RST will be the final packet
|
|
|
|
* for the exchange.
|
|
|
|
*
|
|
|
|
* If we do not forbid deprecated addresses, we accept
|
|
|
|
* the SYN packet. RFC2462 does not suggest dropping
|
|
|
|
* SYN in this case.
|
|
|
|
* If we decipher RFC2462 5.5.4, it says like this:
|
|
|
|
* 1. use of deprecated addr with existing
|
|
|
|
* communication is okay - "SHOULD continue to be
|
|
|
|
* used"
|
|
|
|
* 2. use of it with new communication:
|
|
|
|
* (2a) "SHOULD NOT be used if alternate address
|
|
|
|
* with sufficient scope is available"
|
|
|
|
* (2b) nothing mentioned otherwise.
|
|
|
|
* Here we fall into (2b) case as we have no choice in
|
|
|
|
* our source address selection - we must obey the peer.
|
|
|
|
*
|
|
|
|
* The wording in RFC2462 is confusing, and there are
|
|
|
|
* multiple description text for deprecated address
|
|
|
|
* handling - worse, they are not exactly the same.
|
|
|
|
* I believe 5.5.4 is the best one, so we follow 5.5.4.
|
|
|
|
*/
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
if (isipv6 && !V_ip6_use_deprecated) {
|
2001-11-22 04:50:44 +00:00
|
|
|
struct in6_ifaddr *ia6;
|
2001-06-11 12:39:29 +00:00
|
|
|
|
2009-06-23 20:19:09 +00:00
|
|
|
ia6 = ip6_getdstifaddr(m);
|
|
|
|
if (ia6 != NULL &&
|
2001-11-22 04:50:44 +00:00
|
|
|
(ia6->ia6_flags & IN6_IFF_DEPRECATED)) {
|
2009-06-23 20:19:09 +00:00
|
|
|
ifa_free(&ia6->ia_ifa);
|
2007-05-28 11:03:53 +00:00
|
|
|
if ((s = tcp_log_addrs(&inc, th, NULL, NULL)))
|
|
|
|
log(LOG_DEBUG, "%s; %s: Listen socket: "
|
2007-05-28 23:27:44 +00:00
|
|
|
"Connection attempt to deprecated "
|
|
|
|
"IPv6 address rejected\n",
|
2007-05-28 11:03:53 +00:00
|
|
|
s, __func__);
|
2001-11-22 04:50:44 +00:00
|
|
|
rstreason = BANDLIM_RST_OPENPORT;
|
|
|
|
goto dropwithreset;
|
1996-02-26 21:47:13 +00:00
|
|
|
}
|
2012-06-04 18:43:51 +00:00
|
|
|
if (ia6)
|
|
|
|
ifa_free(&ia6->ia_ifa);
|
2001-11-22 04:50:44 +00:00
|
|
|
}
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif /* INET6 */
|
2001-11-22 04:50:44 +00:00
|
|
|
/*
|
2007-03-21 18:49:43 +00:00
|
|
|
* Basic sanity checks on incoming SYN requests:
|
2007-05-28 23:27:44 +00:00
|
|
|
* Don't respond if the destination is a link layer
|
|
|
|
* broadcast according to RFC1122 4.2.3.10, p. 104.
|
|
|
|
* If it is from this socket it must be forged.
|
|
|
|
* Don't respond if the source or destination is a
|
|
|
|
* global or subnet broad- or multicast address.
|
|
|
|
* Note that it is quite possible to receive unicast
|
|
|
|
* link-layer packets with a broadcast IP address. Use
|
|
|
|
* in_broadcast() to find them.
|
2001-11-22 04:50:44 +00:00
|
|
|
*/
|
2007-05-28 23:27:44 +00:00
|
|
|
if (m->m_flags & (M_BCAST|M_MCAST)) {
|
|
|
|
if ((s = tcp_log_addrs(&inc, th, NULL, NULL)))
|
|
|
|
log(LOG_DEBUG, "%s; %s: Listen socket: "
|
|
|
|
"Connection attempt from broad- or multicast "
|
2007-07-28 12:20:39 +00:00
|
|
|
"link layer address ignored\n", s, __func__);
|
2007-03-23 20:16:50 +00:00
|
|
|
goto dropunlock;
|
2007-05-28 23:27:44 +00:00
|
|
|
}
|
2007-03-21 18:49:43 +00:00
|
|
|
#ifdef INET6
|
2011-04-30 11:21:29 +00:00
|
|
|
if (isipv6) {
|
2007-03-21 18:49:43 +00:00
|
|
|
if (th->th_dport == th->th_sport &&
|
2007-05-28 11:03:53 +00:00
|
|
|
IN6_ARE_ADDR_EQUAL(&ip6->ip6_dst, &ip6->ip6_src)) {
|
|
|
|
if ((s = tcp_log_addrs(&inc, th, NULL, NULL)))
|
|
|
|
log(LOG_DEBUG, "%s; %s: Listen socket: "
|
2007-05-28 23:27:44 +00:00
|
|
|
"Connection attempt to/from self "
|
2007-07-28 12:20:39 +00:00
|
|
|
"ignored\n", s, __func__);
|
2007-03-23 20:16:50 +00:00
|
|
|
goto dropunlock;
|
2007-05-28 11:03:53 +00:00
|
|
|
}
|
2001-11-22 04:50:44 +00:00
|
|
|
if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst) ||
|
2007-05-28 11:03:53 +00:00
|
|
|
IN6_IS_ADDR_MULTICAST(&ip6->ip6_src)) {
|
|
|
|
if ((s = tcp_log_addrs(&inc, th, NULL, NULL)))
|
|
|
|
log(LOG_DEBUG, "%s; %s: Listen socket: "
|
2007-05-28 23:27:44 +00:00
|
|
|
"Connection attempt from/to multicast "
|
2007-07-28 12:20:39 +00:00
|
|
|
"address ignored\n", s, __func__);
|
2007-03-23 20:16:50 +00:00
|
|
|
goto dropunlock;
|
2007-05-28 11:03:53 +00:00
|
|
|
}
|
2011-04-30 11:21:29 +00:00
|
|
|
}
|
2007-03-21 18:49:43 +00:00
|
|
|
#endif
|
2011-04-30 11:21:29 +00:00
|
|
|
#if defined(INET) && defined(INET6)
|
|
|
|
else
|
|
|
|
#endif
|
|
|
|
#ifdef INET
|
|
|
|
{
|
2007-03-21 18:49:43 +00:00
|
|
|
if (th->th_dport == th->th_sport &&
|
2007-05-28 11:03:53 +00:00
|
|
|
ip->ip_dst.s_addr == ip->ip_src.s_addr) {
|
|
|
|
if ((s = tcp_log_addrs(&inc, th, NULL, NULL)))
|
|
|
|
log(LOG_DEBUG, "%s; %s: Listen socket: "
|
2007-05-28 23:27:44 +00:00
|
|
|
"Connection attempt from/to self "
|
2007-07-28 12:20:39 +00:00
|
|
|
"ignored\n", s, __func__);
|
2007-03-23 20:16:50 +00:00
|
|
|
goto dropunlock;
|
2007-05-28 11:03:53 +00:00
|
|
|
}
|
2002-08-17 02:05:25 +00:00
|
|
|
if (IN_MULTICAST(ntohl(ip->ip_dst.s_addr)) ||
|
|
|
|
IN_MULTICAST(ntohl(ip->ip_src.s_addr)) ||
|
|
|
|
ip->ip_src.s_addr == htonl(INADDR_BROADCAST) ||
|
2007-05-28 11:03:53 +00:00
|
|
|
in_broadcast(ip->ip_dst, m->m_pkthdr.rcvif)) {
|
|
|
|
if ((s = tcp_log_addrs(&inc, th, NULL, NULL)))
|
|
|
|
log(LOG_DEBUG, "%s; %s: Listen socket: "
|
2007-05-28 23:27:44 +00:00
|
|
|
"Connection attempt from/to broad- "
|
2007-07-28 12:20:39 +00:00
|
|
|
"or multicast address ignored\n",
|
2007-05-28 11:03:53 +00:00
|
|
|
s, __func__);
|
2007-03-23 20:16:50 +00:00
|
|
|
goto dropunlock;
|
2007-05-28 11:03:53 +00:00
|
|
|
}
|
2002-08-17 02:05:25 +00:00
|
|
|
}
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif
|
2001-11-22 04:50:44 +00:00
|
|
|
/*
|
2007-03-21 18:49:43 +00:00
|
|
|
* SYN appears to be valid. Create compressed TCP state
|
|
|
|
* for syncache.
|
2001-11-22 04:50:44 +00:00
|
|
|
*/
|
2003-08-13 08:46:54 +00:00
|
|
|
#ifdef TCPDEBUG
|
2007-04-20 14:34:54 +00:00
|
|
|
if (so->so_options & SO_DEBUG)
|
|
|
|
tcp_trace(TA_INPUT, ostate, tp,
|
|
|
|
(void *)tcp_saveipgen, &tcp_savetcp, 0);
|
2003-08-13 08:46:54 +00:00
|
|
|
#endif
|
2007-04-20 14:34:54 +00:00
|
|
|
tcp_dooptions(&to, optp, optlen, TO_SYN);
|
2012-06-19 07:34:13 +00:00
|
|
|
syncache_add(&inc, &to, th, inp, &so, m, NULL, NULL);
|
2007-04-20 14:34:54 +00:00
|
|
|
/*
|
|
|
|
* Entry added to syncache and mbuf consumed.
|
2007-05-28 11:03:53 +00:00
|
|
|
* Everything already unlocked by syncache_add().
|
2007-04-20 14:34:54 +00:00
|
|
|
*/
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
INP_INFO_UNLOCK_ASSERT(&V_tcbinfo);
|
2007-04-20 14:34:54 +00:00
|
|
|
return;
|
2002-05-31 11:52:35 +00:00
|
|
|
}
|
2007-03-21 18:49:43 +00:00
|
|
|
|
2011-04-25 17:13:40 +00:00
|
|
|
#ifdef TCP_SIGNATURE
|
|
|
|
if (sig_checked == 0) {
|
|
|
|
tcp_dooptions(&to, optp, optlen,
|
|
|
|
(thflags & TH_SYN) ? TO_SYN : 0);
|
|
|
|
if (!tcp_signature_verify_input(m, off0, tlen, optlen, &to,
|
|
|
|
th, tp->t_flags)) {
|
|
|
|
|
|
|
|
/*
|
|
|
|
* In SYN_SENT state if it receives an RST, it is
|
|
|
|
* allowed for further processing.
|
|
|
|
*/
|
|
|
|
if ((thflags & TH_RST) == 0 ||
|
|
|
|
(tp->t_state == TCPS_SYN_SENT) == 0)
|
|
|
|
goto dropunlock;
|
|
|
|
}
|
|
|
|
sig_checked = 1;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2007-03-23 20:16:50 +00:00
|
|
|
/*
|
2007-05-28 23:27:44 +00:00
|
|
|
* Segment belongs to a connection in SYN_SENT, ESTABLISHED or later
|
|
|
|
* state. tcp_do_segment() always consumes the mbuf chain, unlocks
|
|
|
|
* the inpcb, and unlocks pcbinfo.
|
2007-03-23 20:16:50 +00:00
|
|
|
*/
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
tcp_do_segment(m, th, so, tp, drop_hdrlen, tlen, iptos, ti_locked);
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
INP_INFO_UNLOCK_ASSERT(&V_tcbinfo);
|
2007-03-23 20:16:50 +00:00
|
|
|
return;
|
2001-11-22 04:50:44 +00:00
|
|
|
|
2007-03-23 20:16:50 +00:00
|
|
|
dropwithreset:
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (ti_locked == TI_WLOCKED) {
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WUNLOCK(&V_tcbinfo);
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
ti_locked = TI_UNLOCKED;
|
|
|
|
}
|
|
|
|
#ifdef INVARIANTS
|
|
|
|
else {
|
|
|
|
KASSERT(ti_locked == TI_UNLOCKED, ("%s: dropwithreset "
|
|
|
|
"ti_locked: %d", __func__, ti_locked));
|
|
|
|
INP_INFO_UNLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
}
|
|
|
|
#endif
|
2008-09-25 17:26:54 +00:00
|
|
|
|
|
|
|
if (inp != NULL) {
|
|
|
|
tcp_dropwithreset(m, th, tp, tlen, rstreason);
|
|
|
|
INP_WUNLOCK(inp);
|
2008-10-26 22:03:52 +00:00
|
|
|
} else
|
2008-09-25 17:26:54 +00:00
|
|
|
tcp_dropwithreset(m, th, NULL, tlen, rstreason);
|
2007-03-23 20:16:50 +00:00
|
|
|
m = NULL; /* mbuf chain got consumed. */
|
2008-09-25 17:26:54 +00:00
|
|
|
goto drop;
|
|
|
|
|
2007-03-23 20:16:50 +00:00
|
|
|
dropunlock:
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (ti_locked == TI_WLOCKED) {
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WUNLOCK(&V_tcbinfo);
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
ti_locked = TI_UNLOCKED;
|
|
|
|
}
|
|
|
|
#ifdef INVARIANTS
|
|
|
|
else {
|
|
|
|
KASSERT(ti_locked == TI_UNLOCKED, ("%s: dropunlock "
|
|
|
|
"ti_locked: %d", __func__, ti_locked));
|
|
|
|
INP_INFO_UNLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
}
|
|
|
|
#endif
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
|
2007-04-23 19:41:47 +00:00
|
|
|
if (inp != NULL)
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WUNLOCK(inp);
|
2008-09-25 17:26:54 +00:00
|
|
|
|
2007-03-23 20:16:50 +00:00
|
|
|
drop:
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
INP_INFO_UNLOCK_ASSERT(&V_tcbinfo);
|
2007-05-28 11:03:53 +00:00
|
|
|
if (s != NULL)
|
|
|
|
free(s, M_TCPLOG);
|
2007-03-23 20:16:50 +00:00
|
|
|
if (m != NULL)
|
|
|
|
m_freem(m);
|
|
|
|
}
|
|
|
|
|
2007-05-06 15:16:05 +00:00
|
|
|
static void
|
2007-03-23 20:16:50 +00:00
|
|
|
tcp_do_segment(struct mbuf *m, struct tcphdr *th, struct socket *so,
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
struct tcpcb *tp, int drop_hdrlen, int tlen, uint8_t iptos,
|
|
|
|
int ti_locked)
|
2007-03-23 20:16:50 +00:00
|
|
|
{
|
|
|
|
int thflags, acked, ourfinisacked, needoutput = 0;
|
|
|
|
int rstreason, todrop, win;
|
|
|
|
u_long tiwin;
|
|
|
|
struct tcpopt to;
|
|
|
|
|
2007-03-24 22:15:02 +00:00
|
|
|
#ifdef TCPDEBUG
|
|
|
|
/*
|
|
|
|
* The size of tcp_saveipgen must be the size of the max ip header,
|
|
|
|
* now IPv6.
|
|
|
|
*/
|
|
|
|
u_char tcp_saveipgen[IP6_HDR_LEN];
|
|
|
|
struct tcphdr tcp_savetcp;
|
|
|
|
short ostate = 0;
|
|
|
|
#endif
|
2007-03-23 20:16:50 +00:00
|
|
|
thflags = th->th_flags;
|
2011-01-10 06:12:01 +00:00
|
|
|
tp->sackhint.last_sack_ack = 0;
|
2007-03-23 20:16:50 +00:00
|
|
|
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
/*
|
|
|
|
* If this is either a state-changing packet or current state isn't
|
|
|
|
* established, we require a write lock on tcbinfo. Otherwise, we
|
2012-07-22 17:31:36 +00:00
|
|
|
* allow the tcbinfo to be in either alocked or unlocked, as the
|
|
|
|
* caller may have unnecessarily acquired a write lock due to a race.
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
*/
|
|
|
|
if ((thflags & (TH_SYN | TH_FIN | TH_RST)) != 0 ||
|
|
|
|
tp->t_state != TCPS_ESTABLISHED) {
|
|
|
|
KASSERT(ti_locked == TI_WLOCKED, ("%s ti_locked %d for "
|
|
|
|
"SYN/FIN/RST/!EST", __func__, ti_locked));
|
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
} else {
|
|
|
|
#ifdef INVARIANTS
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (ti_locked == TI_WLOCKED)
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
else {
|
|
|
|
KASSERT(ti_locked == TI_UNLOCKED, ("%s: EST "
|
|
|
|
"ti_locked: %d", __func__, ti_locked));
|
|
|
|
INP_INFO_UNLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
}
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
#endif
|
|
|
|
}
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
2007-05-06 15:16:05 +00:00
|
|
|
KASSERT(tp->t_state > TCPS_LISTEN, ("%s: TCPS_LISTEN",
|
|
|
|
__func__));
|
|
|
|
KASSERT(tp->t_state != TCPS_TIME_WAIT, ("%s: TCPS_TIME_WAIT",
|
|
|
|
__func__));
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Segment received on connection.
|
|
|
|
* Reset idle time and keep-alive timer.
|
2007-06-10 20:59:22 +00:00
|
|
|
* XXX: This should be done after segment
|
|
|
|
* validation to ignore broken/spoofed segs.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1999-08-30 21:17:07 +00:00
|
|
|
tp->t_rcvtime = ticks;
|
1996-09-13 18:47:03 +00:00
|
|
|
if (TCPS_HAVEESTABLISHED(tp->t_state))
|
2012-02-05 16:53:02 +00:00
|
|
|
tcp_timer_activate(tp, TT_KEEP, TP_KEEPIDLE(tp));
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2006-02-28 23:05:59 +00:00
|
|
|
/*
|
|
|
|
* Unscale the window into a 32-bit value.
|
2007-06-10 20:59:22 +00:00
|
|
|
* For the SYN_SENT state the scale is zero.
|
2006-02-28 23:05:59 +00:00
|
|
|
*/
|
|
|
|
tiwin = th->th_win << tp->snd_scale;
|
|
|
|
|
2008-07-31 15:10:09 +00:00
|
|
|
/*
|
|
|
|
* TCP ECN processing.
|
|
|
|
*/
|
|
|
|
if (tp->t_flags & TF_ECN_PERMIT) {
|
2010-04-10 12:47:06 +00:00
|
|
|
if (thflags & TH_CWR)
|
|
|
|
tp->t_flags &= ~TF_ECN_SND_ECE;
|
2008-07-31 15:10:09 +00:00
|
|
|
switch (iptos & IPTOS_ECN_MASK) {
|
|
|
|
case IPTOS_ECN_CE:
|
|
|
|
tp->t_flags |= TF_ECN_SND_ECE;
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_ecn_ce);
|
2008-07-31 15:10:09 +00:00
|
|
|
break;
|
|
|
|
case IPTOS_ECN_ECT0:
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_ecn_ect0);
|
2008-07-31 15:10:09 +00:00
|
|
|
break;
|
|
|
|
case IPTOS_ECN_ECT1:
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_ecn_ect1);
|
2008-07-31 15:10:09 +00:00
|
|
|
break;
|
|
|
|
}
|
2010-11-12 06:41:55 +00:00
|
|
|
/* Congestion experienced. */
|
|
|
|
if (thflags & TH_ECE) {
|
|
|
|
cc_cong_signal(tp, th, CC_ECN);
|
2008-07-31 15:10:09 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2006-06-26 15:35:25 +00:00
|
|
|
/*
|
|
|
|
* Parse options on any incoming segment.
|
|
|
|
*/
|
2007-03-23 20:16:50 +00:00
|
|
|
tcp_dooptions(&to, (u_char *)(th + 1),
|
|
|
|
(th->th_off << 2) - sizeof(struct tcphdr),
|
|
|
|
(thflags & TH_SYN) ? TO_SYN : 0);
|
2006-06-26 15:35:25 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If echoed timestamp is later than the current time,
|
2006-09-13 13:08:27 +00:00
|
|
|
* fall back to non RFC1323 RTT calculation. Normalize
|
|
|
|
* timestamp if syncookies were used when this connection
|
|
|
|
* was established.
|
2006-06-26 15:35:25 +00:00
|
|
|
*/
|
2006-09-13 13:08:27 +00:00
|
|
|
if ((to.to_flags & TOF_TS) && (to.to_tsecr != 0)) {
|
2006-09-26 01:21:46 +00:00
|
|
|
to.to_tsecr -= tp->ts_offset;
|
2012-02-15 16:09:56 +00:00
|
|
|
if (TSTMP_GT(to.to_tsecr, tcp_ts_getticks()))
|
2006-09-13 13:08:27 +00:00
|
|
|
to.to_tsecr = 0;
|
|
|
|
}
|
2006-06-26 15:35:25 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
2003-11-20 20:07:39 +00:00
|
|
|
* Process options only when we get SYN/ACK back. The SYN case
|
|
|
|
* for incoming connections is handled in tcp_syncache.
|
2007-06-09 21:09:49 +00:00
|
|
|
* According to RFC1323 the window field in a SYN (i.e., a <SYN>
|
|
|
|
* or <SYN,ACK>) segment itself is never scaled.
|
2003-11-20 20:07:39 +00:00
|
|
|
* XXX this is traditional behavior, may need to be cleaned up.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2005-06-29 21:36:49 +00:00
|
|
|
if (tp->t_state == TCPS_SYN_SENT && (thflags & TH_SYN)) {
|
2006-02-28 23:05:59 +00:00
|
|
|
if ((to.to_flags & TOF_SCALE) &&
|
|
|
|
(tp->t_flags & TF_REQ_SCALE)) {
|
2001-11-22 04:50:44 +00:00
|
|
|
tp->t_flags |= TF_RCVD_SCALE;
|
2007-03-15 15:59:28 +00:00
|
|
|
tp->snd_scale = to.to_wscale;
|
2001-11-22 04:50:44 +00:00
|
|
|
}
|
2007-06-09 21:09:49 +00:00
|
|
|
/*
|
|
|
|
* Initial send window. It will be updated with
|
|
|
|
* the next incoming segment to the scaled value.
|
|
|
|
*/
|
|
|
|
tp->snd_wnd = th->th_win;
|
2001-11-22 04:50:44 +00:00
|
|
|
if (to.to_flags & TOF_TS) {
|
|
|
|
tp->t_flags |= TF_RCVD_TSTMP;
|
|
|
|
tp->ts_recent = to.to_tsval;
|
2012-02-15 16:09:56 +00:00
|
|
|
tp->ts_recent_age = tcp_ts_getticks();
|
2001-11-22 04:50:44 +00:00
|
|
|
}
|
|
|
|
if (to.to_flags & TOF_MSS)
|
|
|
|
tcp_mss(tp, to.to_mss);
|
2007-05-06 15:56:31 +00:00
|
|
|
if ((tp->t_flags & TF_SACK_PERMIT) &&
|
|
|
|
(to.to_flags & TOF_SACKPERM) == 0)
|
|
|
|
tp->t_flags &= ~TF_SACK_PERMIT;
|
2004-06-23 21:04:37 +00:00
|
|
|
}
|
|
|
|
|
1995-05-30 08:16:23 +00:00
|
|
|
/*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Header prediction: check for the two common cases
|
|
|
|
* of a uni-directional data xfer. If the packet has
|
|
|
|
* no control flags, is in-sequence, the window didn't
|
|
|
|
* change and we're not retransmitting, it's a
|
|
|
|
* candidate. If the length is zero and the ack moved
|
|
|
|
* forward, we're the sender side of the xfer. Just
|
|
|
|
* free the data acked & wake any higher level process
|
|
|
|
* that was blocked waiting for space. If the length
|
|
|
|
* is non-zero and the ack didn't move, we're the
|
|
|
|
* receiver side. If we're getting packets in-order
|
|
|
|
* (the reassembly queue is empty), add the data to
|
|
|
|
* the socket buffer and note that we need a delayed ack.
|
1995-02-09 23:13:27 +00:00
|
|
|
* Make sure that the hidden state-flags are also off.
|
2007-05-06 15:23:51 +00:00
|
|
|
* Since we check for TCPS_ESTABLISHED first, it can only
|
1995-02-09 23:13:27 +00:00
|
|
|
* be TH_NEEDSYN.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
|
|
|
if (tp->t_state == TCPS_ESTABLISHED &&
|
2007-05-06 15:23:51 +00:00
|
|
|
th->th_seq == tp->rcv_nxt &&
|
2000-01-09 19:17:30 +00:00
|
|
|
(thflags & (TH_SYN|TH_FIN|TH_RST|TH_URG|TH_ACK)) == TH_ACK &&
|
2007-05-06 15:23:51 +00:00
|
|
|
tp->snd_nxt == tp->snd_max &&
|
|
|
|
tiwin && tiwin == tp->snd_wnd &&
|
1995-02-09 23:13:27 +00:00
|
|
|
((tp->t_flags & (TF_NEEDSYN|TF_NEEDFIN)) == 0) &&
|
2007-05-06 15:23:51 +00:00
|
|
|
LIST_EMPTY(&tp->t_segq) &&
|
2001-11-22 04:50:44 +00:00
|
|
|
((to.to_flags & TOF_TS) == 0 ||
|
2007-05-06 15:23:51 +00:00
|
|
|
TSTMP_GEQ(to.to_tsval, tp->ts_recent)) ) {
|
1994-05-24 10:09:53 +00:00
|
|
|
|
1995-05-30 08:16:23 +00:00
|
|
|
/*
|
1994-05-24 10:09:53 +00:00
|
|
|
* If last ACK falls within this segment's sequence numbers,
|
1995-02-09 23:13:27 +00:00
|
|
|
* record the timestamp.
|
|
|
|
* NOTE that the test is modified according to the latest
|
|
|
|
* proposal of the tcplw@cray.com list (Braden 1993/04/26).
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2001-11-22 04:50:44 +00:00
|
|
|
if ((to.to_flags & TOF_TS) != 0 &&
|
2002-08-17 02:05:25 +00:00
|
|
|
SEQ_LEQ(th->th_seq, tp->last_ack_sent)) {
|
2012-02-15 16:09:56 +00:00
|
|
|
tp->ts_recent_age = tcp_ts_getticks();
|
1995-02-09 23:13:27 +00:00
|
|
|
tp->ts_recent = to.to_tsval;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
2000-01-09 19:17:30 +00:00
|
|
|
if (tlen == 0) {
|
|
|
|
if (SEQ_GT(th->th_ack, tp->snd_una) &&
|
|
|
|
SEQ_LEQ(th->th_ack, tp->snd_max) &&
|
2010-11-12 06:41:55 +00:00
|
|
|
!IN_RECOVERY(tp->t_flags) &&
|
|
|
|
(to.to_flags & TOF_SACK) == 0 &&
|
|
|
|
TAILQ_EMPTY(&tp->snd_holes)) {
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
2007-06-10 20:59:22 +00:00
|
|
|
* This is a pure ack for outstanding data.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (ti_locked == TI_WLOCKED)
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WUNLOCK(&V_tcbinfo);
|
|
|
|
ti_locked = TI_UNLOCKED;
|
|
|
|
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_predack);
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
|
1999-08-30 21:17:07 +00:00
|
|
|
/*
|
2007-06-10 20:59:22 +00:00
|
|
|
* "bad retransmit" recovery.
|
1999-08-30 21:17:07 +00:00
|
|
|
*/
|
|
|
|
if (tp->t_rxtshift == 1 &&
|
2011-04-29 15:40:12 +00:00
|
|
|
tp->t_flags & TF_PREVVALID &&
|
2009-06-16 19:00:12 +00:00
|
|
|
(int)(ticks - tp->t_badrxtwin) < 0) {
|
2010-11-12 06:41:55 +00:00
|
|
|
cc_cong_signal(tp, th, CC_RTO_ERR);
|
1999-08-30 21:17:07 +00:00
|
|
|
}
|
Guido reported an interesting bug where an FTP connection between a
Windows 2000 box and a FreeBSD box could stall. The problem turned out
to be a timestamp reply bug in the W2K TCP stack. FreeBSD sends a
timestamp with the SYN, W2K returns a timestamp of 0 in the SYN+ACK
causing FreeBSD to calculate an insane SRTT and RTT, resulting in
a maximal retransmit timeout (60 seconds). If there is any packet
loss on the connection for the first six or so packets the retransmit
case may be hit (the window will still be too small for fast-retransmit),
causing a 60+ second pause. The W2K box gives up and closes the
connection.
This commit works around the W2K bug.
15:04:59.374588 FREEBSD.20 > W2K.1036: S 1420807004:1420807004(0) win 65535 <mss 1460,nop,wscale 2,nop,nop,timestamp 188297344 0> (DF) [tos 0x8]
15:04:59.377558 W2K.1036 > FREEBSD.20: S 4134611565:4134611565(0) ack 1420807005 win 17520 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0> (DF)
Bug reported by: Guido van Rooij <guido@gvr.org>
2002-09-17 22:21:37 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Recalculate the transmit timer / rtt.
|
|
|
|
*
|
|
|
|
* Some boxes send broken timestamp replies
|
2004-08-16 18:32:07 +00:00
|
|
|
* during the SYN+ACK phase, ignore
|
Guido reported an interesting bug where an FTP connection between a
Windows 2000 box and a FreeBSD box could stall. The problem turned out
to be a timestamp reply bug in the W2K TCP stack. FreeBSD sends a
timestamp with the SYN, W2K returns a timestamp of 0 in the SYN+ACK
causing FreeBSD to calculate an insane SRTT and RTT, resulting in
a maximal retransmit timeout (60 seconds). If there is any packet
loss on the connection for the first six or so packets the retransmit
case may be hit (the window will still be too small for fast-retransmit),
causing a 60+ second pause. The W2K box gives up and closes the
connection.
This commit works around the W2K bug.
15:04:59.374588 FREEBSD.20 > W2K.1036: S 1420807004:1420807004(0) win 65535 <mss 1460,nop,wscale 2,nop,nop,timestamp 188297344 0> (DF) [tos 0x8]
15:04:59.377558 W2K.1036 > FREEBSD.20: S 4134611565:4134611565(0) ack 1420807005 win 17520 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0> (DF)
Bug reported by: Guido van Rooij <guido@gvr.org>
2002-09-17 22:21:37 +00:00
|
|
|
* timestamps of 0 or we could calculate a
|
|
|
|
* huge RTT and blow up the retransmit timer.
|
|
|
|
*/
|
|
|
|
if ((to.to_flags & TOF_TS) != 0 &&
|
|
|
|
to.to_tsecr) {
|
2012-02-15 16:09:56 +00:00
|
|
|
u_int t;
|
|
|
|
|
|
|
|
t = tcp_ts_getticks() - to.to_tsecr;
|
|
|
|
if (!tp->t_rttlow || tp->t_rttlow > t)
|
|
|
|
tp->t_rttlow = t;
|
1995-02-09 23:13:27 +00:00
|
|
|
tcp_xmit_timer(tp,
|
2012-02-15 16:09:56 +00:00
|
|
|
TCP_TS_TO_TICKS(t) + 1);
|
Guido reported an interesting bug where an FTP connection between a
Windows 2000 box and a FreeBSD box could stall. The problem turned out
to be a timestamp reply bug in the W2K TCP stack. FreeBSD sends a
timestamp with the SYN, W2K returns a timestamp of 0 in the SYN+ACK
causing FreeBSD to calculate an insane SRTT and RTT, resulting in
a maximal retransmit timeout (60 seconds). If there is any packet
loss on the connection for the first six or so packets the retransmit
case may be hit (the window will still be too small for fast-retransmit),
causing a 60+ second pause. The W2K box gives up and closes the
connection.
This commit works around the W2K bug.
15:04:59.374588 FREEBSD.20 > W2K.1036: S 1420807004:1420807004(0) win 65535 <mss 1460,nop,wscale 2,nop,nop,timestamp 188297344 0> (DF) [tos 0x8]
15:04:59.377558 W2K.1036 > FREEBSD.20: S 4134611565:4134611565(0) ack 1420807005 win 17520 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0> (DF)
Bug reported by: Guido van Rooij <guido@gvr.org>
2002-09-17 22:21:37 +00:00
|
|
|
} else if (tp->t_rtttime &&
|
2007-03-23 19:11:22 +00:00
|
|
|
SEQ_GT(th->th_ack, tp->t_rtseq)) {
|
2006-02-16 19:38:07 +00:00
|
|
|
if (!tp->t_rttlow ||
|
|
|
|
tp->t_rttlow > ticks - tp->t_rtttime)
|
|
|
|
tp->t_rttlow = ticks - tp->t_rtttime;
|
2002-08-17 02:05:25 +00:00
|
|
|
tcp_xmit_timer(tp,
|
|
|
|
ticks - tp->t_rtttime);
|
Guido reported an interesting bug where an FTP connection between a
Windows 2000 box and a FreeBSD box could stall. The problem turned out
to be a timestamp reply bug in the W2K TCP stack. FreeBSD sends a
timestamp with the SYN, W2K returns a timestamp of 0 in the SYN+ACK
causing FreeBSD to calculate an insane SRTT and RTT, resulting in
a maximal retransmit timeout (60 seconds). If there is any packet
loss on the connection for the first six or so packets the retransmit
case may be hit (the window will still be too small for fast-retransmit),
causing a 60+ second pause. The W2K box gives up and closes the
connection.
This commit works around the W2K bug.
15:04:59.374588 FREEBSD.20 > W2K.1036: S 1420807004:1420807004(0) win 65535 <mss 1460,nop,wscale 2,nop,nop,timestamp 188297344 0> (DF) [tos 0x8]
15:04:59.377558 W2K.1036 > FREEBSD.20: S 4134611565:4134611565(0) ack 1420807005 win 17520 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0> (DF)
Bug reported by: Guido van Rooij <guido@gvr.org>
2002-09-17 22:21:37 +00:00
|
|
|
}
|
2010-11-12 06:41:55 +00:00
|
|
|
acked = BYTES_THIS_ACK(tp, th);
|
2010-12-28 12:13:30 +00:00
|
|
|
|
|
|
|
/* Run HHOOK_TCP_ESTABLISHED_IN helper hooks. */
|
|
|
|
hhook_run_tcp_est_in(tp, th, &to);
|
|
|
|
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvackpack);
|
|
|
|
TCPSTAT_ADD(tcps_rcvackbyte, acked);
|
1994-05-24 10:09:53 +00:00
|
|
|
sbdrop(&so->so_snd, acked);
|
2003-07-15 21:49:53 +00:00
|
|
|
if (SEQ_GT(tp->snd_una, tp->snd_recover) &&
|
|
|
|
SEQ_LEQ(th->th_ack, tp->snd_recover))
|
|
|
|
tp->snd_recover = th->th_ack - 1;
|
2010-11-12 06:41:55 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Let the congestion control algorithm update
|
|
|
|
* congestion control related information. This
|
|
|
|
* typically means increasing the congestion
|
|
|
|
* window.
|
|
|
|
*/
|
|
|
|
cc_ack_received(tp, th, CC_ACK);
|
|
|
|
|
2003-07-15 21:49:53 +00:00
|
|
|
tp->snd_una = th->th_ack;
|
2002-10-31 23:24:13 +00:00
|
|
|
/*
|
2007-06-10 20:59:22 +00:00
|
|
|
* Pull snd_wl2 up to prevent seq wrap relative
|
2002-10-31 23:24:13 +00:00
|
|
|
* to th_ack.
|
|
|
|
*/
|
2003-01-13 11:01:20 +00:00
|
|
|
tp->snd_wl2 = th->th_ack;
|
2002-08-15 17:13:18 +00:00
|
|
|
tp->t_dupacks = 0;
|
1994-05-24 10:09:53 +00:00
|
|
|
m_freem(m);
|
2007-06-10 20:59:22 +00:00
|
|
|
ND6_HINT(tp); /* Some progress has been made. */
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If all outstanding data are acked, stop
|
|
|
|
* retransmit timer, otherwise restart timer
|
|
|
|
* using current (possibly backed-off) value.
|
|
|
|
* If process is waiting for space,
|
|
|
|
* wakeup/selwakeup/signal. If data
|
|
|
|
* are ready to send, let tcp_output
|
|
|
|
* decide between more output or persist.
|
2007-06-10 20:59:22 +00:00
|
|
|
*/
|
2003-08-13 08:46:54 +00:00
|
|
|
#ifdef TCPDEBUG
|
|
|
|
if (so->so_options & SO_DEBUG)
|
|
|
|
tcp_trace(TA_INPUT, ostate, tp,
|
|
|
|
(void *)tcp_saveipgen,
|
|
|
|
&tcp_savetcp, 0);
|
|
|
|
#endif
|
1994-05-24 10:09:53 +00:00
|
|
|
if (tp->snd_una == tp->snd_max)
|
2007-04-11 09:45:16 +00:00
|
|
|
tcp_timer_activate(tp, TT_REXMT, 0);
|
|
|
|
else if (!tcp_timer_active(tp, TT_PERSIST))
|
|
|
|
tcp_timer_activate(tp, TT_REXMT,
|
|
|
|
tp->t_rxtcur);
|
1998-05-31 18:42:49 +00:00
|
|
|
sowwakeup(so);
|
1994-05-24 10:09:53 +00:00
|
|
|
if (so->so_snd.sb_cc)
|
|
|
|
(void) tcp_output(tp);
|
2003-02-22 21:54:57 +00:00
|
|
|
goto check_delack;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
2000-01-09 19:17:30 +00:00
|
|
|
} else if (th->th_ack == tp->snd_una &&
|
|
|
|
tlen <= sbspace(&so->so_rcv)) {
|
2007-02-01 18:32:13 +00:00
|
|
|
int newsize = 0; /* automatic sockbuf scaling */
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
* This is a pure, in-sequence data packet with
|
|
|
|
* nothing on the reassembly queue and we have enough
|
|
|
|
* buffer space to take it.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (ti_locked == TI_WLOCKED)
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WUNLOCK(&V_tcbinfo);
|
|
|
|
ti_locked = TI_UNLOCKED;
|
|
|
|
|
2004-06-23 21:04:37 +00:00
|
|
|
/* Clean receiver SACK report if present */
|
2007-05-06 15:56:31 +00:00
|
|
|
if ((tp->t_flags & TF_SACK_PERMIT) && tp->rcv_numsacks)
|
2004-06-23 21:04:37 +00:00
|
|
|
tcp_clean_sackreport(tp);
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_preddat);
|
2000-01-09 19:17:30 +00:00
|
|
|
tp->rcv_nxt += tlen;
|
2002-10-31 23:24:13 +00:00
|
|
|
/*
|
|
|
|
* Pull snd_wl1 up to prevent seq wrap relative to
|
|
|
|
* th_seq.
|
|
|
|
*/
|
2012-11-05 09:13:06 +00:00
|
|
|
tp->snd_wl1 = th->th_seq;
|
2002-10-31 23:24:13 +00:00
|
|
|
/*
|
|
|
|
* Pull rcv_up up to prevent seq wrap relative to
|
|
|
|
* rcv_nxt.
|
|
|
|
*/
|
|
|
|
tp->rcv_up = tp->rcv_nxt;
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvpack);
|
|
|
|
TCPSTAT_ADD(tcps_rcvbyte, tlen);
|
2007-06-10 20:59:22 +00:00
|
|
|
ND6_HINT(tp); /* Some progress has been made */
|
2003-08-13 08:46:54 +00:00
|
|
|
#ifdef TCPDEBUG
|
|
|
|
if (so->so_options & SO_DEBUG)
|
|
|
|
tcp_trace(TA_INPUT, ostate, tp,
|
|
|
|
(void *)tcp_saveipgen, &tcp_savetcp, 0);
|
|
|
|
#endif
|
2007-02-01 18:32:13 +00:00
|
|
|
/*
|
|
|
|
* Automatic sizing of receive socket buffer. Often the send
|
|
|
|
* buffer size is not optimally adjusted to the actual network
|
|
|
|
* conditions at hand (delay bandwidth product). Setting the
|
|
|
|
* buffer size too small limits throughput on links with high
|
|
|
|
* bandwidth and high delay (eg. trans-continental/oceanic links).
|
|
|
|
*
|
|
|
|
* On the receive side the socket buffer memory is only rarely
|
|
|
|
* used to any significant extent. This allows us to be much
|
|
|
|
* more aggressive in scaling the receive socket buffer. For
|
|
|
|
* the case that the buffer space is actually used to a large
|
|
|
|
* extent and we run out of kernel memory we can simply drop
|
|
|
|
* the new segments; TCP on the sender will just retransmit it
|
|
|
|
* later. Setting the buffer size too big may only consume too
|
|
|
|
* much kernel memory if the application doesn't read() from
|
|
|
|
* the socket or packet loss or reordering makes use of the
|
|
|
|
* reassembly queue.
|
|
|
|
*
|
|
|
|
* The criteria to step up the receive buffer one notch are:
|
|
|
|
* 1. the number of bytes received during the time it takes
|
|
|
|
* one timestamp to be reflected back to us (the RTT);
|
|
|
|
* 2. received bytes per RTT is within seven eighth of the
|
|
|
|
* current socket buffer size;
|
|
|
|
* 3. receive buffer size has not hit maximal automatic size;
|
|
|
|
*
|
|
|
|
* This algorithm does one step per RTT at most and only if
|
|
|
|
* we receive a bulk stream w/o packet losses or reorderings.
|
|
|
|
* Shrinking the buffer during idle times is not necessary as
|
|
|
|
* it doesn't consume any memory when idle.
|
|
|
|
*
|
|
|
|
* TODO: Only step up if the application is actually serving
|
|
|
|
* the buffer to better manage the socket buffer resources.
|
|
|
|
*/
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
if (V_tcp_do_autorcvbuf &&
|
2007-02-01 18:32:13 +00:00
|
|
|
to.to_tsecr &&
|
|
|
|
(so->so_rcv.sb_flags & SB_AUTOSIZE)) {
|
2010-08-27 12:34:53 +00:00
|
|
|
if (TSTMP_GT(to.to_tsecr, tp->rfbuf_ts) &&
|
2007-02-01 18:32:13 +00:00
|
|
|
to.to_tsecr - tp->rfbuf_ts < hz) {
|
|
|
|
if (tp->rfbuf_cnt >
|
|
|
|
(so->so_rcv.sb_hiwat / 8 * 7) &&
|
|
|
|
so->so_rcv.sb_hiwat <
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
V_tcp_autorcvbuf_max) {
|
2007-02-01 18:32:13 +00:00
|
|
|
newsize =
|
|
|
|
min(so->so_rcv.sb_hiwat +
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
V_tcp_autorcvbuf_inc,
|
|
|
|
V_tcp_autorcvbuf_max);
|
2007-02-01 18:32:13 +00:00
|
|
|
}
|
|
|
|
/* Start over with next RTT. */
|
|
|
|
tp->rfbuf_ts = 0;
|
|
|
|
tp->rfbuf_cnt = 0;
|
|
|
|
} else
|
|
|
|
tp->rfbuf_cnt += tlen; /* add up */
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Add data to socket buffer. */
|
Reduce the number of unnecessary unlock-relocks on socket buffer mutexes
associated with performing a wakeup on the socket buffer:
- When performing an sbappend*() followed by a so[rw]wakeup(), explicitly
acquire the socket buffer lock and use the _locked() variants of both
calls. Note that the _locked() sowakeup() versions unlock the mutex on
return. This is done in uipc_send(), divert_packet(), mroute
socket_send(), raw_append(), tcp_reass(), tcp_input(), and udp_append().
- When the socket buffer lock is dropped before a sowakeup(), remove the
explicit unlock and use the _locked() sowakeup() variant. This is done
in soisdisconnecting(), soisdisconnected() when setting the can't send/
receive flags and dropping data, and in uipc_rcvd() which adjusting
back-pressure on the sockets.
For UNIX domain sockets running mpsafe with a contention-intensive SMP
mysql benchmark, this results in a 1.6% query rate improvement due to
reduce mutex costs.
2004-06-26 19:10:39 +00:00
|
|
|
SOCKBUF_LOCK(&so->so_rcv);
|
2004-06-14 18:16:22 +00:00
|
|
|
if (so->so_rcv.sb_state & SBS_CANTRCVMORE) {
|
2002-09-22 02:54:07 +00:00
|
|
|
m_freem(m);
|
|
|
|
} else {
|
2007-02-01 18:32:13 +00:00
|
|
|
/*
|
|
|
|
* Set new socket buffer size.
|
|
|
|
* Give up when limit is reached.
|
|
|
|
*/
|
|
|
|
if (newsize)
|
|
|
|
if (!sbreserve_locked(&so->so_rcv,
|
2008-10-07 09:41:07 +00:00
|
|
|
newsize, so, NULL))
|
2007-02-01 18:32:13 +00:00
|
|
|
so->so_rcv.sb_flags &= ~SB_AUTOSIZE;
|
2002-09-22 02:54:07 +00:00
|
|
|
m_adj(m, drop_hdrlen); /* delayed header drop */
|
Reduce the number of unnecessary unlock-relocks on socket buffer mutexes
associated with performing a wakeup on the socket buffer:
- When performing an sbappend*() followed by a so[rw]wakeup(), explicitly
acquire the socket buffer lock and use the _locked() variants of both
calls. Note that the _locked() sowakeup() versions unlock the mutex on
return. This is done in uipc_send(), divert_packet(), mroute
socket_send(), raw_append(), tcp_reass(), tcp_input(), and udp_append().
- When the socket buffer lock is dropped before a sowakeup(), remove the
explicit unlock and use the _locked() sowakeup() variant. This is done
in soisdisconnecting(), soisdisconnected() when setting the can't send/
receive flags and dropping data, and in uipc_rcvd() which adjusting
back-pressure on the sockets.
For UNIX domain sockets running mpsafe with a contention-intensive SMP
mysql benchmark, this results in a 1.6% query rate improvement due to
reduce mutex costs.
2004-06-26 19:10:39 +00:00
|
|
|
sbappendstream_locked(&so->so_rcv, m);
|
2002-09-22 02:54:07 +00:00
|
|
|
}
|
2007-06-10 20:59:22 +00:00
|
|
|
/* NB: sorwakeup_locked() does an implicit unlock. */
|
Reduce the number of unnecessary unlock-relocks on socket buffer mutexes
associated with performing a wakeup on the socket buffer:
- When performing an sbappend*() followed by a so[rw]wakeup(), explicitly
acquire the socket buffer lock and use the _locked() variants of both
calls. Note that the _locked() sowakeup() versions unlock the mutex on
return. This is done in uipc_send(), divert_packet(), mroute
socket_send(), raw_append(), tcp_reass(), tcp_input(), and udp_append().
- When the socket buffer lock is dropped before a sowakeup(), remove the
explicit unlock and use the _locked() sowakeup() variant. This is done
in soisdisconnecting(), soisdisconnected() when setting the can't send/
receive flags and dropping data, and in uipc_rcvd() which adjusting
back-pressure on the sockets.
For UNIX domain sockets running mpsafe with a contention-intensive SMP
mysql benchmark, this results in a 1.6% query rate improvement due to
reduce mutex costs.
2004-06-26 19:10:39 +00:00
|
|
|
sorwakeup_locked(so);
|
2001-02-25 15:17:24 +00:00
|
|
|
if (DELAY_ACK(tp)) {
|
2003-02-19 21:18:23 +00:00
|
|
|
tp->t_flags |= TF_DELACK;
|
1998-02-26 05:25:39 +00:00
|
|
|
} else {
|
1995-03-27 07:12:24 +00:00
|
|
|
tp->t_flags |= TF_ACKNOW;
|
|
|
|
tcp_output(tp);
|
|
|
|
}
|
2003-02-22 21:54:57 +00:00
|
|
|
goto check_delack;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Calculate amount of space in receive window,
|
|
|
|
* and then do TCP input processing.
|
|
|
|
* Receive window is amount of space in rcv queue,
|
|
|
|
* but not less than advertised window.
|
|
|
|
*/
|
|
|
|
win = sbspace(&so->so_rcv);
|
|
|
|
if (win < 0)
|
|
|
|
win = 0;
|
1997-07-01 05:42:16 +00:00
|
|
|
tp->rcv_wnd = imax(win, (int)(tp->rcv_adv - tp->rcv_nxt));
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2007-02-01 18:32:13 +00:00
|
|
|
/* Reset receive buffer auto scaling when not in bulk receive mode. */
|
|
|
|
tp->rfbuf_ts = 0;
|
|
|
|
tp->rfbuf_cnt = 0;
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
switch (tp->t_state) {
|
|
|
|
|
1998-01-21 02:05:59 +00:00
|
|
|
/*
|
|
|
|
* If the state is SYN_RECEIVED:
|
|
|
|
* if seg contains an ACK, but not for our SYN/ACK, send a RST.
|
|
|
|
*/
|
|
|
|
case TCPS_SYN_RECEIVED:
|
2000-01-09 19:17:30 +00:00
|
|
|
if ((thflags & TH_ACK) &&
|
|
|
|
(SEQ_LEQ(th->th_ack, tp->snd_una) ||
|
2001-02-11 07:39:51 +00:00
|
|
|
SEQ_GT(th->th_ack, tp->snd_max))) {
|
|
|
|
rstreason = BANDLIM_RST_OPENPORT;
|
|
|
|
goto dropwithreset;
|
|
|
|
}
|
1998-01-21 02:05:59 +00:00
|
|
|
break;
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* If the state is SYN_SENT:
|
|
|
|
* if seg contains an ACK, but not for our SYN, drop the input.
|
|
|
|
* if seg contains a RST, then drop the connection.
|
|
|
|
* if seg does not contain SYN, then drop it.
|
|
|
|
* Otherwise this is an acceptable SYN segment
|
|
|
|
* initialize tp->rcv_nxt and tp->irs
|
|
|
|
* if seg contains ack then advance tp->snd_una
|
2008-07-31 15:10:09 +00:00
|
|
|
* if seg contains an ECE and ECN support is enabled, the stream
|
|
|
|
* is ECN capable.
|
1994-05-24 10:09:53 +00:00
|
|
|
* if SYN has been acked change to ESTABLISHED else SYN_RCVD state
|
|
|
|
* arrange for segment to be acked (eventually)
|
|
|
|
* continue processing rest of data/controls, beginning with URG
|
|
|
|
*/
|
|
|
|
case TCPS_SYN_SENT:
|
2000-01-09 19:17:30 +00:00
|
|
|
if ((thflags & TH_ACK) &&
|
|
|
|
(SEQ_LEQ(th->th_ack, tp->iss) ||
|
|
|
|
SEQ_GT(th->th_ack, tp->snd_max))) {
|
2004-11-02 22:22:22 +00:00
|
|
|
rstreason = BANDLIM_UNLIMITED;
|
|
|
|
goto dropwithreset;
|
1995-02-09 23:13:27 +00:00
|
|
|
}
|
2007-05-06 15:41:06 +00:00
|
|
|
if ((thflags & (TH_ACK|TH_RST)) == (TH_ACK|TH_RST))
|
|
|
|
tp = tcp_drop(tp, ECONNREFUSED);
|
|
|
|
if (thflags & TH_RST)
|
1994-05-24 10:09:53 +00:00
|
|
|
goto drop;
|
2007-05-06 15:41:06 +00:00
|
|
|
if (!(thflags & TH_SYN))
|
1994-05-24 10:09:53 +00:00
|
|
|
goto drop;
|
2006-02-28 23:05:59 +00:00
|
|
|
|
2000-01-09 19:17:30 +00:00
|
|
|
tp->irs = th->th_seq;
|
1994-05-24 10:09:53 +00:00
|
|
|
tcp_rcvseqinit(tp);
|
2000-01-09 19:17:30 +00:00
|
|
|
if (thflags & TH_ACK) {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_connects);
|
1995-11-03 22:31:54 +00:00
|
|
|
soisconnected(so);
|
2002-07-31 19:06:49 +00:00
|
|
|
#ifdef MAC
|
2007-10-24 19:04:04 +00:00
|
|
|
mac_socketpeer_set_from_mbuf(m, so);
|
2002-07-31 19:06:49 +00:00
|
|
|
#endif
|
1995-11-03 22:31:54 +00:00
|
|
|
/* Do window scaling on this connection? */
|
|
|
|
if ((tp->t_flags & (TF_RCVD_SCALE|TF_REQ_SCALE)) ==
|
|
|
|
(TF_RCVD_SCALE|TF_REQ_SCALE)) {
|
|
|
|
tp->rcv_scale = tp->request_r_scale;
|
|
|
|
}
|
2011-03-30 12:35:39 +00:00
|
|
|
tp->rcv_adv += imin(tp->rcv_wnd,
|
|
|
|
TCP_MAXWIN << tp->rcv_scale);
|
1995-02-09 23:13:27 +00:00
|
|
|
tp->snd_una++; /* SYN is acked */
|
|
|
|
/*
|
|
|
|
* If there's data, delay ACK; if there's also a FIN
|
|
|
|
* ACKNOW will be turned on later.
|
|
|
|
*/
|
2001-02-25 15:17:24 +00:00
|
|
|
if (DELAY_ACK(tp) && tlen != 0)
|
2007-04-11 09:45:16 +00:00
|
|
|
tcp_timer_activate(tp, TT_DELACK,
|
|
|
|
tcp_delacktime);
|
1995-02-09 23:13:27 +00:00
|
|
|
else
|
|
|
|
tp->t_flags |= TF_ACKNOW;
|
2008-07-31 15:10:09 +00:00
|
|
|
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
if ((thflags & TH_ECE) && V_tcp_do_ecn) {
|
2008-07-31 15:10:09 +00:00
|
|
|
tp->t_flags |= TF_ECN_PERMIT;
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_ecn_shs);
|
2008-07-31 15:10:09 +00:00
|
|
|
}
|
|
|
|
|
1995-02-09 23:13:27 +00:00
|
|
|
/*
|
|
|
|
* Received <SYN,ACK> in SYN_SENT[*] state.
|
|
|
|
* Transitions:
|
|
|
|
* SYN_SENT --> ESTABLISHED
|
|
|
|
* SYN_SENT* --> FIN_WAIT_1
|
|
|
|
*/
|
1999-08-30 21:17:07 +00:00
|
|
|
tp->t_starttime = ticks;
|
1995-02-09 23:13:27 +00:00
|
|
|
if (tp->t_flags & TF_NEEDFIN) {
|
|
|
|
tp->t_state = TCPS_FIN_WAIT_1;
|
|
|
|
tp->t_flags &= ~TF_NEEDFIN;
|
2000-01-09 19:17:30 +00:00
|
|
|
thflags &= ~TH_SYN;
|
1996-09-13 18:47:03 +00:00
|
|
|
} else {
|
1995-02-09 23:13:27 +00:00
|
|
|
tp->t_state = TCPS_ESTABLISHED;
|
2010-11-12 06:41:55 +00:00
|
|
|
cc_conn_init(tp);
|
2012-02-05 16:53:02 +00:00
|
|
|
tcp_timer_activate(tp, TT_KEEP,
|
|
|
|
TP_KEEPIDLE(tp));
|
1996-09-13 18:47:03 +00:00
|
|
|
}
|
1995-02-09 23:13:27 +00:00
|
|
|
} else {
|
2002-08-17 02:05:25 +00:00
|
|
|
/*
|
2004-08-16 18:32:07 +00:00
|
|
|
* Received initial SYN in SYN-SENT[*] state =>
|
|
|
|
* simultaneous open. If segment contains CC option
|
|
|
|
* and there is a cached CC, apply TAO test.
|
|
|
|
* If it succeeds, connection is * half-synchronized.
|
|
|
|
* Otherwise, do 3-way handshake:
|
|
|
|
* SYN-SENT -> SYN-RECEIVED
|
|
|
|
* SYN-SENT* -> SYN-RECEIVED*
|
|
|
|
* If there was no CC option, clear cached CC value.
|
|
|
|
*/
|
2006-02-23 21:14:34 +00:00
|
|
|
tp->t_flags |= (TF_ACKNOW | TF_NEEDSYN);
|
2007-04-11 09:45:16 +00:00
|
|
|
tcp_timer_activate(tp, TT_REXMT, 0);
|
2004-11-02 22:22:22 +00:00
|
|
|
tp->t_state = TCPS_SYN_RECEIVED;
|
1995-05-30 08:16:23 +00:00
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
KASSERT(ti_locked == TI_WLOCKED, ("%s: trimthenstep6: "
|
|
|
|
"ti_locked %d", __func__, ti_locked));
|
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
2004-07-12 19:28:07 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
2000-01-09 19:17:30 +00:00
|
|
|
* Advance th->th_seq to correspond to first data byte.
|
1994-05-24 10:09:53 +00:00
|
|
|
* If data, trim to stay within window,
|
|
|
|
* dropping FIN if necessary.
|
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
th->th_seq++;
|
|
|
|
if (tlen > tp->rcv_wnd) {
|
|
|
|
todrop = tlen - tp->rcv_wnd;
|
1994-05-24 10:09:53 +00:00
|
|
|
m_adj(m, -todrop);
|
2000-01-09 19:17:30 +00:00
|
|
|
tlen = tp->rcv_wnd;
|
|
|
|
thflags &= ~TH_FIN;
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvpackafterwin);
|
|
|
|
TCPSTAT_ADD(tcps_rcvbyteafterwin, todrop);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
2000-01-09 19:17:30 +00:00
|
|
|
tp->snd_wl1 = th->th_seq - 1;
|
|
|
|
tp->rcv_up = th->th_seq;
|
1995-02-09 23:13:27 +00:00
|
|
|
/*
|
2002-08-17 02:05:25 +00:00
|
|
|
* Client side of transaction: already sent SYN and data.
|
|
|
|
* If the remote host used T/TCP to validate the SYN,
|
|
|
|
* our data will be ACK'd; if so, enter normal data segment
|
|
|
|
* processing in the middle of step 5, ack processing.
|
|
|
|
* Otherwise, goto step 6.
|
1995-05-30 08:16:23 +00:00
|
|
|
*/
|
2004-08-16 18:32:07 +00:00
|
|
|
if (thflags & TH_ACK)
|
1995-02-09 23:13:27 +00:00
|
|
|
goto process_ACK;
|
2002-08-17 02:05:25 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
goto step6;
|
2002-08-17 02:05:25 +00:00
|
|
|
|
1995-02-09 23:13:27 +00:00
|
|
|
/*
|
|
|
|
* If the state is LAST_ACK or CLOSING or TIME_WAIT:
|
2004-11-02 22:22:22 +00:00
|
|
|
* do normal processing.
|
1995-02-09 23:13:27 +00:00
|
|
|
*
|
2004-11-02 22:22:22 +00:00
|
|
|
* NB: Leftover from RFC1644 T/TCP. Cases to be reused later.
|
1995-05-30 08:16:23 +00:00
|
|
|
*/
|
1995-02-09 23:13:27 +00:00
|
|
|
case TCPS_LAST_ACK:
|
|
|
|
case TCPS_CLOSING:
|
2004-08-16 18:32:07 +00:00
|
|
|
break; /* continue normal processing */
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* States other than LISTEN or SYN_SENT.
|
1998-09-11 16:04:03 +00:00
|
|
|
* First check the RST flag and sequence number since reset segments
|
|
|
|
* are exempt from the timestamp and connection count tests. This
|
|
|
|
* fixes a bug introduced by the Stevens, vol. 2, p. 960 bugfix
|
|
|
|
* below which allowed reset segments in half the sequence space
|
|
|
|
* to fall though and be processed (which gives forged reset
|
|
|
|
* segments with a random sequence number a 50 percent chance of
|
|
|
|
* killing a connection).
|
|
|
|
* Then check timestamp, if present.
|
1995-02-09 23:13:27 +00:00
|
|
|
* Then check the connection count, if present.
|
1995-05-30 08:16:23 +00:00
|
|
|
* Then check that at least some bytes of segment are within
|
1994-05-24 10:09:53 +00:00
|
|
|
* receive window. If segment begins before rcv_nxt,
|
|
|
|
* drop leading data (and SYN); if nothing left, just ack.
|
1995-02-09 23:13:27 +00:00
|
|
|
*
|
1998-09-11 16:04:03 +00:00
|
|
|
*
|
|
|
|
* If the RST bit is set, check the sequence number to see
|
|
|
|
* if this is a valid reset segment.
|
|
|
|
* RFC 793 page 37:
|
|
|
|
* In all states except SYN-SENT, all reset (RST) segments
|
|
|
|
* are validated by checking their SEQ-fields. A reset is
|
|
|
|
* valid if its sequence number is in the window.
|
|
|
|
* Note: this does not take into account delayed ACKs, so
|
|
|
|
* we should test against last_ack_sent instead of rcv_nxt.
|
1999-12-11 04:05:52 +00:00
|
|
|
* The sequence number in the reset segment is normally an
|
|
|
|
* echo of our outgoing acknowlegement numbers, but some hosts
|
|
|
|
* send a reset with the sequence number at the rightmost edge
|
|
|
|
* of our receive window, and we have to handle this case.
|
2004-04-26 02:56:31 +00:00
|
|
|
* Note 2: Paul Watson's paper "Slipping in the Window" has shown
|
|
|
|
* that brute force RST attacks are possible. To combat this,
|
|
|
|
* we use a much stricter check while in the ESTABLISHED state,
|
|
|
|
* only accepting RSTs where the sequence number is equal to
|
|
|
|
* last_ack_sent. In all other states (the states in which a
|
2004-08-16 18:32:07 +00:00
|
|
|
* RST is more likely), the more permissive check is used.
|
2008-11-06 16:30:20 +00:00
|
|
|
* If we have multiple segments in flight, the initial reset
|
1998-09-11 16:04:03 +00:00
|
|
|
* segment sequence numbers will be to the left of last_ack_sent,
|
|
|
|
* but they will eventually catch up.
|
|
|
|
* In any case, it never made sense to trim reset segments to
|
|
|
|
* fit the receive window since RFC 1122 says:
|
|
|
|
* 4.2.2.12 RST Segment: RFC-793 Section 3.4
|
|
|
|
*
|
|
|
|
* A TCP SHOULD allow a received RST segment to include data.
|
|
|
|
*
|
|
|
|
* DISCUSSION
|
|
|
|
* It has been suggested that a RST segment could contain
|
|
|
|
* ASCII text that encoded and explained the cause of the
|
|
|
|
* RST. No standard has yet been established for such
|
|
|
|
* data.
|
|
|
|
*
|
|
|
|
* If the reset segment passes the sequence number test examine
|
|
|
|
* the state:
|
|
|
|
* SYN_RECEIVED STATE:
|
|
|
|
* If passive open, return to LISTEN state.
|
|
|
|
* If active open, inform user that connection was refused.
|
2001-08-23 22:34:29 +00:00
|
|
|
* ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, CLOSE_WAIT STATES:
|
1998-09-11 16:04:03 +00:00
|
|
|
* Inform user that connection was reset, and close tcb.
|
1999-07-18 14:42:48 +00:00
|
|
|
* CLOSING, LAST_ACK STATES:
|
1998-09-11 16:04:03 +00:00
|
|
|
* Close the tcb.
|
1999-07-18 14:42:48 +00:00
|
|
|
* TIME_WAIT STATE:
|
1998-09-11 16:04:03 +00:00
|
|
|
* Drop the segment - see Stevens, vol. 2, p. 964 and
|
|
|
|
* RFC 1337.
|
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
if (thflags & TH_RST) {
|
2007-03-07 23:21:59 +00:00
|
|
|
if (SEQ_GEQ(th->th_seq, tp->last_ack_sent - 1) &&
|
|
|
|
SEQ_LEQ(th->th_seq, tp->last_ack_sent + tp->rcv_wnd)) {
|
1998-09-11 16:04:03 +00:00
|
|
|
switch (tp->t_state) {
|
|
|
|
|
|
|
|
case TCPS_SYN_RECEIVED:
|
|
|
|
so->so_error = ECONNREFUSED;
|
|
|
|
goto close;
|
|
|
|
|
|
|
|
case TCPS_ESTABLISHED:
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
if (V_tcp_insecure_rst == 0 &&
|
2007-03-07 23:21:59 +00:00
|
|
|
!(SEQ_GEQ(th->th_seq, tp->rcv_nxt - 1) &&
|
|
|
|
SEQ_LEQ(th->th_seq, tp->rcv_nxt + 1)) &&
|
|
|
|
!(SEQ_GEQ(th->th_seq, tp->last_ack_sent - 1) &&
|
|
|
|
SEQ_LEQ(th->th_seq, tp->last_ack_sent + 1))) {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_badrst);
|
2004-04-26 02:56:31 +00:00
|
|
|
goto drop;
|
|
|
|
}
|
2007-07-25 18:48:24 +00:00
|
|
|
/* FALLTHROUGH */
|
1998-09-11 16:04:03 +00:00
|
|
|
case TCPS_FIN_WAIT_1:
|
|
|
|
case TCPS_FIN_WAIT_2:
|
|
|
|
case TCPS_CLOSE_WAIT:
|
|
|
|
so->so_error = ECONNRESET;
|
|
|
|
close:
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
KASSERT(ti_locked == TI_WLOCKED,
|
|
|
|
("tcp_do_segment: TH_RST 1 ti_locked %d",
|
|
|
|
ti_locked));
|
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
|
1998-09-11 16:04:03 +00:00
|
|
|
tp->t_state = TCPS_CLOSED;
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_drops);
|
1998-09-11 16:04:03 +00:00
|
|
|
tp = tcp_close(tp);
|
|
|
|
break;
|
|
|
|
|
|
|
|
case TCPS_CLOSING:
|
|
|
|
case TCPS_LAST_ACK:
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
KASSERT(ti_locked == TI_WLOCKED,
|
|
|
|
("tcp_do_segment: TH_RST 2 ti_locked %d",
|
|
|
|
ti_locked));
|
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
|
1998-09-11 16:04:03 +00:00
|
|
|
tp = tcp_close(tp);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
goto drop;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
1994-05-24 10:09:53 +00:00
|
|
|
* RFC 1323 PAWS: If we have a timestamp reply on this segment
|
|
|
|
* and it's less than ts_recent, drop it.
|
|
|
|
*/
|
2001-11-22 04:50:44 +00:00
|
|
|
if ((to.to_flags & TOF_TS) != 0 && tp->ts_recent &&
|
1998-09-11 16:04:03 +00:00
|
|
|
TSTMP_LT(to.to_tsval, tp->ts_recent)) {
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/* Check to see if ts_recent is over 24 days old. */
|
2012-02-15 16:09:56 +00:00
|
|
|
if (tcp_ts_getticks() - tp->ts_recent_age > TCP_PAWS_IDLE) {
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* Invalidate ts_recent. If this segment updates
|
|
|
|
* ts_recent, the age will be reset later and ts_recent
|
|
|
|
* will get a valid value. If it does not, setting
|
|
|
|
* ts_recent to zero will at least satisfy the
|
|
|
|
* requirement that zero be placed in the timestamp
|
|
|
|
* echo reply when ts_recent isn't valid. The
|
|
|
|
* age isn't reset until we get a valid ts_recent
|
|
|
|
* because we don't want out-of-order segments to be
|
|
|
|
* dropped when ts_recent is old.
|
|
|
|
*/
|
|
|
|
tp->ts_recent = 0;
|
|
|
|
} else {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvduppack);
|
|
|
|
TCPSTAT_ADD(tcps_rcvdupbyte, tlen);
|
|
|
|
TCPSTAT_INC(tcps_pawsdrop);
|
2002-12-30 19:31:04 +00:00
|
|
|
if (tlen)
|
2002-12-17 00:24:48 +00:00
|
|
|
goto dropafterack;
|
2002-12-30 19:31:04 +00:00
|
|
|
goto drop;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
1998-09-11 16:04:03 +00:00
|
|
|
/*
|
|
|
|
* In the SYN-RECEIVED state, validate that the packet belongs to
|
|
|
|
* this connection before trimming the data to fit the receive
|
|
|
|
* window. Check the sequence number versus IRS since we know
|
|
|
|
* the sequence numbers haven't wrapped. This is a partial fix
|
|
|
|
* for the "LAND" DoS attack.
|
|
|
|
*/
|
2001-02-11 07:39:51 +00:00
|
|
|
if (tp->t_state == TCPS_SYN_RECEIVED && SEQ_LT(th->th_seq, tp->irs)) {
|
|
|
|
rstreason = BANDLIM_RST_OPENPORT;
|
|
|
|
goto dropwithreset;
|
|
|
|
}
|
1998-09-11 16:04:03 +00:00
|
|
|
|
2000-01-09 19:17:30 +00:00
|
|
|
todrop = tp->rcv_nxt - th->th_seq;
|
1994-05-24 10:09:53 +00:00
|
|
|
if (todrop > 0) {
|
2009-05-27 17:02:10 +00:00
|
|
|
/*
|
|
|
|
* If this is a duplicate SYN for our current connection,
|
|
|
|
* advance over it and pretend and it's not a SYN.
|
|
|
|
*/
|
|
|
|
if (thflags & TH_SYN && th->th_seq == tp->irs) {
|
2000-01-09 19:17:30 +00:00
|
|
|
thflags &= ~TH_SYN;
|
|
|
|
th->th_seq++;
|
|
|
|
if (th->th_urp > 1)
|
|
|
|
th->th_urp--;
|
1994-05-24 10:09:53 +00:00
|
|
|
else
|
2000-01-09 19:17:30 +00:00
|
|
|
thflags &= ~TH_URG;
|
1994-05-24 10:09:53 +00:00
|
|
|
todrop--;
|
|
|
|
}
|
1995-02-16 01:39:19 +00:00
|
|
|
/*
|
|
|
|
* Following if statement from Stevens, vol. 2, p. 960.
|
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
if (todrop > tlen
|
|
|
|
|| (todrop == tlen && (thflags & TH_FIN) == 0)) {
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
1995-02-16 01:39:19 +00:00
|
|
|
* Any valid FIN must be to the left of the window.
|
|
|
|
* At this point the FIN must be a duplicate or out
|
|
|
|
* of sequence; drop it.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
thflags &= ~TH_FIN;
|
1995-02-16 01:39:19 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Send an ACK to resynchronize and drop any data.
|
|
|
|
* But keep on processing for RST or ACK.
|
|
|
|
*/
|
|
|
|
tp->t_flags |= TF_ACKNOW;
|
2000-01-09 19:17:30 +00:00
|
|
|
todrop = tlen;
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvduppack);
|
|
|
|
TCPSTAT_ADD(tcps_rcvdupbyte, todrop);
|
1994-05-24 10:09:53 +00:00
|
|
|
} else {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvpartduppack);
|
|
|
|
TCPSTAT_ADD(tcps_rcvpartdupbyte, todrop);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
2000-01-09 19:17:30 +00:00
|
|
|
drop_hdrlen += todrop; /* drop from the top afterwards */
|
|
|
|
th->th_seq += todrop;
|
|
|
|
tlen -= todrop;
|
|
|
|
if (th->th_urp > todrop)
|
|
|
|
th->th_urp -= todrop;
|
1994-05-24 10:09:53 +00:00
|
|
|
else {
|
2000-01-09 19:17:30 +00:00
|
|
|
thflags &= ~TH_URG;
|
|
|
|
th->th_urp = 0;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If new data are received on a connection after the
|
|
|
|
* user processes are gone, then RST the other end.
|
|
|
|
*/
|
|
|
|
if ((so->so_state & SS_NOFDREF) &&
|
2000-01-09 19:17:30 +00:00
|
|
|
tp->t_state > TCPS_CLOSE_WAIT && tlen) {
|
2007-07-28 12:20:39 +00:00
|
|
|
char *s;
|
|
|
|
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
KASSERT(ti_locked == TI_WLOCKED, ("%s: SS_NOFDEREF && "
|
|
|
|
"CLOSE_WAIT && tlen ti_locked %d", __func__, ti_locked));
|
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
|
2007-07-28 12:20:39 +00:00
|
|
|
if ((s = tcp_log_addrs(&tp->t_inpcb->inp_inc, th, NULL, NULL))) {
|
2007-10-07 00:07:27 +00:00
|
|
|
log(LOG_DEBUG, "%s; %s: %s: Received %d bytes of data after socket "
|
2007-07-28 12:20:39 +00:00
|
|
|
"was closed, sending RST and removing tcpcb\n",
|
2007-10-07 00:07:27 +00:00
|
|
|
s, __func__, tcpstates[tp->t_state], tlen);
|
2007-07-28 12:20:39 +00:00
|
|
|
free(s, M_TCPLOG);
|
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
tp = tcp_close(tp);
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvafterclose);
|
2001-02-11 07:39:51 +00:00
|
|
|
rstreason = BANDLIM_UNLIMITED;
|
1994-05-24 10:09:53 +00:00
|
|
|
goto dropwithreset;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If segment ends after window, drop trailing data
|
|
|
|
* (and PUSH and FIN); if nothing left, just ACK.
|
|
|
|
*/
|
2007-05-06 15:41:06 +00:00
|
|
|
todrop = (th->th_seq + tlen) - (tp->rcv_nxt + tp->rcv_wnd);
|
1994-05-24 10:09:53 +00:00
|
|
|
if (todrop > 0) {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvpackafterwin);
|
2000-01-09 19:17:30 +00:00
|
|
|
if (todrop >= tlen) {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_ADD(tcps_rcvbyteafterwin, tlen);
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* If window is closed can only take segments at
|
|
|
|
* window edge, and have to drop data and PUSH from
|
|
|
|
* incoming segments. Continue processing, but
|
|
|
|
* remember to ack. Otherwise, drop segment
|
|
|
|
* and ack.
|
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
if (tp->rcv_wnd == 0 && th->th_seq == tp->rcv_nxt) {
|
1994-05-24 10:09:53 +00:00
|
|
|
tp->t_flags |= TF_ACKNOW;
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvwinprobe);
|
1994-05-24 10:09:53 +00:00
|
|
|
} else
|
|
|
|
goto dropafterack;
|
|
|
|
} else
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_ADD(tcps_rcvbyteafterwin, todrop);
|
1994-05-24 10:09:53 +00:00
|
|
|
m_adj(m, -todrop);
|
2000-01-09 19:17:30 +00:00
|
|
|
tlen -= todrop;
|
|
|
|
thflags &= ~(TH_PUSH|TH_FIN);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If last ACK falls within this segment's sequence numbers,
|
|
|
|
* record its timestamp.
|
2005-04-10 05:24:59 +00:00
|
|
|
* NOTE:
|
|
|
|
* 1) That the test incorporates suggestions from the latest
|
|
|
|
* proposal of the tcplw@cray.com list (Braden 1993/04/26).
|
|
|
|
* 2) That updating only on newer timestamps interferes with
|
|
|
|
* our earlier PAWS tests, so this check should be solely
|
|
|
|
* predicated on the sequence space of this segment.
|
|
|
|
* 3) That we modify the segment boundary check to be
|
|
|
|
* Last.ACK.Sent <= SEG.SEQ + SEG.Len
|
|
|
|
* instead of RFC1323's
|
|
|
|
* Last.ACK.Sent < SEG.SEQ + SEG.Len,
|
|
|
|
* This modified check allows us to overcome RFC1323's
|
|
|
|
* limitations as described in Stevens TCP/IP Illustrated
|
|
|
|
* Vol. 2 p.869. In such cases, we can still calculate the
|
|
|
|
* RTT correctly when RCV.NXT == Last.ACK.Sent.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2001-11-22 04:50:44 +00:00
|
|
|
if ((to.to_flags & TOF_TS) != 0 &&
|
2005-04-10 05:24:59 +00:00
|
|
|
SEQ_LEQ(th->th_seq, tp->last_ack_sent) &&
|
|
|
|
SEQ_LEQ(tp->last_ack_sent, th->th_seq + tlen +
|
|
|
|
((thflags & (TH_SYN|TH_FIN)) != 0))) {
|
2012-02-15 16:09:56 +00:00
|
|
|
tp->ts_recent_age = tcp_ts_getticks();
|
1995-02-09 23:13:27 +00:00
|
|
|
tp->ts_recent = to.to_tsval;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If a SYN is in the window, then this is an
|
|
|
|
* error and we send an RST and drop the connection.
|
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
if (thflags & TH_SYN) {
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
KASSERT(ti_locked == TI_WLOCKED,
|
|
|
|
("tcp_do_segment: TH_SYN ti_locked %d", ti_locked));
|
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
tp = tcp_drop(tp, ECONNRESET);
|
2001-02-11 07:39:51 +00:00
|
|
|
rstreason = BANDLIM_UNLIMITED;
|
2003-11-12 19:38:01 +00:00
|
|
|
goto drop;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
1995-02-09 23:13:27 +00:00
|
|
|
/*
|
|
|
|
* If the ACK bit is off: if in SYN-RECEIVED state or SENDSYN
|
|
|
|
* flag is on (half-synchronized state), then queue data for
|
|
|
|
* later processing; else drop segment and return.
|
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
if ((thflags & TH_ACK) == 0) {
|
1995-02-09 23:13:27 +00:00
|
|
|
if (tp->t_state == TCPS_SYN_RECEIVED ||
|
|
|
|
(tp->t_flags & TF_NEEDSYN))
|
|
|
|
goto step6;
|
2006-06-19 12:33:52 +00:00
|
|
|
else if (tp->t_flags & TF_ACKNOW)
|
|
|
|
goto dropafterack;
|
1995-02-09 23:13:27 +00:00
|
|
|
else
|
|
|
|
goto drop;
|
|
|
|
}
|
1995-05-30 08:16:23 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* Ack processing.
|
|
|
|
*/
|
|
|
|
switch (tp->t_state) {
|
|
|
|
|
|
|
|
/*
|
1998-01-21 02:05:59 +00:00
|
|
|
* In SYN_RECEIVED state, the ack ACKs our SYN, so enter
|
|
|
|
* ESTABLISHED state and continue processing.
|
|
|
|
* The ACK was checked above.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
|
|
|
case TCPS_SYN_RECEIVED:
|
1995-02-09 23:13:27 +00:00
|
|
|
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_connects);
|
1994-05-24 10:09:53 +00:00
|
|
|
soisconnected(so);
|
|
|
|
/* Do window scaling? */
|
|
|
|
if ((tp->t_flags & (TF_RCVD_SCALE|TF_REQ_SCALE)) ==
|
|
|
|
(TF_RCVD_SCALE|TF_REQ_SCALE)) {
|
|
|
|
tp->rcv_scale = tp->request_r_scale;
|
2006-02-28 23:05:59 +00:00
|
|
|
tp->snd_wnd = tiwin;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
1995-02-09 23:13:27 +00:00
|
|
|
/*
|
1995-05-30 08:16:23 +00:00
|
|
|
* Make transitions:
|
1995-02-09 23:13:27 +00:00
|
|
|
* SYN-RECEIVED -> ESTABLISHED
|
|
|
|
* SYN-RECEIVED* -> FIN-WAIT-1
|
|
|
|
*/
|
1999-08-30 21:17:07 +00:00
|
|
|
tp->t_starttime = ticks;
|
1995-02-09 23:13:27 +00:00
|
|
|
if (tp->t_flags & TF_NEEDFIN) {
|
|
|
|
tp->t_state = TCPS_FIN_WAIT_1;
|
|
|
|
tp->t_flags &= ~TF_NEEDFIN;
|
1996-09-13 18:47:03 +00:00
|
|
|
} else {
|
1995-02-09 23:13:27 +00:00
|
|
|
tp->t_state = TCPS_ESTABLISHED;
|
2010-11-12 06:41:55 +00:00
|
|
|
cc_conn_init(tp);
|
2012-02-05 16:53:02 +00:00
|
|
|
tcp_timer_activate(tp, TT_KEEP, TP_KEEPIDLE(tp));
|
1996-09-13 18:47:03 +00:00
|
|
|
}
|
1995-05-30 08:16:23 +00:00
|
|
|
/*
|
1995-02-09 23:13:27 +00:00
|
|
|
* If segment contains data or ACK, will call tcp_reass()
|
|
|
|
* later; if not, do so now to pass queued data to user.
|
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
if (tlen == 0 && (thflags & TH_FIN) == 0)
|
|
|
|
(void) tcp_reass(tp, (struct tcphdr *)0, 0,
|
1995-02-09 23:13:27 +00:00
|
|
|
(struct mbuf *)0);
|
2012-11-05 09:13:06 +00:00
|
|
|
tp->snd_wl1 = th->th_seq - 1;
|
2002-08-25 13:23:09 +00:00
|
|
|
/* FALLTHROUGH */
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* In ESTABLISHED state: drop duplicate ACKs; ACK out of range
|
|
|
|
* ACKs. If the ack is in the range
|
2000-01-09 19:17:30 +00:00
|
|
|
* tp->snd_una < th->th_ack <= tp->snd_max
|
|
|
|
* then advance tp->snd_una to th->th_ack and drop
|
1994-05-24 10:09:53 +00:00
|
|
|
* data from the retransmission queue. If this ACK reflects
|
|
|
|
* more up to date window information we update our window information.
|
|
|
|
*/
|
|
|
|
case TCPS_ESTABLISHED:
|
|
|
|
case TCPS_FIN_WAIT_1:
|
|
|
|
case TCPS_FIN_WAIT_2:
|
|
|
|
case TCPS_CLOSE_WAIT:
|
|
|
|
case TCPS_CLOSING:
|
|
|
|
case TCPS_LAST_ACK:
|
2005-06-27 22:27:42 +00:00
|
|
|
if (SEQ_GT(th->th_ack, tp->snd_max)) {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvacktoomuch);
|
2005-06-27 22:27:42 +00:00
|
|
|
goto dropafterack;
|
|
|
|
}
|
2007-05-06 15:56:31 +00:00
|
|
|
if ((tp->t_flags & TF_SACK_PERMIT) &&
|
2007-03-23 18:33:21 +00:00
|
|
|
((to.to_flags & TOF_SACK) ||
|
|
|
|
!TAILQ_EMPTY(&tp->snd_holes)))
|
2005-06-27 22:27:42 +00:00
|
|
|
tcp_sack_doack(tp, &to, th->th_ack);
|
2010-12-28 12:13:30 +00:00
|
|
|
|
|
|
|
/* Run HHOOK_TCP_ESTABLISHED_IN helper hooks. */
|
|
|
|
hhook_run_tcp_est_in(tp, th, &to);
|
|
|
|
|
2000-01-09 19:17:30 +00:00
|
|
|
if (SEQ_LEQ(th->th_ack, tp->snd_una)) {
|
|
|
|
if (tlen == 0 && tiwin == tp->snd_wnd) {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvdupack);
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* If we have outstanding data (other than
|
|
|
|
* a window probe), this is a completely
|
|
|
|
* duplicate ack (ie, window info didn't
|
|
|
|
* change), the ack is the biggest we've
|
|
|
|
* seen and we've seen exactly our rexmt
|
|
|
|
* threshhold of them, assume a packet
|
|
|
|
* has been dropped and retransmit it.
|
|
|
|
* Kludge snd_nxt & the congestion
|
|
|
|
* window so we send only this one
|
|
|
|
* packet.
|
|
|
|
*
|
|
|
|
* We know we're losing at the current
|
|
|
|
* window size so do congestion avoidance
|
|
|
|
* (set ssthresh to half the current window
|
|
|
|
* and pull our congestion window back to
|
|
|
|
* the new ssthresh).
|
|
|
|
*
|
|
|
|
* Dup acks mean that packets have left the
|
1995-05-30 08:16:23 +00:00
|
|
|
* network (they're now cached at the receiver)
|
1994-05-24 10:09:53 +00:00
|
|
|
* so bump cwnd by the amount in the receiver
|
|
|
|
* to keep a constant cwnd packets in the
|
|
|
|
* network.
|
2008-07-31 15:10:09 +00:00
|
|
|
*
|
|
|
|
* When using TCP ECN, notify the peer that
|
|
|
|
* we reduced the cwnd.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2007-04-11 09:45:16 +00:00
|
|
|
if (!tcp_timer_active(tp, TT_REXMT) ||
|
2000-01-09 19:17:30 +00:00
|
|
|
th->th_ack != tp->snd_una)
|
1994-05-24 10:09:53 +00:00
|
|
|
tp->t_dupacks = 0;
|
2003-01-13 11:01:20 +00:00
|
|
|
else if (++tp->t_dupacks > tcprexmtthresh ||
|
2010-11-12 06:41:55 +00:00
|
|
|
IN_FASTRECOVERY(tp->t_flags)) {
|
|
|
|
cc_ack_received(tp, th, CC_DUPACK);
|
2007-05-06 15:56:31 +00:00
|
|
|
if ((tp->t_flags & TF_SACK_PERMIT) &&
|
2010-11-12 06:41:55 +00:00
|
|
|
IN_FASTRECOVERY(tp->t_flags)) {
|
2005-05-25 17:55:27 +00:00
|
|
|
int awnd;
|
2005-04-14 20:09:52 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Compute the amount of data in flight first.
|
|
|
|
* We can inject new data into the pipe iff
|
2011-01-07 21:40:34 +00:00
|
|
|
* we have less than 1/2 the original window's
|
2005-04-14 20:09:52 +00:00
|
|
|
* worth of data in flight.
|
|
|
|
*/
|
2005-05-25 17:55:27 +00:00
|
|
|
awnd = (tp->snd_nxt - tp->snd_fack) +
|
2005-05-11 21:37:42 +00:00
|
|
|
tp->sackhint.sack_bytes_rexmit;
|
2005-05-25 17:55:27 +00:00
|
|
|
if (awnd < tp->snd_ssthresh) {
|
2005-04-14 20:09:52 +00:00
|
|
|
tp->snd_cwnd += tp->t_maxseg;
|
|
|
|
if (tp->snd_cwnd > tp->snd_ssthresh)
|
|
|
|
tp->snd_cwnd = tp->snd_ssthresh;
|
|
|
|
}
|
|
|
|
} else
|
|
|
|
tp->snd_cwnd += tp->t_maxseg;
|
2012-08-25 09:26:37 +00:00
|
|
|
if ((thflags & TH_FIN) &&
|
|
|
|
(TCPS_HAVERCVDFIN(tp->t_state) == 0)) {
|
|
|
|
/*
|
|
|
|
* If its a fin we need to process
|
|
|
|
* it to avoid a race where both
|
|
|
|
* sides enter FIN-WAIT and send FIN|ACK
|
|
|
|
* at the same time.
|
|
|
|
*/
|
|
|
|
break;
|
|
|
|
}
|
2003-01-13 11:01:20 +00:00
|
|
|
(void) tcp_output(tp);
|
|
|
|
goto drop;
|
|
|
|
} else if (tp->t_dupacks == tcprexmtthresh) {
|
1994-05-24 10:09:53 +00:00
|
|
|
tcp_seq onxt = tp->snd_nxt;
|
2004-07-01 23:34:06 +00:00
|
|
|
|
|
|
|
/*
|
2004-08-16 18:32:07 +00:00
|
|
|
* If we're doing sack, check to
|
|
|
|
* see if we're already in sack
|
2004-07-01 23:34:06 +00:00
|
|
|
* recovery. If we're not doing sack,
|
|
|
|
* check to see if we're in newreno
|
|
|
|
* recovery.
|
|
|
|
*/
|
2007-05-06 15:56:31 +00:00
|
|
|
if (tp->t_flags & TF_SACK_PERMIT) {
|
2010-11-12 06:41:55 +00:00
|
|
|
if (IN_FASTRECOVERY(tp->t_flags)) {
|
2004-07-01 23:34:06 +00:00
|
|
|
tp->t_dupacks = 0;
|
|
|
|
break;
|
|
|
|
}
|
2010-11-12 06:41:55 +00:00
|
|
|
} else {
|
2004-07-01 23:34:06 +00:00
|
|
|
if (SEQ_LEQ(th->th_ack,
|
|
|
|
tp->snd_recover)) {
|
|
|
|
tp->t_dupacks = 0;
|
|
|
|
break;
|
|
|
|
}
|
2000-05-06 03:31:09 +00:00
|
|
|
}
|
2010-11-12 06:41:55 +00:00
|
|
|
/* Congestion signal before ack. */
|
|
|
|
cc_cong_signal(tp, th, CC_NDUPACK);
|
|
|
|
cc_ack_received(tp, th, CC_DUPACK);
|
2007-04-11 09:45:16 +00:00
|
|
|
tcp_timer_activate(tp, TT_REXMT, 0);
|
1999-08-30 21:17:07 +00:00
|
|
|
tp->t_rtttime = 0;
|
2007-05-06 15:56:31 +00:00
|
|
|
if (tp->t_flags & TF_SACK_PERMIT) {
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(
|
|
|
|
tcps_sack_recovery_episode);
|
2004-10-05 18:36:24 +00:00
|
|
|
tp->sack_newdata = tp->snd_nxt;
|
2005-02-14 21:01:08 +00:00
|
|
|
tp->snd_cwnd = tp->t_maxseg;
|
2004-06-23 21:04:37 +00:00
|
|
|
(void) tcp_output(tp);
|
|
|
|
goto drop;
|
|
|
|
}
|
2000-01-09 19:17:30 +00:00
|
|
|
tp->snd_nxt = th->th_ack;
|
1994-05-24 10:09:53 +00:00
|
|
|
tp->snd_cwnd = tp->t_maxseg;
|
2012-08-25 09:26:37 +00:00
|
|
|
if ((thflags & TH_FIN) &&
|
|
|
|
(TCPS_HAVERCVDFIN(tp->t_state) == 0)) {
|
|
|
|
/*
|
|
|
|
* If its a fin we need to process
|
|
|
|
* it to avoid a race where both
|
|
|
|
* sides enter FIN-WAIT and send FIN|ACK
|
|
|
|
* at the same time.
|
|
|
|
*/
|
|
|
|
break;
|
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
(void) tcp_output(tp);
|
2003-04-01 21:16:46 +00:00
|
|
|
KASSERT(tp->snd_limited <= 2,
|
2007-03-23 20:16:50 +00:00
|
|
|
("%s: tp->snd_limited too big",
|
|
|
|
__func__));
|
1994-05-24 10:09:53 +00:00
|
|
|
tp->snd_cwnd = tp->snd_ssthresh +
|
2003-04-01 21:16:46 +00:00
|
|
|
tp->t_maxseg *
|
|
|
|
(tp->t_dupacks - tp->snd_limited);
|
1994-05-24 10:09:53 +00:00
|
|
|
if (SEQ_GT(onxt, tp->snd_nxt))
|
|
|
|
tp->snd_nxt = onxt;
|
|
|
|
goto drop;
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
} else if (V_tcp_do_rfc3042) {
|
2010-11-12 06:41:55 +00:00
|
|
|
cc_ack_received(tp, th, CC_DUPACK);
|
2003-03-12 20:27:28 +00:00
|
|
|
u_long oldcwnd = tp->snd_cwnd;
|
2003-04-01 21:16:46 +00:00
|
|
|
tcp_seq oldsndmax = tp->snd_max;
|
|
|
|
u_int sent;
|
2004-02-25 08:53:17 +00:00
|
|
|
|
2003-03-12 20:27:28 +00:00
|
|
|
KASSERT(tp->t_dupacks == 1 ||
|
|
|
|
tp->t_dupacks == 2,
|
2007-03-23 20:16:50 +00:00
|
|
|
("%s: dupacks not 1 or 2",
|
|
|
|
__func__));
|
2004-01-20 21:40:25 +00:00
|
|
|
if (tp->t_dupacks == 1)
|
2003-04-01 21:16:46 +00:00
|
|
|
tp->snd_limited = 0;
|
2004-01-20 21:40:25 +00:00
|
|
|
tp->snd_cwnd =
|
|
|
|
(tp->snd_nxt - tp->snd_una) +
|
|
|
|
(tp->t_dupacks - tp->snd_limited) *
|
|
|
|
tp->t_maxseg;
|
2012-08-25 09:26:37 +00:00
|
|
|
if ((thflags & TH_FIN) &&
|
|
|
|
(TCPS_HAVERCVDFIN(tp->t_state) == 0)) {
|
|
|
|
/*
|
|
|
|
* If its a fin we need to process
|
|
|
|
* it to avoid a race where both
|
|
|
|
* sides enter FIN-WAIT and send FIN|ACK
|
|
|
|
* at the same time.
|
|
|
|
*/
|
|
|
|
break;
|
|
|
|
}
|
2003-03-12 20:27:28 +00:00
|
|
|
(void) tcp_output(tp);
|
2003-04-01 21:16:46 +00:00
|
|
|
sent = tp->snd_max - oldsndmax;
|
|
|
|
if (sent > tp->t_maxseg) {
|
2004-02-25 08:53:17 +00:00
|
|
|
KASSERT((tp->t_dupacks == 2 &&
|
|
|
|
tp->snd_limited == 0) ||
|
|
|
|
(sent == tp->t_maxseg + 1 &&
|
|
|
|
tp->t_flags & TF_SENTFIN),
|
2007-03-23 20:16:50 +00:00
|
|
|
("%s: sent too much",
|
|
|
|
__func__));
|
2003-04-01 21:16:46 +00:00
|
|
|
tp->snd_limited = 2;
|
|
|
|
} else if (sent > 0)
|
|
|
|
++tp->snd_limited;
|
2003-03-12 20:27:28 +00:00
|
|
|
tp->snd_cwnd = oldcwnd;
|
|
|
|
goto drop;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
} else
|
|
|
|
tp->t_dupacks = 0;
|
|
|
|
break;
|
|
|
|
}
|
2003-01-13 11:01:20 +00:00
|
|
|
|
2007-03-23 20:16:50 +00:00
|
|
|
KASSERT(SEQ_GT(th->th_ack, tp->snd_una),
|
|
|
|
("%s: th_ack <= snd_una", __func__));
|
2003-01-13 11:01:20 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* If the congestion window was inflated to account
|
|
|
|
* for the other side's cached packets, retract it.
|
|
|
|
*/
|
2010-11-12 06:41:55 +00:00
|
|
|
if (IN_FASTRECOVERY(tp->t_flags)) {
|
|
|
|
if (SEQ_LT(th->th_ack, tp->snd_recover)) {
|
|
|
|
if (tp->t_flags & TF_SACK_PERMIT)
|
|
|
|
tcp_sack_partialack(tp, th);
|
|
|
|
else
|
|
|
|
tcp_newreno_partial_ack(tp, th);
|
|
|
|
} else
|
|
|
|
cc_post_recovery(tp, th);
|
2004-08-16 18:32:07 +00:00
|
|
|
}
|
2003-01-13 11:01:20 +00:00
|
|
|
tp->t_dupacks = 0;
|
1995-02-09 23:13:27 +00:00
|
|
|
/*
|
2002-08-17 02:05:25 +00:00
|
|
|
* If we reach this point, ACK is not a duplicate,
|
1995-02-09 23:13:27 +00:00
|
|
|
* i.e., it ACKs something we sent.
|
|
|
|
*/
|
|
|
|
if (tp->t_flags & TF_NEEDSYN) {
|
1995-05-30 08:16:23 +00:00
|
|
|
/*
|
1996-01-31 08:22:24 +00:00
|
|
|
* T/TCP: Connection was half-synchronized, and our
|
|
|
|
* SYN has been ACK'd (so connection is now fully
|
|
|
|
* synchronized). Go to non-starred state,
|
|
|
|
* increment snd_una for ACK of SYN, and check if
|
|
|
|
* we can do window scaling.
|
1995-02-09 23:13:27 +00:00
|
|
|
*/
|
|
|
|
tp->t_flags &= ~TF_NEEDSYN;
|
|
|
|
tp->snd_una++;
|
1996-01-31 08:22:24 +00:00
|
|
|
/* Do window scaling? */
|
|
|
|
if ((tp->t_flags & (TF_RCVD_SCALE|TF_REQ_SCALE)) ==
|
|
|
|
(TF_RCVD_SCALE|TF_REQ_SCALE)) {
|
|
|
|
tp->rcv_scale = tp->request_r_scale;
|
2006-02-28 23:05:59 +00:00
|
|
|
/* Send window already scaled. */
|
1996-01-31 08:22:24 +00:00
|
|
|
}
|
1995-02-09 23:13:27 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
process_ACK:
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
2004-07-12 19:28:07 +00:00
|
|
|
|
2010-11-12 06:41:55 +00:00
|
|
|
acked = BYTES_THIS_ACK(tp, th);
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvackpack);
|
|
|
|
TCPSTAT_ADD(tcps_rcvackbyte, acked);
|
1994-05-24 10:09:53 +00:00
|
|
|
|
1999-08-30 21:17:07 +00:00
|
|
|
/*
|
|
|
|
* If we just performed our first retransmit, and the ACK
|
|
|
|
* arrives within our recovery window, then it was a mistake
|
|
|
|
* to do the retransmit in the first place. Recover our
|
|
|
|
* original cwnd and ssthresh, and proceed to transmit where
|
|
|
|
* we left off.
|
|
|
|
*/
|
2011-04-29 15:40:12 +00:00
|
|
|
if (tp->t_rxtshift == 1 && tp->t_flags & TF_PREVVALID &&
|
|
|
|
(int)(ticks - tp->t_badrxtwin) < 0)
|
2010-11-12 06:41:55 +00:00
|
|
|
cc_cong_signal(tp, th, CC_RTO_ERR);
|
1999-08-30 21:17:07 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* If we have a timestamp reply, update smoothed
|
|
|
|
* round trip time. If no timestamp is present but
|
|
|
|
* transmit timer is running and timed sequence
|
|
|
|
* number was acked, update smoothed round trip time.
|
|
|
|
* Since we now have an rtt measurement, cancel the
|
|
|
|
* timer backoff (cf., Phil Karn's retransmit alg.).
|
|
|
|
* Recompute the initial retransmit timer.
|
Guido reported an interesting bug where an FTP connection between a
Windows 2000 box and a FreeBSD box could stall. The problem turned out
to be a timestamp reply bug in the W2K TCP stack. FreeBSD sends a
timestamp with the SYN, W2K returns a timestamp of 0 in the SYN+ACK
causing FreeBSD to calculate an insane SRTT and RTT, resulting in
a maximal retransmit timeout (60 seconds). If there is any packet
loss on the connection for the first six or so packets the retransmit
case may be hit (the window will still be too small for fast-retransmit),
causing a 60+ second pause. The W2K box gives up and closes the
connection.
This commit works around the W2K bug.
15:04:59.374588 FREEBSD.20 > W2K.1036: S 1420807004:1420807004(0) win 65535 <mss 1460,nop,wscale 2,nop,nop,timestamp 188297344 0> (DF) [tos 0x8]
15:04:59.377558 W2K.1036 > FREEBSD.20: S 4134611565:4134611565(0) ack 1420807005 win 17520 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0> (DF)
Bug reported by: Guido van Rooij <guido@gvr.org>
2002-09-17 22:21:37 +00:00
|
|
|
*
|
|
|
|
* Some boxes send broken timestamp replies
|
2004-08-16 18:32:07 +00:00
|
|
|
* during the SYN+ACK phase, ignore
|
Guido reported an interesting bug where an FTP connection between a
Windows 2000 box and a FreeBSD box could stall. The problem turned out
to be a timestamp reply bug in the W2K TCP stack. FreeBSD sends a
timestamp with the SYN, W2K returns a timestamp of 0 in the SYN+ACK
causing FreeBSD to calculate an insane SRTT and RTT, resulting in
a maximal retransmit timeout (60 seconds). If there is any packet
loss on the connection for the first six or so packets the retransmit
case may be hit (the window will still be too small for fast-retransmit),
causing a 60+ second pause. The W2K box gives up and closes the
connection.
This commit works around the W2K bug.
15:04:59.374588 FREEBSD.20 > W2K.1036: S 1420807004:1420807004(0) win 65535 <mss 1460,nop,wscale 2,nop,nop,timestamp 188297344 0> (DF) [tos 0x8]
15:04:59.377558 W2K.1036 > FREEBSD.20: S 4134611565:4134611565(0) ack 1420807005 win 17520 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0> (DF)
Bug reported by: Guido van Rooij <guido@gvr.org>
2002-09-17 22:21:37 +00:00
|
|
|
* timestamps of 0 or we could calculate a
|
|
|
|
* huge RTT and blow up the retransmit timer.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2012-02-15 16:09:56 +00:00
|
|
|
if ((to.to_flags & TOF_TS) != 0 && to.to_tsecr) {
|
|
|
|
u_int t;
|
|
|
|
|
|
|
|
t = tcp_ts_getticks() - to.to_tsecr;
|
|
|
|
if (!tp->t_rttlow || tp->t_rttlow > t)
|
|
|
|
tp->t_rttlow = t;
|
|
|
|
tcp_xmit_timer(tp, TCP_TS_TO_TICKS(t) + 1);
|
Guido reported an interesting bug where an FTP connection between a
Windows 2000 box and a FreeBSD box could stall. The problem turned out
to be a timestamp reply bug in the W2K TCP stack. FreeBSD sends a
timestamp with the SYN, W2K returns a timestamp of 0 in the SYN+ACK
causing FreeBSD to calculate an insane SRTT and RTT, resulting in
a maximal retransmit timeout (60 seconds). If there is any packet
loss on the connection for the first six or so packets the retransmit
case may be hit (the window will still be too small for fast-retransmit),
causing a 60+ second pause. The W2K box gives up and closes the
connection.
This commit works around the W2K bug.
15:04:59.374588 FREEBSD.20 > W2K.1036: S 1420807004:1420807004(0) win 65535 <mss 1460,nop,wscale 2,nop,nop,timestamp 188297344 0> (DF) [tos 0x8]
15:04:59.377558 W2K.1036 > FREEBSD.20: S 4134611565:4134611565(0) ack 1420807005 win 17520 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0> (DF)
Bug reported by: Guido van Rooij <guido@gvr.org>
2002-09-17 22:21:37 +00:00
|
|
|
} else if (tp->t_rtttime && SEQ_GT(th->th_ack, tp->t_rtseq)) {
|
2006-02-16 19:38:07 +00:00
|
|
|
if (!tp->t_rttlow || tp->t_rttlow > ticks - tp->t_rtttime)
|
|
|
|
tp->t_rttlow = ticks - tp->t_rtttime;
|
1999-08-30 21:17:07 +00:00
|
|
|
tcp_xmit_timer(tp, ticks - tp->t_rtttime);
|
Guido reported an interesting bug where an FTP connection between a
Windows 2000 box and a FreeBSD box could stall. The problem turned out
to be a timestamp reply bug in the W2K TCP stack. FreeBSD sends a
timestamp with the SYN, W2K returns a timestamp of 0 in the SYN+ACK
causing FreeBSD to calculate an insane SRTT and RTT, resulting in
a maximal retransmit timeout (60 seconds). If there is any packet
loss on the connection for the first six or so packets the retransmit
case may be hit (the window will still be too small for fast-retransmit),
causing a 60+ second pause. The W2K box gives up and closes the
connection.
This commit works around the W2K bug.
15:04:59.374588 FREEBSD.20 > W2K.1036: S 1420807004:1420807004(0) win 65535 <mss 1460,nop,wscale 2,nop,nop,timestamp 188297344 0> (DF) [tos 0x8]
15:04:59.377558 W2K.1036 > FREEBSD.20: S 4134611565:4134611565(0) ack 1420807005 win 17520 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0> (DF)
Bug reported by: Guido van Rooij <guido@gvr.org>
2002-09-17 22:21:37 +00:00
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If all outstanding data is acked, stop retransmit
|
|
|
|
* timer and remember to restart (more output or persist).
|
|
|
|
* If there is more data to be acked, restart retransmit
|
|
|
|
* timer, using current (possibly backed-off) value.
|
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
if (th->th_ack == tp->snd_max) {
|
2007-04-11 09:45:16 +00:00
|
|
|
tcp_timer_activate(tp, TT_REXMT, 0);
|
1994-05-24 10:09:53 +00:00
|
|
|
needoutput = 1;
|
2007-04-11 09:45:16 +00:00
|
|
|
} else if (!tcp_timer_active(tp, TT_PERSIST))
|
|
|
|
tcp_timer_activate(tp, TT_REXMT, tp->t_rxtcur);
|
1995-02-09 23:13:27 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If no data (only SYN) was ACK'd,
|
|
|
|
* skip rest of ACK processing.
|
|
|
|
*/
|
|
|
|
if (acked == 0)
|
|
|
|
goto step6;
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
2010-11-12 06:41:55 +00:00
|
|
|
* Let the congestion control algorithm update congestion
|
|
|
|
* control related information. This typically means increasing
|
|
|
|
* the congestion window.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2010-11-12 06:41:55 +00:00
|
|
|
cc_ack_received(tp, th, CC_ACK);
|
|
|
|
|
2004-06-24 03:07:27 +00:00
|
|
|
SOCKBUF_LOCK(&so->so_snd);
|
1994-05-24 10:09:53 +00:00
|
|
|
if (acked > so->so_snd.sb_cc) {
|
2012-11-05 09:13:06 +00:00
|
|
|
tp->snd_wnd -= so->so_snd.sb_cc;
|
2004-06-24 03:07:27 +00:00
|
|
|
sbdrop_locked(&so->so_snd, (int)so->so_snd.sb_cc);
|
1994-05-24 10:09:53 +00:00
|
|
|
ourfinisacked = 1;
|
|
|
|
} else {
|
2004-06-24 03:07:27 +00:00
|
|
|
sbdrop_locked(&so->so_snd, acked);
|
2012-11-05 09:13:06 +00:00
|
|
|
tp->snd_wnd -= acked;
|
1994-05-24 10:09:53 +00:00
|
|
|
ourfinisacked = 0;
|
|
|
|
}
|
2007-07-25 18:48:24 +00:00
|
|
|
/* NB: sowwakeup_locked() does an implicit unlock. */
|
Reduce the number of unnecessary unlock-relocks on socket buffer mutexes
associated with performing a wakeup on the socket buffer:
- When performing an sbappend*() followed by a so[rw]wakeup(), explicitly
acquire the socket buffer lock and use the _locked() variants of both
calls. Note that the _locked() sowakeup() versions unlock the mutex on
return. This is done in uipc_send(), divert_packet(), mroute
socket_send(), raw_append(), tcp_reass(), tcp_input(), and udp_append().
- When the socket buffer lock is dropped before a sowakeup(), remove the
explicit unlock and use the _locked() sowakeup() variant. This is done
in soisdisconnecting(), soisdisconnected() when setting the can't send/
receive flags and dropping data, and in uipc_rcvd() which adjusting
back-pressure on the sockets.
For UNIX domain sockets running mpsafe with a contention-intensive SMP
mysql benchmark, this results in a 1.6% query rate improvement due to
reduce mutex costs.
2004-06-26 19:10:39 +00:00
|
|
|
sowwakeup_locked(so);
|
2007-06-10 20:59:22 +00:00
|
|
|
/* Detect una wraparound. */
|
2010-11-12 06:41:55 +00:00
|
|
|
if (!IN_RECOVERY(tp->t_flags) &&
|
2003-07-15 21:49:53 +00:00
|
|
|
SEQ_GT(tp->snd_una, tp->snd_recover) &&
|
|
|
|
SEQ_LEQ(th->th_ack, tp->snd_recover))
|
|
|
|
tp->snd_recover = th->th_ack - 1;
|
2010-11-12 06:41:55 +00:00
|
|
|
/* XXXLAS: Can this be moved up into cc_post_recovery? */
|
|
|
|
if (IN_RECOVERY(tp->t_flags) &&
|
2009-01-15 06:44:22 +00:00
|
|
|
SEQ_GEQ(th->th_ack, tp->snd_recover)) {
|
2010-11-12 06:41:55 +00:00
|
|
|
EXIT_RECOVERY(tp->t_flags);
|
2009-01-15 06:44:22 +00:00
|
|
|
}
|
2000-01-09 19:17:30 +00:00
|
|
|
tp->snd_una = th->th_ack;
|
2007-05-06 15:56:31 +00:00
|
|
|
if (tp->t_flags & TF_SACK_PERMIT) {
|
2004-06-23 21:04:37 +00:00
|
|
|
if (SEQ_GT(tp->snd_una, tp->snd_recover))
|
|
|
|
tp->snd_recover = tp->snd_una;
|
2004-08-16 18:32:07 +00:00
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
if (SEQ_LT(tp->snd_nxt, tp->snd_una))
|
|
|
|
tp->snd_nxt = tp->snd_una;
|
|
|
|
|
|
|
|
switch (tp->t_state) {
|
|
|
|
|
|
|
|
/*
|
|
|
|
* In FIN_WAIT_1 STATE in addition to the processing
|
|
|
|
* for the ESTABLISHED state if our FIN is now acknowledged
|
|
|
|
* then enter FIN_WAIT_2.
|
|
|
|
*/
|
|
|
|
case TCPS_FIN_WAIT_1:
|
|
|
|
if (ourfinisacked) {
|
|
|
|
/*
|
|
|
|
* If we can't receive any more
|
|
|
|
* data, then closing user can proceed.
|
|
|
|
* Starting the timer is contrary to the
|
|
|
|
* specification, but if we don't get a FIN
|
|
|
|
* we'll hang forever.
|
2007-06-10 20:59:22 +00:00
|
|
|
*
|
|
|
|
* XXXjl:
|
|
|
|
* we should release the tp also, and use a
|
|
|
|
* compressed state.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2004-06-14 18:16:22 +00:00
|
|
|
if (so->so_rcv.sb_state & SBS_CANTRCVMORE) {
|
2002-06-18 07:42:02 +00:00
|
|
|
soisdisconnected(so);
|
2012-02-05 16:53:02 +00:00
|
|
|
tcp_timer_activate(tp, TT_2MSL,
|
|
|
|
(tcp_fast_finwait2_recycle ?
|
|
|
|
tcp_finwait2_timeout :
|
|
|
|
TP_MAXIDLE(tp)));
|
2002-05-31 11:52:35 +00:00
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
tp->t_state = TCPS_FIN_WAIT_2;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
|
2004-08-16 18:32:07 +00:00
|
|
|
/*
|
1994-05-24 10:09:53 +00:00
|
|
|
* In CLOSING STATE in addition to the processing for
|
|
|
|
* the ESTABLISHED state if the ACK acknowledges our FIN
|
|
|
|
* then enter the TIME-WAIT state, otherwise ignore
|
|
|
|
* the segment.
|
|
|
|
*/
|
|
|
|
case TCPS_CLOSING:
|
|
|
|
if (ourfinisacked) {
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
2003-02-24 00:52:03 +00:00
|
|
|
tcp_twstart(tp);
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
INP_INFO_WUNLOCK(&V_tcbinfo);
|
2003-02-19 22:32:43 +00:00
|
|
|
m_freem(m);
|
2007-05-06 15:16:05 +00:00
|
|
|
return;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* In LAST_ACK, we may still be waiting for data to drain
|
|
|
|
* and/or to be acked, as well as for the ack of our FIN.
|
|
|
|
* If our FIN is now acknowledged, delete the TCB,
|
|
|
|
* enter the closed state and return.
|
|
|
|
*/
|
|
|
|
case TCPS_LAST_ACK:
|
|
|
|
if (ourfinisacked) {
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
1994-05-24 10:09:53 +00:00
|
|
|
tp = tcp_close(tp);
|
|
|
|
goto drop;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
step6:
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
2004-07-12 19:28:07 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
2012-11-05 09:13:06 +00:00
|
|
|
* Update window information.
|
|
|
|
* Don't look at window if no ACK: TAC's send garbage on first SYN.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2012-11-05 09:13:06 +00:00
|
|
|
if ((thflags & TH_ACK) &&
|
|
|
|
(SEQ_LT(tp->snd_wl1, th->th_seq) ||
|
|
|
|
(tp->snd_wl1 == th->th_seq && (SEQ_LT(tp->snd_wl2, th->th_ack) ||
|
|
|
|
(tp->snd_wl2 == th->th_ack && tiwin > tp->snd_wnd))))) {
|
|
|
|
/* keep track of pure window updates */
|
|
|
|
if (tlen == 0 &&
|
|
|
|
tp->snd_wl2 == th->th_ack && tiwin > tp->snd_wnd)
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvwinupd);
|
1994-05-24 10:09:53 +00:00
|
|
|
tp->snd_wnd = tiwin;
|
2012-11-05 09:13:06 +00:00
|
|
|
tp->snd_wl1 = th->th_seq;
|
|
|
|
tp->snd_wl2 = th->th_ack;
|
1994-05-24 10:09:53 +00:00
|
|
|
if (tp->snd_wnd > tp->max_sndwnd)
|
|
|
|
tp->max_sndwnd = tp->snd_wnd;
|
2012-11-05 09:13:06 +00:00
|
|
|
needoutput = 1;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Process segments with URG.
|
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
if ((thflags & TH_URG) && th->th_urp &&
|
1994-05-24 10:09:53 +00:00
|
|
|
TCPS_HAVERCVDFIN(tp->t_state) == 0) {
|
|
|
|
/*
|
|
|
|
* This is a kludge, but if we receive and accept
|
|
|
|
* random urgent pointers, we'll crash in
|
|
|
|
* soreceive. It's hard to imagine someone
|
|
|
|
* actually wanting to send this much urgent data.
|
|
|
|
*/
|
2004-11-28 11:01:31 +00:00
|
|
|
SOCKBUF_LOCK(&so->so_rcv);
|
2000-01-09 19:17:30 +00:00
|
|
|
if (th->th_urp + so->so_rcv.sb_cc > sb_max) {
|
|
|
|
th->th_urp = 0; /* XXX */
|
|
|
|
thflags &= ~TH_URG; /* XXX */
|
2004-11-28 11:01:31 +00:00
|
|
|
SOCKBUF_UNLOCK(&so->so_rcv); /* XXX */
|
1994-05-24 10:09:53 +00:00
|
|
|
goto dodata; /* XXX */
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* If this segment advances the known urgent pointer,
|
|
|
|
* then mark the data stream. This should not happen
|
|
|
|
* in CLOSE_WAIT, CLOSING, LAST_ACK or TIME_WAIT STATES since
|
1995-05-30 08:16:23 +00:00
|
|
|
* a FIN has been received from the remote side.
|
1994-05-24 10:09:53 +00:00
|
|
|
* In these states we ignore the URG.
|
|
|
|
*
|
|
|
|
* According to RFC961 (Assigned Protocols),
|
|
|
|
* the urgent pointer points to the last octet
|
|
|
|
* of urgent data. We continue, however,
|
|
|
|
* to consider it to indicate the first octet
|
1995-05-30 08:16:23 +00:00
|
|
|
* of data past the urgent section as the original
|
1994-05-24 10:09:53 +00:00
|
|
|
* spec states (in one of two places).
|
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
if (SEQ_GT(th->th_seq+th->th_urp, tp->rcv_up)) {
|
|
|
|
tp->rcv_up = th->th_seq + th->th_urp;
|
1994-05-24 10:09:53 +00:00
|
|
|
so->so_oobmark = so->so_rcv.sb_cc +
|
|
|
|
(tp->rcv_up - tp->rcv_nxt) - 1;
|
2004-06-24 02:57:12 +00:00
|
|
|
if (so->so_oobmark == 0)
|
2004-06-14 18:16:22 +00:00
|
|
|
so->so_rcv.sb_state |= SBS_RCVATMARK;
|
1994-05-24 10:09:53 +00:00
|
|
|
sohasoutofband(so);
|
|
|
|
tp->t_oobflags &= ~(TCPOOB_HAVEDATA | TCPOOB_HADDATA);
|
|
|
|
}
|
2004-11-28 11:01:31 +00:00
|
|
|
SOCKBUF_UNLOCK(&so->so_rcv);
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* Remove out of band data so doesn't get presented to user.
|
|
|
|
* This can happen independent of advancing the URG pointer,
|
|
|
|
* but if two URG's are pending at once, some out-of-band
|
|
|
|
* data may creep in... ick.
|
|
|
|
*/
|
2002-10-30 08:32:19 +00:00
|
|
|
if (th->th_urp <= (u_long)tlen &&
|
|
|
|
!(so->so_options & SO_OOBINLINE)) {
|
|
|
|
/* hdr drop is delayed */
|
|
|
|
tcp_pulloutofband(so, th, m, drop_hdrlen);
|
|
|
|
}
|
2002-08-17 02:05:25 +00:00
|
|
|
} else {
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* If no out of band data is expected,
|
|
|
|
* pull receive urgent pointer along
|
|
|
|
* with the receive window.
|
|
|
|
*/
|
|
|
|
if (SEQ_GT(tp->rcv_nxt, tp->rcv_up))
|
|
|
|
tp->rcv_up = tp->rcv_nxt;
|
2002-08-17 02:05:25 +00:00
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
dodata: /* XXX */
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
2004-07-12 19:28:07 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* Process the segment text, merging it into the TCP sequencing queue,
|
|
|
|
* and arranging for acknowledgment of receipt if necessary.
|
|
|
|
* This process logically involves adjusting tp->rcv_wnd as data
|
|
|
|
* is presented to the user (this happens in tcp_usrreq.c,
|
|
|
|
* case PRU_RCVD). If a FIN has already been received on this
|
|
|
|
* connection then we just ignore the text.
|
|
|
|
*/
|
2002-08-17 02:05:25 +00:00
|
|
|
if ((tlen || (thflags & TH_FIN)) &&
|
1994-05-24 10:09:53 +00:00
|
|
|
TCPS_HAVERCVDFIN(tp->t_state) == 0) {
|
2005-07-01 22:52:46 +00:00
|
|
|
tcp_seq save_start = th->th_seq;
|
2000-01-09 19:17:30 +00:00
|
|
|
m_adj(m, drop_hdrlen); /* delayed header drop */
|
2001-05-29 19:54:45 +00:00
|
|
|
/*
|
2002-08-17 02:05:25 +00:00
|
|
|
* Insert segment which includes th into TCP reassembly queue
|
|
|
|
* with control block tp. Set thflags to whether reassembly now
|
|
|
|
* includes a segment with FIN. This handles the common case
|
|
|
|
* inline (segment is the next to be received on an established
|
|
|
|
* connection, and the queue is empty), avoiding linkage into
|
|
|
|
* and removal from the queue and repetition of various
|
|
|
|
* conversions.
|
|
|
|
* Set DELACK for segments received in order, but ack
|
|
|
|
* immediately when segments are out of order (so
|
|
|
|
* fast retransmit can work).
|
2001-05-29 19:54:45 +00:00
|
|
|
*/
|
|
|
|
if (th->th_seq == tp->rcv_nxt &&
|
|
|
|
LIST_EMPTY(&tp->t_segq) &&
|
|
|
|
TCPS_HAVEESTABLISHED(tp->t_state)) {
|
|
|
|
if (DELAY_ACK(tp))
|
2003-02-19 21:18:23 +00:00
|
|
|
tp->t_flags |= TF_DELACK;
|
2001-05-29 19:54:45 +00:00
|
|
|
else
|
|
|
|
tp->t_flags |= TF_ACKNOW;
|
|
|
|
tp->rcv_nxt += tlen;
|
|
|
|
thflags = th->th_flags & TH_FIN;
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rcvpack);
|
|
|
|
TCPSTAT_ADD(tcps_rcvbyte, tlen);
|
2001-05-29 19:54:45 +00:00
|
|
|
ND6_HINT(tp);
|
Reduce the number of unnecessary unlock-relocks on socket buffer mutexes
associated with performing a wakeup on the socket buffer:
- When performing an sbappend*() followed by a so[rw]wakeup(), explicitly
acquire the socket buffer lock and use the _locked() variants of both
calls. Note that the _locked() sowakeup() versions unlock the mutex on
return. This is done in uipc_send(), divert_packet(), mroute
socket_send(), raw_append(), tcp_reass(), tcp_input(), and udp_append().
- When the socket buffer lock is dropped before a sowakeup(), remove the
explicit unlock and use the _locked() sowakeup() variant. This is done
in soisdisconnecting(), soisdisconnected() when setting the can't send/
receive flags and dropping data, and in uipc_rcvd() which adjusting
back-pressure on the sockets.
For UNIX domain sockets running mpsafe with a contention-intensive SMP
mysql benchmark, this results in a 1.6% query rate improvement due to
reduce mutex costs.
2004-06-26 19:10:39 +00:00
|
|
|
SOCKBUF_LOCK(&so->so_rcv);
|
2004-06-14 18:16:22 +00:00
|
|
|
if (so->so_rcv.sb_state & SBS_CANTRCVMORE)
|
2002-09-22 02:54:07 +00:00
|
|
|
m_freem(m);
|
|
|
|
else
|
Reduce the number of unnecessary unlock-relocks on socket buffer mutexes
associated with performing a wakeup on the socket buffer:
- When performing an sbappend*() followed by a so[rw]wakeup(), explicitly
acquire the socket buffer lock and use the _locked() variants of both
calls. Note that the _locked() sowakeup() versions unlock the mutex on
return. This is done in uipc_send(), divert_packet(), mroute
socket_send(), raw_append(), tcp_reass(), tcp_input(), and udp_append().
- When the socket buffer lock is dropped before a sowakeup(), remove the
explicit unlock and use the _locked() sowakeup() variant. This is done
in soisdisconnecting(), soisdisconnected() when setting the can't send/
receive flags and dropping data, and in uipc_rcvd() which adjusting
back-pressure on the sockets.
For UNIX domain sockets running mpsafe with a contention-intensive SMP
mysql benchmark, this results in a 1.6% query rate improvement due to
reduce mutex costs.
2004-06-26 19:10:39 +00:00
|
|
|
sbappendstream_locked(&so->so_rcv, m);
|
2007-06-10 20:59:22 +00:00
|
|
|
/* NB: sorwakeup_locked() does an implicit unlock. */
|
Reduce the number of unnecessary unlock-relocks on socket buffer mutexes
associated with performing a wakeup on the socket buffer:
- When performing an sbappend*() followed by a so[rw]wakeup(), explicitly
acquire the socket buffer lock and use the _locked() variants of both
calls. Note that the _locked() sowakeup() versions unlock the mutex on
return. This is done in uipc_send(), divert_packet(), mroute
socket_send(), raw_append(), tcp_reass(), tcp_input(), and udp_append().
- When the socket buffer lock is dropped before a sowakeup(), remove the
explicit unlock and use the _locked() sowakeup() variant. This is done
in soisdisconnecting(), soisdisconnected() when setting the can't send/
receive flags and dropping data, and in uipc_rcvd() which adjusting
back-pressure on the sockets.
For UNIX domain sockets running mpsafe with a contention-intensive SMP
mysql benchmark, this results in a 1.6% query rate improvement due to
reduce mutex costs.
2004-06-26 19:10:39 +00:00
|
|
|
sorwakeup_locked(so);
|
2001-05-29 19:54:45 +00:00
|
|
|
} else {
|
2007-06-10 20:59:22 +00:00
|
|
|
/*
|
|
|
|
* XXX: Due to the header drop above "th" is
|
|
|
|
* theoretically invalid by now. Fortunately
|
|
|
|
* m_adj() doesn't actually frees any mbufs
|
|
|
|
* when trimming from the head.
|
|
|
|
*/
|
2001-05-29 19:54:45 +00:00
|
|
|
thflags = tcp_reass(tp, th, &tlen, m);
|
|
|
|
tp->t_flags |= TF_ACKNOW;
|
|
|
|
}
|
2007-05-06 15:56:31 +00:00
|
|
|
if (tlen > 0 && (tp->t_flags & TF_SACK_PERMIT))
|
2007-06-10 21:07:21 +00:00
|
|
|
tcp_update_sack_list(tp, save_start, save_start + tlen);
|
2007-03-23 20:16:50 +00:00
|
|
|
#if 0
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* Note the amount of data that peer has sent into
|
|
|
|
* our window, in order to estimate the sender's
|
|
|
|
* buffer size.
|
2007-03-23 20:16:50 +00:00
|
|
|
* XXX: Unused.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
Handle a rare edge case with nearly full TCP receive buffers. If a TCP
buffer fills up causing the remote sender to enter into persist mode, but
there is still room available in the receive buffer when a window probe
arrives (either due to window scaling, or due to the local application
very slowing draining data from the receive buffer), then the single byte
of data in the window probe is accepted. However, this can cause rcv_nxt
to be greater than rcv_adv. This condition will only last until the next
ACK packet is pushed out via tcp_output(), and since the previous ACK
advertised a zero window, the ACK should be pushed out while the TCP
pcb is write-locked.
During the window while rcv_nxt is greather than rcv_adv, a few places
would compute the remaining receive window via rcv_adv - rcv_nxt.
However, this value was then (uint32_t)-1. On a 64 bit machine this
could expand to a positive 2^32 - 1 when cast to a long. In particular,
when calculating the receive window in tcp_output(), the result would be
that the receive window was computed as 2^32 - 1 resulting in advertising
a far larger window to the remote peer than actually existed.
Fix various places that compute the remaining receive window to either
assert that it is not negative (i.e. rcv_nxt <= rcv_adv), or treat the
window as full if rcv_nxt is greather than rcv_adv.
Reviewed by: bz
MFC after: 1 month
2011-05-02 21:05:52 +00:00
|
|
|
if (SEQ_GT(tp->rcv_adv, tp->rcv_nxt))
|
|
|
|
len = so->so_rcv.sb_hiwat - (tp->rcv_adv - tp->rcv_nxt);
|
|
|
|
else
|
|
|
|
len = so->so_rcv.sb_hiwat;
|
2007-03-23 20:16:50 +00:00
|
|
|
#endif
|
1994-05-24 10:09:53 +00:00
|
|
|
} else {
|
|
|
|
m_freem(m);
|
2000-01-09 19:17:30 +00:00
|
|
|
thflags &= ~TH_FIN;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If FIN is received ACK the FIN and let the user know
|
|
|
|
* that the connection is closing.
|
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
if (thflags & TH_FIN) {
|
1994-05-24 10:09:53 +00:00
|
|
|
if (TCPS_HAVERCVDFIN(tp->t_state) == 0) {
|
|
|
|
socantrcvmore(so);
|
1995-02-09 23:13:27 +00:00
|
|
|
/*
|
2002-08-17 02:05:25 +00:00
|
|
|
* If connection is half-synchronized
|
|
|
|
* (ie NEEDSYN flag on) then delay ACK,
|
|
|
|
* so it may be piggybacked when SYN is sent.
|
|
|
|
* Otherwise, since we received a FIN then no
|
|
|
|
* more input can be expected, send ACK now.
|
1995-02-09 23:13:27 +00:00
|
|
|
*/
|
2003-02-19 21:18:23 +00:00
|
|
|
if (tp->t_flags & TF_NEEDSYN)
|
|
|
|
tp->t_flags |= TF_DELACK;
|
1995-05-30 08:16:23 +00:00
|
|
|
else
|
1995-02-09 23:13:27 +00:00
|
|
|
tp->t_flags |= TF_ACKNOW;
|
1994-05-24 10:09:53 +00:00
|
|
|
tp->rcv_nxt++;
|
|
|
|
}
|
|
|
|
switch (tp->t_state) {
|
|
|
|
|
2004-08-16 18:32:07 +00:00
|
|
|
/*
|
1994-05-24 10:09:53 +00:00
|
|
|
* In SYN_RECEIVED and ESTABLISHED STATES
|
|
|
|
* enter the CLOSE_WAIT state.
|
|
|
|
*/
|
|
|
|
case TCPS_SYN_RECEIVED:
|
1999-08-30 21:17:07 +00:00
|
|
|
tp->t_starttime = ticks;
|
2007-07-25 18:48:24 +00:00
|
|
|
/* FALLTHROUGH */
|
1994-05-24 10:09:53 +00:00
|
|
|
case TCPS_ESTABLISHED:
|
|
|
|
tp->t_state = TCPS_CLOSE_WAIT;
|
|
|
|
break;
|
|
|
|
|
2004-08-16 18:32:07 +00:00
|
|
|
/*
|
1994-05-24 10:09:53 +00:00
|
|
|
* If still in FIN_WAIT_1 STATE FIN has not been acked so
|
|
|
|
* enter the CLOSING state.
|
|
|
|
*/
|
|
|
|
case TCPS_FIN_WAIT_1:
|
|
|
|
tp->t_state = TCPS_CLOSING;
|
|
|
|
break;
|
|
|
|
|
2004-08-16 18:32:07 +00:00
|
|
|
/*
|
1994-05-24 10:09:53 +00:00
|
|
|
* In FIN_WAIT_2 state enter the TIME_WAIT state,
|
1995-05-30 08:16:23 +00:00
|
|
|
* starting the time-wait timer, turning off the other
|
1994-05-24 10:09:53 +00:00
|
|
|
* standard timers.
|
|
|
|
*/
|
|
|
|
case TCPS_FIN_WAIT_2:
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
KASSERT(ti_locked == TI_WLOCKED, ("%s: dodata "
|
|
|
|
"TCP_FIN_WAIT_2 ti_locked: %d", __func__,
|
|
|
|
ti_locked));
|
|
|
|
|
2003-02-19 22:32:43 +00:00
|
|
|
tcp_twstart(tp);
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
INP_INFO_WUNLOCK(&V_tcbinfo);
|
2007-05-06 15:16:05 +00:00
|
|
|
return;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
}
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (ti_locked == TI_WLOCKED)
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WUNLOCK(&V_tcbinfo);
|
|
|
|
ti_locked = TI_UNLOCKED;
|
|
|
|
|
1994-09-15 10:36:56 +00:00
|
|
|
#ifdef TCPDEBUG
|
2002-05-31 11:52:35 +00:00
|
|
|
if (so->so_options & SO_DEBUG)
|
2000-01-09 19:17:30 +00:00
|
|
|
tcp_trace(TA_INPUT, ostate, tp, (void *)tcp_saveipgen,
|
|
|
|
&tcp_savetcp, 0);
|
1994-09-15 10:36:56 +00:00
|
|
|
#endif
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Return any desired output.
|
|
|
|
*/
|
|
|
|
if (needoutput || (tp->t_flags & TF_ACKNOW))
|
|
|
|
(void) tcp_output(tp);
|
2003-03-13 11:46:57 +00:00
|
|
|
|
2003-02-22 21:54:57 +00:00
|
|
|
check_delack:
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
KASSERT(ti_locked == TI_UNLOCKED, ("%s: check_delack ti_locked %d",
|
|
|
|
__func__, ti_locked));
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
INP_INFO_UNLOCK_ASSERT(&V_tcbinfo);
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
|
2003-02-22 21:54:57 +00:00
|
|
|
if (tp->t_flags & TF_DELACK) {
|
2003-02-19 21:18:23 +00:00
|
|
|
tp->t_flags &= ~TF_DELACK;
|
2007-04-11 09:45:16 +00:00
|
|
|
tcp_timer_activate(tp, TT_DELACK, tcp_delacktime);
|
2003-02-19 21:18:23 +00:00
|
|
|
}
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WUNLOCK(tp->t_inpcb);
|
2007-05-06 15:16:05 +00:00
|
|
|
return;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
dropafterack:
|
|
|
|
/*
|
|
|
|
* Generate an ACK dropping incoming segment if it occupies
|
|
|
|
* sequence space, where the ACK reflects our state.
|
1998-09-11 16:04:03 +00:00
|
|
|
*
|
|
|
|
* We can now skip the test for the RST flag since all
|
|
|
|
* paths to this code happen after packets containing
|
|
|
|
* RST have been dropped.
|
|
|
|
*
|
|
|
|
* In the SYN-RECEIVED state, don't send an ACK unless the
|
|
|
|
* segment we received passes the SYN-RECEIVED ACK test.
|
|
|
|
* If it fails send a RST. This breaks the loop in the
|
|
|
|
* "LAND" DoS attack, and also prevents an ACK storm
|
|
|
|
* between two listening ports that have been sent forged
|
|
|
|
* SYN segments, each with the source address of the other.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2000-01-09 19:17:30 +00:00
|
|
|
if (tp->t_state == TCPS_SYN_RECEIVED && (thflags & TH_ACK) &&
|
|
|
|
(SEQ_GT(tp->snd_una, th->th_ack) ||
|
2001-02-11 07:39:51 +00:00
|
|
|
SEQ_GT(th->th_ack, tp->snd_max)) ) {
|
|
|
|
rstreason = BANDLIM_RST_OPENPORT;
|
|
|
|
goto dropwithreset;
|
|
|
|
}
|
1995-02-09 23:13:27 +00:00
|
|
|
#ifdef TCPDEBUG
|
2002-05-31 11:52:35 +00:00
|
|
|
if (so->so_options & SO_DEBUG)
|
2000-01-09 19:17:30 +00:00
|
|
|
tcp_trace(TA_DROP, ostate, tp, (void *)tcp_saveipgen,
|
|
|
|
&tcp_savetcp, 0);
|
1995-02-09 23:13:27 +00:00
|
|
|
#endif
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (ti_locked == TI_WLOCKED)
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WUNLOCK(&V_tcbinfo);
|
|
|
|
ti_locked = TI_UNLOCKED;
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
tp->t_flags |= TF_ACKNOW;
|
|
|
|
(void) tcp_output(tp);
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WUNLOCK(tp->t_inpcb);
|
2004-11-07 19:19:35 +00:00
|
|
|
m_freem(m);
|
2007-05-06 15:16:05 +00:00
|
|
|
return;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
dropwithreset:
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (ti_locked == TI_WLOCKED)
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WUNLOCK(&V_tcbinfo);
|
|
|
|
ti_locked = TI_UNLOCKED;
|
2007-03-23 20:16:50 +00:00
|
|
|
|
2008-09-24 11:07:03 +00:00
|
|
|
if (tp != NULL) {
|
|
|
|
tcp_dropwithreset(m, th, tp, tlen, rstreason);
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WUNLOCK(tp->t_inpcb);
|
2008-10-26 22:03:52 +00:00
|
|
|
} else
|
2008-09-24 11:07:03 +00:00
|
|
|
tcp_dropwithreset(m, th, NULL, tlen, rstreason);
|
2007-05-06 15:16:05 +00:00
|
|
|
return;
|
2007-03-23 20:16:50 +00:00
|
|
|
|
|
|
|
drop:
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
if (ti_locked == TI_WLOCKED) {
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
INP_INFO_WUNLOCK(&V_tcbinfo);
|
Decompose the current single inpcbinfo lock into two locks:
- The existing ipi_lock continues to protect the global inpcb list and
inpcb counter. This lock is now relegated to a small number of
allocation and free operations, and occasional operations that walk
all connections (including, awkwardly, certain UDP multicast receive
operations -- something to revisit).
- A new ipi_hash_lock protects the two inpcbinfo hash tables for
looking up connections and bound sockets, manipulated using new
INP_HASH_*() macros. This lock, combined with inpcb locks, protects
the 4-tuple address space.
Unlike the current ipi_lock, ipi_hash_lock follows the individual inpcb
connection locks, so may be acquired while manipulating a connection on
which a lock is already held, avoiding the need to acquire the inpcbinfo
lock preemptively when a binding change might later be required. As a
result, however, lookup operations necessarily go through a reference
acquire while holding the lookup lock, later acquiring an inpcb lock --
if required.
A new function in_pcblookup() looks up connections, and accepts flags
indicating how to return the inpcb. Due to lock order changes, callers
no longer need acquire locks before performing a lookup: the lookup
routine will acquire the ipi_hash_lock as needed. In the future, it will
also be able to use alternative lookup and locking strategies
transparently to callers, such as pcbgroup lookup. New lookup flags are,
supplementing the existing INPLOOKUP_WILDCARD flag:
INPLOOKUP_RLOCKPCB - Acquire a read lock on the returned inpcb
INPLOOKUP_WLOCKPCB - Acquire a write lock on the returned inpcb
Callers must pass exactly one of these flags (for the time being).
Some notes:
- All protocols are updated to work within the new regime; especially,
TCP, UDPv4, and UDPv6. pcbinfo ipi_lock acquisitions are largely
eliminated, and global hash lock hold times are dramatically reduced
compared to previous locking.
- The TCP syncache still relies on the pcbinfo lock, something that we
may want to revisit.
- Support for reverting to the FreeBSD 7.x locking strategy in TCP input
is no longer available -- hash lookup locks are now held only very
briefly during inpcb lookup, rather than for potentially extended
periods. However, the pcbinfo ipi_lock will still be acquired if a
connection state might change such that a connection is added or
removed.
- Raw IP sockets continue to use the pcbinfo ipi_lock for protection,
due to maintaining their own hash tables.
- The interface in6_pcblookup_hash_locked() is maintained, which allows
callers to acquire hash locks and perform one or more lookups atomically
with 4-tuple allocation: this is required only for TCPv6, as there is no
in6_pcbconnect_setup(), which there should be.
- UDPv6 locking remains significantly more conservative than UDPv4
locking, which relates to source address selection. This needs
attention, as it likely significantly reduces parallelism in this code
for multithreaded socket use (such as in BIND).
- In the UDPv4 and UDPv6 multicast cases, we need to revisit locking
somewhat, as they relied on ipi_lock to stablise 4-tuple matches, which
is no longer sufficient. A second check once the inpcb lock is held
should do the trick, keeping the general case from requiring the inpcb
lock for every inpcb visited.
- This work reminds us that we need to revisit locking of the v4/v6 flags,
which may be accessed lock-free both before and after this change.
- Right now, a single lock name is used for the pcbhash lock -- this is
undesirable, and probably another argument is required to take care of
this (or a char array name field in the pcbinfo?).
This is not an MFC candidate for 8.x due to its impact on lookup and
locking semantics. It's possible some of these issues could be worked
around with compatibility wrappers, if necessary.
Reviewed by: bz
Sponsored by: Juniper Networks, Inc.
2011-05-30 09:43:55 +00:00
|
|
|
ti_locked = TI_UNLOCKED;
|
|
|
|
}
|
Move from solely write-locking the global tcbinfo in tcp_input()
to read-locking in the TCP input path, allowing greater TCP
input parallelism where multiple ithreads or ithread and netisr
are able to run in parallel. Previously, most TCP input paths
held a write lock on the global tcbinfo lock, effectively
serializing TCP input.
Before looking up the connection, acquire a write lock if a
potentially state-changing flag is set on the TCP segment header
(FIN, RST, SYN), and otherwise a read lock. We may later have
to upgrade to a write lock in certain cases (ACKs received by the
syncache or during TIMEWAIT) in order to support global state
transitions, but this is never required for steady-state packets.
Upgrading from a write lock to a read lock must be done as a
trylock operation to avoid deadlocks, and actually violates the
lock order as the tcbinfo lock preceeds the inpcb lock held at
the time of upgrade. If the trylock fails, we bump the refcount
on the inpcb, drop both locks, and re-acquire in-order. If
another thread has freed the connection while the locks are
dropped, we free the inpcb and repeat the lookup (this should
hardly ever or never happen in practice).
For now, maintain a number of new counters measuring how many
times various cases execute, and in particular whether various
optimistic assumptions about when read locks can be used, whether
upgrades are done using the fast path, and whether connections
close in practice in the above-described race, actually occur.
MFC after: 6 weeks
Discussed with: kmacy
Reviewed by: bz, gnn, kmacy
Tested by: kmacy
2008-12-08 20:27:00 +00:00
|
|
|
#ifdef INVARIANTS
|
|
|
|
else
|
|
|
|
INP_INFO_UNLOCK_ASSERT(&V_tcbinfo);
|
|
|
|
#endif
|
|
|
|
|
2007-03-23 20:16:50 +00:00
|
|
|
/*
|
|
|
|
* Drop space held by incoming segment and return.
|
|
|
|
*/
|
|
|
|
#ifdef TCPDEBUG
|
|
|
|
if (tp == NULL || (tp->t_inpcb->inp_socket->so_options & SO_DEBUG))
|
|
|
|
tcp_trace(TA_DROP, ostate, tp, (void *)tcp_saveipgen,
|
|
|
|
&tcp_savetcp, 0);
|
|
|
|
#endif
|
|
|
|
if (tp != NULL)
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WUNLOCK(tp->t_inpcb);
|
2007-03-23 20:16:50 +00:00
|
|
|
m_freem(m);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2007-05-06 15:41:06 +00:00
|
|
|
* Issue RST and make ACK acceptable to originator of segment.
|
|
|
|
* The mbuf must still include the original packet header.
|
|
|
|
* tp may be NULL.
|
2007-03-23 20:16:50 +00:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
tcp_dropwithreset(struct mbuf *m, struct tcphdr *th, struct tcpcb *tp,
|
|
|
|
int tlen, int rstreason)
|
|
|
|
{
|
2011-04-30 11:21:29 +00:00
|
|
|
#ifdef INET
|
2007-03-23 20:16:50 +00:00
|
|
|
struct ip *ip;
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif
|
2007-03-23 20:16:50 +00:00
|
|
|
#ifdef INET6
|
|
|
|
struct ip6_hdr *ip6;
|
|
|
|
#endif
|
2008-04-07 12:41:45 +00:00
|
|
|
|
|
|
|
if (tp != NULL) {
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
2008-04-07 12:41:45 +00:00
|
|
|
}
|
|
|
|
|
2007-05-06 15:41:06 +00:00
|
|
|
/* Don't bother if destination was broadcast/multicast. */
|
2007-03-23 20:16:50 +00:00
|
|
|
if ((th->th_flags & TH_RST) || m->m_flags & (M_BCAST|M_MCAST))
|
1994-05-24 10:09:53 +00:00
|
|
|
goto drop;
|
2007-03-23 20:16:50 +00:00
|
|
|
#ifdef INET6
|
|
|
|
if (mtod(m, struct ip *)->ip_v == 6) {
|
|
|
|
ip6 = mtod(m, struct ip6_hdr *);
|
2000-01-28 06:13:09 +00:00
|
|
|
if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst) ||
|
|
|
|
IN6_IS_ADDR_MULTICAST(&ip6->ip6_src))
|
2000-01-09 19:17:30 +00:00
|
|
|
goto drop;
|
2007-03-23 20:16:50 +00:00
|
|
|
/* IPv6 anycast check is done at tcp6_input() */
|
2011-04-30 11:21:29 +00:00
|
|
|
}
|
2007-03-23 20:16:50 +00:00
|
|
|
#endif
|
2011-04-30 11:21:29 +00:00
|
|
|
#if defined(INET) && defined(INET6)
|
|
|
|
else
|
|
|
|
#endif
|
|
|
|
#ifdef INET
|
2007-03-23 20:16:50 +00:00
|
|
|
{
|
|
|
|
ip = mtod(m, struct ip *);
|
2002-08-17 02:05:25 +00:00
|
|
|
if (IN_MULTICAST(ntohl(ip->ip_dst.s_addr)) ||
|
|
|
|
IN_MULTICAST(ntohl(ip->ip_src.s_addr)) ||
|
2004-08-16 18:32:07 +00:00
|
|
|
ip->ip_src.s_addr == htonl(INADDR_BROADCAST) ||
|
|
|
|
in_broadcast(ip->ip_dst, m->m_pkthdr.rcvif))
|
2002-08-17 02:05:25 +00:00
|
|
|
goto drop;
|
|
|
|
}
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif
|
2001-02-11 07:39:51 +00:00
|
|
|
|
2007-03-23 20:16:50 +00:00
|
|
|
/* Perform bandwidth limiting. */
|
2001-02-11 07:39:51 +00:00
|
|
|
if (badport_bandlim(rstreason) < 0)
|
|
|
|
goto drop;
|
2004-08-16 18:32:07 +00:00
|
|
|
|
2007-03-23 20:16:50 +00:00
|
|
|
/* tcp_respond consumes the mbuf chain. */
|
|
|
|
if (th->th_flags & TH_ACK) {
|
|
|
|
tcp_respond(tp, mtod(m, void *), th, m, (tcp_seq)0,
|
|
|
|
th->th_ack, TH_RST);
|
|
|
|
} else {
|
|
|
|
if (th->th_flags & TH_SYN)
|
2000-01-09 19:17:30 +00:00
|
|
|
tlen++;
|
|
|
|
tcp_respond(tp, mtod(m, void *), th, m, th->th_seq+tlen,
|
2007-03-23 20:16:50 +00:00
|
|
|
(tcp_seq)0, TH_RST|TH_ACK);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
return;
|
|
|
|
drop:
|
2004-11-07 19:19:35 +00:00
|
|
|
m_freem(m);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
2001-11-22 04:50:44 +00:00
|
|
|
/*
|
|
|
|
* Parse TCP options and place in tcpopt.
|
|
|
|
*/
|
1995-11-14 20:34:56 +00:00
|
|
|
static void
|
2007-03-21 19:37:55 +00:00
|
|
|
tcp_dooptions(struct tcpopt *to, u_char *cp, int cnt, int flags)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
|
|
|
int opt, optlen;
|
|
|
|
|
2001-11-22 04:50:44 +00:00
|
|
|
to->to_flags = 0;
|
1994-05-24 10:09:53 +00:00
|
|
|
for (; cnt > 0; cnt -= optlen, cp += optlen) {
|
|
|
|
opt = cp[0];
|
|
|
|
if (opt == TCPOPT_EOL)
|
|
|
|
break;
|
|
|
|
if (opt == TCPOPT_NOP)
|
|
|
|
optlen = 1;
|
|
|
|
else {
|
2000-07-09 13:01:59 +00:00
|
|
|
if (cnt < 2)
|
|
|
|
break;
|
1994-05-24 10:09:53 +00:00
|
|
|
optlen = cp[1];
|
2000-07-09 13:01:59 +00:00
|
|
|
if (optlen < 2 || optlen > cnt)
|
1994-05-24 10:09:53 +00:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
switch (opt) {
|
|
|
|
case TCPOPT_MAXSEG:
|
|
|
|
if (optlen != TCPOLEN_MAXSEG)
|
|
|
|
continue;
|
2006-06-26 15:35:25 +00:00
|
|
|
if (!(flags & TO_SYN))
|
1994-05-24 10:09:53 +00:00
|
|
|
continue;
|
2001-11-22 04:50:44 +00:00
|
|
|
to->to_flags |= TOF_MSS;
|
|
|
|
bcopy((char *)cp + 2,
|
|
|
|
(char *)&to->to_mss, sizeof(to->to_mss));
|
2002-02-18 20:35:27 +00:00
|
|
|
to->to_mss = ntohs(to->to_mss);
|
1994-05-24 10:09:53 +00:00
|
|
|
break;
|
|
|
|
case TCPOPT_WINDOW:
|
|
|
|
if (optlen != TCPOLEN_WINDOW)
|
|
|
|
continue;
|
2006-06-26 15:35:25 +00:00
|
|
|
if (!(flags & TO_SYN))
|
1994-05-24 10:09:53 +00:00
|
|
|
continue;
|
2001-11-22 04:50:44 +00:00
|
|
|
to->to_flags |= TOF_SCALE;
|
2007-03-15 15:59:28 +00:00
|
|
|
to->to_wscale = min(cp[2], TCP_MAX_WINSHIFT);
|
1994-05-24 10:09:53 +00:00
|
|
|
break;
|
|
|
|
case TCPOPT_TIMESTAMP:
|
|
|
|
if (optlen != TCPOLEN_TIMESTAMP)
|
|
|
|
continue;
|
2001-11-22 04:50:44 +00:00
|
|
|
to->to_flags |= TOF_TS;
|
1995-02-09 23:13:27 +00:00
|
|
|
bcopy((char *)cp + 2,
|
|
|
|
(char *)&to->to_tsval, sizeof(to->to_tsval));
|
2002-02-18 20:35:27 +00:00
|
|
|
to->to_tsval = ntohl(to->to_tsval);
|
1995-02-09 23:13:27 +00:00
|
|
|
bcopy((char *)cp + 6,
|
|
|
|
(char *)&to->to_tsecr, sizeof(to->to_tsecr));
|
2002-02-18 20:35:27 +00:00
|
|
|
to->to_tsecr = ntohl(to->to_tsecr);
|
1994-05-24 10:09:53 +00:00
|
|
|
break;
|
Initial import of RFC 2385 (TCP-MD5) digest support.
This is the first of two commits; bringing in the kernel support first.
This can be enabled by compiling a kernel with options TCP_SIGNATURE
and FAST_IPSEC.
For the uninitiated, this is a TCP option which provides for a means of
authenticating TCP sessions which came into being before IPSEC. It is
still relevant today, however, as it is used by many commercial router
vendors, particularly with BGP, and as such has become a requirement for
interconnect at many major Internet points of presence.
Several parts of the TCP and IP headers, including the segment payload,
are digested with MD5, including a shared secret. The PF_KEY interface
is used to manage the secrets using security associations in the SADB.
There is a limitation here in that as there is no way to map a TCP flow
per-port back to an SPI without polluting tcpcb or using the SPD; the
code to do the latter is unstable at this time. Therefore this code only
supports per-host keying granularity.
Whilst FAST_IPSEC is mutually exclusive with KAME IPSEC (and thus IPv6),
TCP_SIGNATURE applies only to IPv4. For the vast majority of prospective
users of this feature, this will not pose any problem.
This implementation is output-only; that is, the option is honoured when
responding to a host initiating a TCP session, but no effort is made
[yet] to authenticate inbound traffic. This is, however, sufficient to
interwork with Cisco equipment.
Tested with a Cisco 2501 running IOS 12.0(27), and Quagga 0.96.4 with
local patches. Patches for tcpdump to validate TCP-MD5 sessions are also
available from me upon request.
Sponsored by: sentex.net
2004-02-11 04:26:04 +00:00
|
|
|
#ifdef TCP_SIGNATURE
|
|
|
|
/*
|
|
|
|
* XXX In order to reply to a host which has set the
|
|
|
|
* TCP_SIGNATURE option in its initial SYN, we have to
|
|
|
|
* record the fact that the option was observed here
|
|
|
|
* for the syncache code to perform the correct response.
|
|
|
|
*/
|
|
|
|
case TCPOPT_SIGNATURE:
|
|
|
|
if (optlen != TCPOLEN_SIGNATURE)
|
|
|
|
continue;
|
2007-04-20 15:28:01 +00:00
|
|
|
to->to_flags |= TOF_SIGNATURE;
|
|
|
|
to->to_signature = cp + 2;
|
Initial import of RFC 2385 (TCP-MD5) digest support.
This is the first of two commits; bringing in the kernel support first.
This can be enabled by compiling a kernel with options TCP_SIGNATURE
and FAST_IPSEC.
For the uninitiated, this is a TCP option which provides for a means of
authenticating TCP sessions which came into being before IPSEC. It is
still relevant today, however, as it is used by many commercial router
vendors, particularly with BGP, and as such has become a requirement for
interconnect at many major Internet points of presence.
Several parts of the TCP and IP headers, including the segment payload,
are digested with MD5, including a shared secret. The PF_KEY interface
is used to manage the secrets using security associations in the SADB.
There is a limitation here in that as there is no way to map a TCP flow
per-port back to an SPI without polluting tcpcb or using the SPD; the
code to do the latter is unstable at this time. Therefore this code only
supports per-host keying granularity.
Whilst FAST_IPSEC is mutually exclusive with KAME IPSEC (and thus IPv6),
TCP_SIGNATURE applies only to IPv4. For the vast majority of prospective
users of this feature, this will not pose any problem.
This implementation is output-only; that is, the option is honoured when
responding to a host initiating a TCP session, but no effort is made
[yet] to authenticate inbound traffic. This is, however, sufficient to
interwork with Cisco equipment.
Tested with a Cisco 2501 running IOS 12.0(27), and Quagga 0.96.4 with
local patches. Patches for tcpdump to validate TCP-MD5 sessions are also
available from me upon request.
Sponsored by: sentex.net
2004-02-11 04:26:04 +00:00
|
|
|
break;
|
2004-02-13 18:21:45 +00:00
|
|
|
#endif
|
2004-06-23 21:04:37 +00:00
|
|
|
case TCPOPT_SACK_PERMITTED:
|
2006-06-26 15:35:25 +00:00
|
|
|
if (optlen != TCPOLEN_SACK_PERMITTED)
|
2004-06-23 21:04:37 +00:00
|
|
|
continue;
|
2006-06-26 15:35:25 +00:00
|
|
|
if (!(flags & TO_SYN))
|
|
|
|
continue;
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
if (!V_tcp_do_sack)
|
2006-06-26 15:35:25 +00:00
|
|
|
continue;
|
2007-03-23 18:33:21 +00:00
|
|
|
to->to_flags |= TOF_SACKPERM;
|
2004-06-23 21:04:37 +00:00
|
|
|
break;
|
|
|
|
case TCPOPT_SACK:
|
2005-06-27 22:27:42 +00:00
|
|
|
if (optlen <= 2 || (optlen - 2) % TCPOLEN_SACK != 0)
|
2004-06-23 21:04:37 +00:00
|
|
|
continue;
|
2007-04-04 14:39:49 +00:00
|
|
|
if (flags & TO_SYN)
|
|
|
|
continue;
|
2007-03-23 18:33:21 +00:00
|
|
|
to->to_flags |= TOF_SACK;
|
2005-06-27 22:27:42 +00:00
|
|
|
to->to_nsacks = (optlen - 2) / TCPOLEN_SACK;
|
|
|
|
to->to_sacks = cp + 2;
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_sack_rcv_blocks);
|
2004-06-23 21:04:37 +00:00
|
|
|
break;
|
2001-11-22 04:50:44 +00:00
|
|
|
default:
|
|
|
|
continue;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Pull out of band byte out of a segment so
|
|
|
|
* it doesn't appear in the user's data queue.
|
|
|
|
* It is still reflected in the segment length for
|
|
|
|
* sequencing purposes.
|
|
|
|
*/
|
1995-11-14 20:34:56 +00:00
|
|
|
static void
|
2007-03-21 19:37:55 +00:00
|
|
|
tcp_pulloutofband(struct socket *so, struct tcphdr *th, struct mbuf *m,
|
|
|
|
int off)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2000-01-09 19:17:30 +00:00
|
|
|
int cnt = off + th->th_urp - 1;
|
1995-05-30 08:16:23 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
while (cnt >= 0) {
|
|
|
|
if (m->m_len > cnt) {
|
|
|
|
char *cp = mtod(m, caddr_t) + cnt;
|
|
|
|
struct tcpcb *tp = sototcpcb(so);
|
|
|
|
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
2008-04-07 12:41:45 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
tp->t_iobc = *cp;
|
|
|
|
tp->t_oobflags |= TCPOOB_HAVEDATA;
|
|
|
|
bcopy(cp+1, cp, (unsigned)(m->m_len - cnt - 1));
|
|
|
|
m->m_len--;
|
2000-01-25 01:26:47 +00:00
|
|
|
if (m->m_flags & M_PKTHDR)
|
|
|
|
m->m_pkthdr.len--;
|
1994-05-24 10:09:53 +00:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
cnt -= m->m_len;
|
|
|
|
m = m->m_next;
|
2007-03-23 19:11:22 +00:00
|
|
|
if (m == NULL)
|
1994-05-24 10:09:53 +00:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
panic("tcp_pulloutofband");
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Collect new round-trip time estimate
|
|
|
|
* and update averages and current timeout.
|
|
|
|
*/
|
1995-11-14 20:34:56 +00:00
|
|
|
static void
|
2007-03-21 19:37:55 +00:00
|
|
|
tcp_xmit_timer(struct tcpcb *tp, int rtt)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2007-03-21 19:37:55 +00:00
|
|
|
int delta;
|
1996-03-22 18:09:21 +00:00
|
|
|
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
2004-11-28 11:06:22 +00:00
|
|
|
|
2009-04-11 22:07:19 +00:00
|
|
|
TCPSTAT_INC(tcps_rttupdated);
|
1996-03-22 18:09:21 +00:00
|
|
|
tp->t_rttupdated++;
|
|
|
|
if (tp->t_srtt != 0) {
|
|
|
|
/*
|
|
|
|
* srtt is stored as fixed point with 5 bits after the
|
|
|
|
* binary point (i.e., scaled by 8). The following magic
|
|
|
|
* is equivalent to the smoothing algorithm in rfc793 with
|
|
|
|
* an alpha of .875 (srtt = rtt/8 + srtt*7/8 in fixed
|
|
|
|
* point). Adjust rtt to origin 0.
|
|
|
|
*/
|
|
|
|
delta = ((rtt - 1) << TCP_DELTA_SHIFT)
|
|
|
|
- (tp->t_srtt >> (TCP_RTT_SHIFT - TCP_DELTA_SHIFT));
|
|
|
|
|
|
|
|
if ((tp->t_srtt += delta) <= 0)
|
|
|
|
tp->t_srtt = 1;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We accumulate a smoothed rtt variance (actually, a
|
|
|
|
* smoothed mean difference), then set the retransmit
|
|
|
|
* timer to smoothed rtt + 4 times the smoothed variance.
|
|
|
|
* rttvar is stored as fixed point with 4 bits after the
|
|
|
|
* binary point (scaled by 16). The following is
|
|
|
|
* equivalent to rfc793 smoothing with an alpha of .75
|
|
|
|
* (rttvar = rttvar*3/4 + |delta| / 4). This replaces
|
|
|
|
* rfc793's wired-in beta.
|
|
|
|
*/
|
|
|
|
if (delta < 0)
|
|
|
|
delta = -delta;
|
|
|
|
delta -= tp->t_rttvar >> (TCP_RTTVAR_SHIFT - TCP_DELTA_SHIFT);
|
|
|
|
if ((tp->t_rttvar += delta) <= 0)
|
|
|
|
tp->t_rttvar = 1;
|
2002-08-17 18:26:02 +00:00
|
|
|
if (tp->t_rttbest > tp->t_srtt + tp->t_rttvar)
|
|
|
|
tp->t_rttbest = tp->t_srtt + tp->t_rttvar;
|
1996-03-22 18:09:21 +00:00
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* No rtt measurement yet - use the unsmoothed rtt.
|
|
|
|
* Set the variance to half the rtt (so our first
|
|
|
|
* retransmit happens at 3*rtt).
|
|
|
|
*/
|
|
|
|
tp->t_srtt = rtt << TCP_RTT_SHIFT;
|
|
|
|
tp->t_rttvar = rtt << (TCP_RTTVAR_SHIFT - 1);
|
2002-08-17 18:26:02 +00:00
|
|
|
tp->t_rttbest = tp->t_srtt + tp->t_rttvar;
|
1996-03-22 18:09:21 +00:00
|
|
|
}
|
1999-08-30 21:17:07 +00:00
|
|
|
tp->t_rtttime = 0;
|
1994-05-24 10:09:53 +00:00
|
|
|
tp->t_rxtshift = 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* the retransmit should happen at rtt + 4 * rttvar.
|
|
|
|
* Because of the way we do the smoothing, srtt and rttvar
|
|
|
|
* will each average +1/2 tick of bias. When we compute
|
|
|
|
* the retransmit timer, we want 1/2 tick of rounding and
|
|
|
|
* 1 extra tick because of +-1/2 tick uncertainty in the
|
|
|
|
* firing of the timer. The bias will give us exactly the
|
|
|
|
* 1.5 tick we need. But, because the bias is
|
|
|
|
* statistical, we have to test that we don't drop below
|
|
|
|
* the minimum feasible timer (which is 2 ticks).
|
|
|
|
*/
|
1996-03-22 18:09:21 +00:00
|
|
|
TCPT_RANGESET(tp->t_rxtcur, TCP_REXMTVAL(tp),
|
1996-03-25 20:13:21 +00:00
|
|
|
max(tp->t_rttmin, rtt + 2), TCPTV_REXMTMAX);
|
1995-05-30 08:16:23 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* We received an ack for a packet that wasn't retransmitted;
|
|
|
|
* it is probably safe to discard any error indications we've
|
|
|
|
* received recently. This isn't quite right, but close enough
|
|
|
|
* for now (a route might have failed after we sent a segment,
|
|
|
|
* and the return path might not be symmetrical).
|
|
|
|
*/
|
|
|
|
tp->t_softerror = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Determine a reasonable value for maxseg size.
|
|
|
|
* If the route is known, check route for mtu.
|
2012-10-28 18:33:52 +00:00
|
|
|
* If none, use an mss that can be handled on the outgoing interface
|
|
|
|
* without forcing IP to fragment. If no route is found, route has no mtu,
|
1994-05-24 10:09:53 +00:00
|
|
|
* or the destination isn't local, use a default, hopefully conservative
|
|
|
|
* size (usually 512 or the default IP max size, but no more than the mtu
|
|
|
|
* of the interface), as we can't discover anything about intervening
|
|
|
|
* gateways or networks. We also initialize the congestion/slow start
|
|
|
|
* window to be a single segment if the destination isn't local.
|
|
|
|
* While looking at the routing entry, we also initialize other path-dependent
|
|
|
|
* parameters from pre-set or cached values in the routing entry.
|
1995-02-09 23:13:27 +00:00
|
|
|
*
|
|
|
|
* Also take into account the space needed for options that we
|
|
|
|
* send regularly. Make maxseg shorter by that amount to assure
|
|
|
|
* that we can send maxseg amount of data even when the options
|
|
|
|
* are present. Store the upper limit of the length of options plus
|
|
|
|
* data in maxopd.
|
|
|
|
*
|
2003-11-20 20:07:39 +00:00
|
|
|
* NOTE that this routine is only called when we process an incoming
|
2012-04-16 13:49:03 +00:00
|
|
|
* segment, or an ICMP need fragmentation datagram. Outgoing SYN/ACK MSS
|
|
|
|
* settings are handled in tcp_mssopt().
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1995-02-09 23:13:27 +00:00
|
|
|
void
|
2012-04-16 13:49:03 +00:00
|
|
|
tcp_mss_update(struct tcpcb *tp, int offer, int mtuoffer,
|
2008-11-06 13:25:59 +00:00
|
|
|
struct hc_metrics_lite *metricptr, int *mtuflags)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2011-04-30 11:21:29 +00:00
|
|
|
int mss = 0;
|
|
|
|
u_long maxmtu = 0;
|
2002-08-17 02:05:25 +00:00
|
|
|
struct inpcb *inp = tp->t_inpcb;
|
2003-11-20 20:07:39 +00:00
|
|
|
struct hc_metrics_lite metrics;
|
2012-04-16 13:49:03 +00:00
|
|
|
int origoffer;
|
2000-01-09 19:17:30 +00:00
|
|
|
#ifdef INET6
|
2002-08-17 02:05:25 +00:00
|
|
|
int isipv6 = ((inp->inp_vflag & INP_IPV6) != 0) ? 1 : 0;
|
|
|
|
size_t min_protoh = isipv6 ?
|
|
|
|
sizeof (struct ip6_hdr) + sizeof (struct tcphdr) :
|
|
|
|
sizeof (struct tcpiphdr);
|
2000-01-09 19:17:30 +00:00
|
|
|
#else
|
2003-11-20 20:07:39 +00:00
|
|
|
const size_t min_protoh = sizeof(struct tcpiphdr);
|
2000-01-09 19:17:30 +00:00
|
|
|
#endif
|
2002-08-17 02:05:25 +00:00
|
|
|
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
2008-04-07 12:41:45 +00:00
|
|
|
|
2012-04-16 13:49:03 +00:00
|
|
|
if (mtuoffer != -1) {
|
|
|
|
KASSERT(offer == -1, ("%s: conflict", __func__));
|
|
|
|
offer = mtuoffer - min_protoh;
|
|
|
|
}
|
|
|
|
origoffer = offer;
|
|
|
|
|
2007-06-10 20:59:22 +00:00
|
|
|
/* Initialize. */
|
2003-11-20 20:07:39 +00:00
|
|
|
#ifdef INET6
|
|
|
|
if (isipv6) {
|
2008-11-06 13:25:59 +00:00
|
|
|
maxmtu = tcp_maxmtu6(&inp->inp_inc, mtuflags);
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
tp->t_maxopd = tp->t_maxseg = V_tcp_v6mssdflt;
|
2011-04-30 11:21:29 +00:00
|
|
|
}
|
2003-11-20 20:07:39 +00:00
|
|
|
#endif
|
2011-04-30 11:21:29 +00:00
|
|
|
#if defined(INET) && defined(INET6)
|
|
|
|
else
|
|
|
|
#endif
|
|
|
|
#ifdef INET
|
2003-11-20 20:07:39 +00:00
|
|
|
{
|
2008-11-06 13:25:59 +00:00
|
|
|
maxmtu = tcp_maxmtu(&inp->inp_inc, mtuflags);
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
tp->t_maxopd = tp->t_maxseg = V_tcp_mssdflt;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif
|
1994-05-24 10:09:53 +00:00
|
|
|
|
1995-02-09 23:13:27 +00:00
|
|
|
/*
|
2007-06-10 20:59:22 +00:00
|
|
|
* No route to sender, stay with default mss and return.
|
1995-02-09 23:13:27 +00:00
|
|
|
*/
|
2008-11-06 12:33:33 +00:00
|
|
|
if (maxmtu == 0) {
|
|
|
|
/*
|
2008-11-06 16:30:20 +00:00
|
|
|
* In case we return early we need to initialize metrics
|
2008-11-06 12:33:33 +00:00
|
|
|
* to a defined state as tcp_hc_get() would do for us
|
|
|
|
* if there was no cache hit.
|
|
|
|
*/
|
|
|
|
if (metricptr != NULL)
|
|
|
|
bzero(metricptr, sizeof(struct hc_metrics_lite));
|
2003-11-20 20:07:39 +00:00
|
|
|
return;
|
2008-11-06 12:33:33 +00:00
|
|
|
}
|
2003-11-20 20:07:39 +00:00
|
|
|
|
2007-06-10 20:59:22 +00:00
|
|
|
/* What have we got? */
|
2003-11-20 20:07:39 +00:00
|
|
|
switch (offer) {
|
|
|
|
case 0:
|
|
|
|
/*
|
|
|
|
* Offer == 0 means that there was no MSS on the SYN
|
2008-03-02 08:40:47 +00:00
|
|
|
* segment, in this case we use tcp_mssdflt as
|
|
|
|
* already assigned to t_maxopd above.
|
2003-11-20 20:07:39 +00:00
|
|
|
*/
|
2008-03-02 08:40:47 +00:00
|
|
|
offer = tp->t_maxopd;
|
2003-11-20 20:07:39 +00:00
|
|
|
break;
|
|
|
|
|
|
|
|
case -1:
|
|
|
|
/*
|
2004-11-02 22:22:22 +00:00
|
|
|
* Offer == -1 means that we didn't receive SYN yet.
|
2003-11-20 20:07:39 +00:00
|
|
|
*/
|
|
|
|
/* FALLTHROUGH */
|
|
|
|
|
|
|
|
default:
|
2004-01-08 17:40:07 +00:00
|
|
|
/*
|
|
|
|
* Prevent DoS attack with too small MSS. Round up
|
|
|
|
* to at least minmss.
|
|
|
|
*/
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
offer = max(offer, V_tcp_minmss);
|
2003-11-20 20:07:39 +00:00
|
|
|
}
|
1995-02-09 23:13:27 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
2007-06-10 20:59:22 +00:00
|
|
|
* rmx information is now retrieved from tcp_hostcache.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2003-11-20 20:07:39 +00:00
|
|
|
tcp_hc_get(&inp->inp_inc, &metrics);
|
2008-09-07 18:50:25 +00:00
|
|
|
if (metricptr != NULL)
|
|
|
|
bcopy(&metrics, metricptr, sizeof(struct hc_metrics_lite));
|
2003-11-20 20:07:39 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
2007-06-10 20:59:22 +00:00
|
|
|
* If there's a discovered mtu int tcp hostcache, use it
|
2000-01-09 19:17:30 +00:00
|
|
|
* else, use the link mtu.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2003-11-20 20:07:39 +00:00
|
|
|
if (metrics.rmx_mtu)
|
2004-04-23 22:44:59 +00:00
|
|
|
mss = min(metrics.rmx_mtu, maxmtu) - min_protoh;
|
2002-08-17 02:05:25 +00:00
|
|
|
else {
|
2003-10-20 16:19:01 +00:00
|
|
|
#ifdef INET6
|
2000-01-09 19:17:30 +00:00
|
|
|
if (isipv6) {
|
2003-11-20 20:07:39 +00:00
|
|
|
mss = maxmtu - min_protoh;
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
if (!V_path_mtu_discovery &&
|
2003-11-20 20:07:39 +00:00
|
|
|
!in6_localaddr(&inp->in6p_faddr))
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
mss = min(mss, V_tcp_v6mssdflt);
|
2011-04-30 11:21:29 +00:00
|
|
|
}
|
2003-10-20 16:19:01 +00:00
|
|
|
#endif
|
2011-04-30 11:21:29 +00:00
|
|
|
#if defined(INET) && defined(INET6)
|
|
|
|
else
|
|
|
|
#endif
|
|
|
|
#ifdef INET
|
2003-11-20 20:07:39 +00:00
|
|
|
{
|
|
|
|
mss = maxmtu - min_protoh;
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
if (!V_path_mtu_discovery &&
|
2003-11-20 20:07:39 +00:00
|
|
|
!in_localaddr(inp->inp_faddr))
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
mss = min(mss, V_tcp_mssdflt);
|
2003-11-20 20:07:39 +00:00
|
|
|
}
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif
|
2008-09-07 18:50:25 +00:00
|
|
|
/*
|
|
|
|
* XXX - The above conditional (mss = maxmtu - min_protoh)
|
|
|
|
* probably violates the TCP spec.
|
|
|
|
* The problem is that, since we don't know the
|
|
|
|
* other end's MSS, we are supposed to use a conservative
|
|
|
|
* default. But, if we do that, then MTU discovery will
|
|
|
|
* never actually take place, because the conservative
|
|
|
|
* default is much less than the MTUs typically seen
|
|
|
|
* on the Internet today. For the moment, we'll sweep
|
|
|
|
* this under the carpet.
|
|
|
|
*
|
|
|
|
* The conservative default might not actually be a problem
|
|
|
|
* if the only case this occurs is when sending an initial
|
|
|
|
* SYN with options and data to a host we've never talked
|
|
|
|
* to before. Then, they will reply with an MSS value which
|
|
|
|
* will get recorded and the new parameters should get
|
|
|
|
* recomputed. For Further Study.
|
|
|
|
*/
|
1995-02-09 23:13:27 +00:00
|
|
|
}
|
|
|
|
mss = min(mss, offer);
|
2003-11-20 20:07:39 +00:00
|
|
|
|
2008-09-07 18:50:25 +00:00
|
|
|
/*
|
|
|
|
* Sanity check: make sure that maxopd will be large
|
|
|
|
* enough to allow some data on segments even if the
|
|
|
|
* all the option space is used (40bytes). Otherwise
|
|
|
|
* funny things may happen in tcp_output.
|
|
|
|
*/
|
|
|
|
mss = max(mss, 64);
|
|
|
|
|
1995-02-09 23:13:27 +00:00
|
|
|
/*
|
|
|
|
* maxopd stores the maximum length of data AND options
|
|
|
|
* in a segment; maxseg is the amount of data in a normal
|
|
|
|
* segment. We need to store this value (maxopd) apart
|
|
|
|
* from maxseg, because now every segment carries options
|
|
|
|
* and thus we normally have somewhat less data in segments.
|
|
|
|
*/
|
|
|
|
tp->t_maxopd = mss;
|
|
|
|
|
|
|
|
/*
|
2007-06-10 20:59:22 +00:00
|
|
|
* origoffer==-1 indicates that no segments were received yet.
|
2004-11-02 22:22:22 +00:00
|
|
|
* In this case we just guess.
|
1995-02-09 23:13:27 +00:00
|
|
|
*/
|
2004-08-16 18:32:07 +00:00
|
|
|
if ((tp->t_flags & (TF_REQ_TSTMP|TF_NOOPT)) == TF_REQ_TSTMP &&
|
1995-02-09 23:13:27 +00:00
|
|
|
(origoffer == -1 ||
|
|
|
|
(tp->t_flags & TF_RCVD_TSTMP) == TF_RCVD_TSTMP))
|
|
|
|
mss -= TCPOLEN_TSTAMP_APPA;
|
|
|
|
|
2003-11-20 20:07:39 +00:00
|
|
|
tp->t_maxseg = mss;
|
2008-09-07 18:50:25 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
tcp_mss(struct tcpcb *tp, int offer)
|
|
|
|
{
|
2010-11-12 06:41:55 +00:00
|
|
|
int mss;
|
2008-09-07 18:50:25 +00:00
|
|
|
u_long bufsize;
|
|
|
|
struct inpcb *inp;
|
|
|
|
struct socket *so;
|
|
|
|
struct hc_metrics_lite metrics;
|
2008-11-06 13:25:59 +00:00
|
|
|
int mtuflags = 0;
|
2010-11-12 06:41:55 +00:00
|
|
|
|
2008-09-07 18:50:25 +00:00
|
|
|
KASSERT(tp != NULL, ("%s: tp == NULL", __func__));
|
|
|
|
|
2012-04-16 13:49:03 +00:00
|
|
|
tcp_mss_update(tp, offer, -1, &metrics, &mtuflags);
|
2008-09-07 18:50:25 +00:00
|
|
|
|
|
|
|
mss = tp->t_maxseg;
|
|
|
|
inp = tp->t_inpcb;
|
2003-11-20 20:07:39 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
2003-11-20 20:07:39 +00:00
|
|
|
* If there's a pipesize, change the socket buffer to that size,
|
|
|
|
* don't change if sb_hiwat is different than default (then it
|
|
|
|
* has been changed on purpose with setsockopt).
|
|
|
|
* Make the socket buffers an integral number of mss units;
|
|
|
|
* if the mss is larger than the socket buffer, decrease the mss.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2008-03-02 08:40:47 +00:00
|
|
|
so = inp->inp_socket;
|
2004-06-24 01:37:04 +00:00
|
|
|
SOCKBUF_LOCK(&so->so_snd);
|
2011-10-16 15:08:43 +00:00
|
|
|
if ((so->so_snd.sb_hiwat == V_tcp_sendspace) && metrics.rmx_sendpipe)
|
2003-11-20 20:07:39 +00:00
|
|
|
bufsize = metrics.rmx_sendpipe;
|
|
|
|
else
|
1995-02-09 23:13:27 +00:00
|
|
|
bufsize = so->so_snd.sb_hiwat;
|
|
|
|
if (bufsize < mss)
|
|
|
|
mss = bufsize;
|
|
|
|
else {
|
|
|
|
bufsize = roundup(bufsize, mss);
|
|
|
|
if (bufsize > sb_max)
|
|
|
|
bufsize = sb_max;
|
2002-07-22 22:31:09 +00:00
|
|
|
if (bufsize > so->so_snd.sb_hiwat)
|
2004-06-24 01:37:04 +00:00
|
|
|
(void)sbreserve_locked(&so->so_snd, bufsize, so, NULL);
|
1995-02-09 23:13:27 +00:00
|
|
|
}
|
2004-06-24 01:37:04 +00:00
|
|
|
SOCKBUF_UNLOCK(&so->so_snd);
|
1995-02-09 23:13:27 +00:00
|
|
|
tp->t_maxseg = mss;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2004-06-24 01:37:04 +00:00
|
|
|
SOCKBUF_LOCK(&so->so_rcv);
|
2011-10-16 15:08:43 +00:00
|
|
|
if ((so->so_rcv.sb_hiwat == V_tcp_recvspace) && metrics.rmx_recvpipe)
|
2003-11-20 20:07:39 +00:00
|
|
|
bufsize = metrics.rmx_recvpipe;
|
|
|
|
else
|
1995-02-09 23:13:27 +00:00
|
|
|
bufsize = so->so_rcv.sb_hiwat;
|
|
|
|
if (bufsize > mss) {
|
|
|
|
bufsize = roundup(bufsize, mss);
|
|
|
|
if (bufsize > sb_max)
|
|
|
|
bufsize = sb_max;
|
2002-07-22 22:31:09 +00:00
|
|
|
if (bufsize > so->so_rcv.sb_hiwat)
|
2004-06-24 01:37:04 +00:00
|
|
|
(void)sbreserve_locked(&so->so_rcv, bufsize, so, NULL);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
2004-06-24 01:37:04 +00:00
|
|
|
SOCKBUF_UNLOCK(&so->so_rcv);
|
2008-11-06 13:25:59 +00:00
|
|
|
|
|
|
|
/* Check the interface for TSO capabilities. */
|
|
|
|
if (mtuflags & CSUM_TSO)
|
|
|
|
tp->t_flags |= TF_TSO;
|
1995-02-09 23:13:27 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Determine the MSS option to send on an outgoing SYN.
|
|
|
|
*/
|
|
|
|
int
|
2007-03-21 19:37:55 +00:00
|
|
|
tcp_mssopt(struct in_conninfo *inc)
|
1995-02-09 23:13:27 +00:00
|
|
|
{
|
2003-11-20 20:07:39 +00:00
|
|
|
int mss = 0;
|
|
|
|
u_long maxmtu = 0;
|
|
|
|
u_long thcmtu = 0;
|
|
|
|
size_t min_protoh;
|
2002-08-17 02:05:25 +00:00
|
|
|
|
2003-11-20 20:07:39 +00:00
|
|
|
KASSERT(inc != NULL, ("tcp_mssopt with NULL in_conninfo pointer"));
|
1995-02-09 23:13:27 +00:00
|
|
|
|
2003-10-20 15:27:48 +00:00
|
|
|
#ifdef INET6
|
2008-12-17 12:52:34 +00:00
|
|
|
if (inc->inc_flags & INC_ISIPV6) {
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
mss = V_tcp_v6mssdflt;
|
2006-09-06 21:51:59 +00:00
|
|
|
maxmtu = tcp_maxmtu6(inc, NULL);
|
2003-11-20 20:07:39 +00:00
|
|
|
min_protoh = sizeof(struct ip6_hdr) + sizeof(struct tcphdr);
|
2011-04-30 11:21:29 +00:00
|
|
|
}
|
|
|
|
#endif
|
|
|
|
#if defined(INET) && defined(INET6)
|
|
|
|
else
|
2003-10-20 15:27:48 +00:00
|
|
|
#endif
|
2011-04-30 11:21:29 +00:00
|
|
|
#ifdef INET
|
2003-11-20 20:07:39 +00:00
|
|
|
{
|
Commit step 1 of the vimage project, (network stack)
virtualization work done by Marko Zec (zec@).
This is the first in a series of commits over the course
of the next few weeks.
Mark all uses of global variables to be virtualized
with a V_ prefix.
Use macros to map them back to their global names for
now, so this is a NOP change only.
We hope to have caught at least 85-90% of what is needed
so we do not invalidate a lot of outstanding patches again.
Obtained from: //depot/projects/vimage-commit2/...
Reviewed by: brooks, des, ed, mav, julian,
jamie, kris, rwatson, zec, ...
(various people I forgot, different versions)
md5 (with a bit of help)
Sponsored by: NLnet Foundation, The FreeBSD Foundation
X-MFC after: never
V_Commit_Message_Reviewed_By: more people than the patch
2008-08-17 23:27:27 +00:00
|
|
|
mss = V_tcp_mssdflt;
|
2006-09-06 21:51:59 +00:00
|
|
|
maxmtu = tcp_maxmtu(inc, NULL);
|
2003-11-20 20:07:39 +00:00
|
|
|
min_protoh = sizeof(struct tcpiphdr);
|
|
|
|
}
|
2011-04-30 11:21:29 +00:00
|
|
|
#endif
|
2012-05-25 01:13:39 +00:00
|
|
|
#if defined(INET6) || defined(INET)
|
|
|
|
thcmtu = tcp_hc_getmtu(inc); /* IPv4 and IPv6 */
|
|
|
|
#endif
|
|
|
|
|
2003-11-20 20:07:39 +00:00
|
|
|
if (maxmtu && thcmtu)
|
|
|
|
mss = min(maxmtu, thcmtu) - min_protoh;
|
|
|
|
else if (maxmtu || thcmtu)
|
|
|
|
mss = max(maxmtu, thcmtu) - min_protoh;
|
|
|
|
|
|
|
|
return (mss);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
2000-05-06 03:31:09 +00:00
|
|
|
|
|
|
|
|
|
|
|
/*
|
2002-08-17 02:05:25 +00:00
|
|
|
* On a partial ack arrives, force the retransmission of the
|
|
|
|
* next unacknowledged segment. Do not clear tp->t_dupacks.
|
|
|
|
* By setting snd_nxt to ti_ack, this forces retransmission timer to
|
|
|
|
* be started again.
|
2000-05-06 03:31:09 +00:00
|
|
|
*/
|
2002-08-17 02:05:25 +00:00
|
|
|
static void
|
2007-03-21 19:37:55 +00:00
|
|
|
tcp_newreno_partial_ack(struct tcpcb *tp, struct tcphdr *th)
|
2000-05-06 03:31:09 +00:00
|
|
|
{
|
2002-08-17 02:05:25 +00:00
|
|
|
tcp_seq onxt = tp->snd_nxt;
|
|
|
|
u_long ocwnd = tp->snd_cwnd;
|
2000-05-06 03:31:09 +00:00
|
|
|
|
2008-04-17 21:38:18 +00:00
|
|
|
INP_WLOCK_ASSERT(tp->t_inpcb);
|
2008-04-07 12:41:45 +00:00
|
|
|
|
2007-04-11 09:45:16 +00:00
|
|
|
tcp_timer_activate(tp, TT_REXMT, 0);
|
2002-08-17 02:05:25 +00:00
|
|
|
tp->t_rtttime = 0;
|
|
|
|
tp->snd_nxt = th->th_ack;
|
|
|
|
/*
|
|
|
|
* Set snd_cwnd to one segment beyond acknowledged offset.
|
|
|
|
* (tp->snd_una has not yet been updated when this function is called.)
|
|
|
|
*/
|
2010-11-12 06:41:55 +00:00
|
|
|
tp->snd_cwnd = tp->t_maxseg + BYTES_THIS_ACK(tp, th);
|
2002-09-30 18:55:45 +00:00
|
|
|
tp->t_flags |= TF_ACKNOW;
|
2002-08-17 02:05:25 +00:00
|
|
|
(void) tcp_output(tp);
|
|
|
|
tp->snd_cwnd = ocwnd;
|
|
|
|
if (SEQ_GT(onxt, tp->snd_nxt))
|
|
|
|
tp->snd_nxt = onxt;
|
|
|
|
/*
|
|
|
|
* Partial window deflation. Relies on fact that tp->snd_una
|
|
|
|
* not updated yet.
|
|
|
|
*/
|
2010-11-12 06:41:55 +00:00
|
|
|
if (tp->snd_cwnd > BYTES_THIS_ACK(tp, th))
|
|
|
|
tp->snd_cwnd -= BYTES_THIS_ACK(tp, th);
|
2005-07-05 19:23:02 +00:00
|
|
|
else
|
|
|
|
tp->snd_cwnd = 0;
|
|
|
|
tp->snd_cwnd += tp->t_maxseg;
|
2000-05-06 03:31:09 +00:00
|
|
|
}
|