Commit Graph

91 Commits

Author SHA1 Message Date
Gleb Smirnoff
687adb703d Refactor the name hash and the ID hash, that are used to address nodes:
- Make hash sizes growable, to satisfy users running large mpd
  installations, having thousands of nodes.
- NG_NAMEHASH() proved to give a very bad distribution in real life
  name sets, while generic hash32_str(name, HASHINIT) proved to give
  an even one, so you the latter for name hash.
- Do not store unnamed nodes in slot 0 of name hash, no reason for that.
- Use the ID hash in cases when we need to run through all nodes: the
  NGM_LISTNODES command and in the vnet_netgraph_uninit().
- Implement NGM_LISTNODES and NGM_LISTNAMES as separate code, the former
  iterates through the ID hash, and the latter through the name hash.
- Keep count of all nodes and of named nodes, so that we don't need
  to count nodes in NGM_LISTNODES and NGM_LISTNAMES. The counters are
  also used to estimate whether we need to grow hashes.
- Close a race between two threads running ng_name_node() assigning same
  name to different nodes.
2012-02-16 19:10:01 +00:00
Max Khon
707d205808 Constify "address" argument of ng_address_path(). 2011-11-06 05:20:27 +00:00
Gleb Smirnoff
e5fe87b387 - If KDB & NETGRAPH_DEBUG are on, print traces on discovered failed
invariants.
- Reduce tautology in NETGRAPH_DEBUG output.
2011-10-27 09:43:25 +00:00
Gleb Smirnoff
d33dc2fa5c Fix build with NETGRAPH_DEBUG. 2011-07-04 20:50:09 +00:00
Gleb Smirnoff
f8dd68c912 Fix build with NETGRAPH_DEBUG. 2011-07-04 13:55:55 +00:00
Gleb Smirnoff
3fbdf77459 - Use refcount(9) API to manage node and hook refcounting.
- Make ng_unref_node() void, since caller shouldn't be
  interested in whether node is valid after call or not,
  since it can't be guaranteed to be valid. [1]

Ok from:	julian [1]
2011-07-04 07:03:44 +00:00
Robert Watson
d0728d7174 Introduce and use a sysinit-based initialization scheme for virtual
network stacks, VNET_SYSINIT:

- Add VNET_SYSINIT and VNET_SYSUNINIT macros to declare events that will
  occur each time a network stack is instantiated and destroyed.  In the
  !VIMAGE case, these are simply mapped into regular SYSINIT/SYSUNINIT.
  For the VIMAGE case, we instead use SYSINIT's to track their order and
  properties on registration, using them for each vnet when created/
  destroyed, or immediately on module load for already-started vnets.
- Remove vnet_modinfo mechanism that existed to serve this purpose
  previously, as well as its dependency scheme: we now just use the
  SYSINIT ordering scheme.
- Implement VNET_DOMAIN_SET() to allow protocol domains to declare that
  they want init functions to be called for each virtual network stack
  rather than just once at boot, compiling down to DOMAIN_SET() in the
  non-VIMAGE case.
- Walk all virtualized kernel subsystems and make use of these instead
  of modinfo or DOMAIN_SET() for init/uninit events.  In some cases,
  convert modular components from using modevent to using sysinit (where
  appropriate).  In some cases, do minor rejuggling of SYSINIT ordering
  to make room for or better manage events.

Portions submitted by:	jhb (VNET_SYSINIT), bz (cleanup)
Discussed with:		jhb, bz, julian, zec
Reviewed by:		bz
Approved by:		re (VIMAGE blanket)
2009-07-23 20:46:49 +00:00
Robert Watson
eddfbb763d Build on Jeff Roberson's linker-set based dynamic per-CPU allocator
(DPCPU), as suggested by Peter Wemm, and implement a new per-virtual
network stack memory allocator.  Modify vnet to use the allocator
instead of monolithic global container structures (vinet, ...).  This
change solves many binary compatibility problems associated with
VIMAGE, and restores ELF symbols for virtualized global variables.

Each virtualized global variable exists as a "reference copy", and also
once per virtual network stack.  Virtualized global variables are
tagged at compile-time, placing the in a special linker set, which is
loaded into a contiguous region of kernel memory.  Virtualized global
variables in the base kernel are linked as normal, but those in modules
are copied and relocated to a reserved portion of the kernel's vnet
region with the help of a the kernel linker.

Virtualized global variables exist in per-vnet memory set up when the
network stack instance is created, and are initialized statically from
the reference copy.  Run-time access occurs via an accessor macro, which
converts from the current vnet and requested symbol to a per-vnet
address.  When "options VIMAGE" is not compiled into the kernel, normal
global ELF symbols will be used instead and indirection is avoided.

This change restores static initialization for network stack global
variables, restores support for non-global symbols and types, eliminates
the need for many subsystem constructors, eliminates large per-subsystem
structures that caused many binary compatibility issues both for
monitoring applications (netstat) and kernel modules, removes the
per-function INIT_VNET_*() macros throughout the stack, eliminates the
need for vnet_symmap ksym(2) munging, and eliminates duplicate
definitions of virtualized globals under VIMAGE_GLOBALS.

Bump __FreeBSD_version and update UPDATING.

Portions submitted by:  bz
Reviewed by:            bz, zec
Discussed with:         gnn, jamie, jeff, jhb, julian, sam
Suggested by:           peter
Approved by:            re (kensmith)
2009-07-14 22:48:30 +00:00
Marko Zec
f089869fa5 Introduce a mechanism for detecting calls from outbound path of the
network stack when reentering the inbound path from netgraph, and
force queueing of mbufs at the outbound netgraph node.

The mechanism relies on two components.  First, in netgraph nodes
where outbound path of the network stack calls into netgraph, the
current thread has to be appropriately marked using the new
NG_OUTBOUND_THREAD_REF() macro before proceeding to call further
into the netgraph topology, and unmarked using the
NG_OUTBOUND_THREAD_UNREF() macro before returning to the caller.
Second, netgraph nodes which can potentially reenter the network
stack in the inbound path have to mark their inbound hooks using
NG_HOOK_SET_TO_INBOUND() macro.  The netgraph framework will then
detect when there is a danger of a call graph looping back from
outbound to inbound path via netgraph, and defer handing off the
mbufs to the "inbound" node to a worker thread with a clean stack.

In this first pass only the most obvious netgraph nodes have been
updated to ensure no outbound to inbound calls can occur.  Nodes
such as ng_ipfw, ng_gif etc. should be further examined whether a
potential for outbound to inbound call looping exists.

This commit changes the layout of struct thread, but due to
__FreeBSD_version number shortage a version bump has been omitted
at this time, nevertheless kernel and modules have to be rebuilt.

Reviewed by:	julian, rwatson, bz
Approved by:	julian (mentor)
2009-06-11 16:50:49 +00:00
Marko Zec
21ca7b57bd Change the curvnet variable from a global const struct vnet *,
previously always pointing to the default vnet context, to a
dynamically changing thread-local one.  The currvnet context
should be set on entry to networking code via CURVNET_SET() macros,
and reverted to previous state via CURVNET_RESTORE().  Recursions
on curvnet are permitted, though strongly discuouraged.

This change should have no functional impact on nooptions VIMAGE
kernel builds, where CURVNET_* macros expand to whitespace.

The curthread->td_vnet (aka curvnet) variable's purpose is to be an
indicator of the vnet context in which the current network-related
operation takes place, in case we cannot deduce the current vnet
context from any other source, such as by looking at mbuf's
m->m_pkthdr.rcvif->if_vnet, sockets's so->so_vnet etc.  Moreover, so
far curvnet has turned out to be an invaluable consistency checking
aid: it helps to catch cases when sockets, ifnets or any other
vnet-aware structures may have leaked from one vnet to another.

The exact placement of the CURVNET_SET() / CURVNET_RESTORE() macros
was a result of an empirical iterative process, whith an aim to
reduce recursions on CURVNET_SET() to a minimum, while still reducing
the scope of CURVNET_SET() to networking only operations - the
alternative would be calling CURVNET_SET() on each system call entry.
In general, curvnet has to be set in three typicall cases: when
processing socket-related requests from userspace or from within the
kernel; when processing inbound traffic flowing from device drivers
to upper layers of the networking stack, and when executing
timer-driven networking functions.

This change also introduces a DDB subcommand to show the list of all
vnet instances.

Approved by:	julian (mentor)
2009-05-05 10:56:12 +00:00
Marko Zec
aef8f3445b In preparation to make options VIMAGE operational, where needed,
initialize / release netgraph related state in iattach() / idetach()
functions called via the vnet module registration / initialization
framework, instead of initialization / cleanups being done in
mod_event handlers.

While here, introduce a crude hack aimed at preventing ng_ether to
autoattach to ng_eiface ifnets, which are also netgraph nodes already.

Reviewed by:	bz
Approved by:	julian (mentor)
2009-04-26 07:14:50 +00:00
Alexander Motin
15cea89fab Revert rev. 183277:
Remove ng_rmnode_flags() function.
ng_rmnode_self() was made to be called only while having node locked.
When node is properly locked, any function call sent to it will always be
queued. So turning ng_rmnode_self() into the ng_rmnode_flags() is not just
meaningless, but incorrent, as it violates node locking when called outside.

No objections:	julian, thompsa
2008-12-13 22:26:24 +00:00
Marko Zec
385195c062 Conditionally compile out V_ globals while instantiating the appropriate
container structures, depending on VIMAGE_GLOBALS compile time option.

Make VIMAGE_GLOBALS a new compile-time option, which by default will not
be defined, resulting in instatiations of global variables selected for
V_irtualization (enclosed in #ifdef VIMAGE_GLOBALS blocks) to be
effectively compiled out.  Instantiate new global container structures
to hold V_irtualized variables: vnet_net_0, vnet_inet_0, vnet_inet6_0,
vnet_ipsec_0, vnet_netgraph_0, and vnet_gif_0.

Update the VSYM() macro so that depending on VIMAGE_GLOBALS the V_
macros resolve either to the original globals, or to fields inside
container structures, i.e. effectively

#ifdef VIMAGE_GLOBALS
#define V_rt_tables rt_tables
#else
#define V_rt_tables vnet_net_0._rt_tables
#endif

Update SYSCTL_V_*() macros to operate either on globals or on fields
inside container structs.

Extend the internal kldsym() lookups with the ability to resolve
selected fields inside the virtualization container structs.  This
applies only to the fields which are explicitly registered for kldsym()
visibility via VNET_MOD_DECLARE() and vnet_mod_register(), currently
this is done only in sys/net/if.c.

Fix a few broken instances of MODULE_GLOBAL() macro use in SCTP code,
and modify the MODULE_GLOBAL() macro to resolve to V_ macros, which in
turn result in proper code being generated depending on VIMAGE_GLOBALS.

De-virtualize local static variables in sys/contrib/pf/net/pf_subr.c
which were prematurely V_irtualized by automated V_ prepending scripts
during earlier merging steps.  PF virtualization will be done
separately, most probably after next PF import.

Convert a few variable initializations at instantiation to
initialization in init functions, most notably in ipfw.  Also convert
TUNABLE_INT() initializers for V_ variables to TUNABLE_FETCH_INT() in
initializer functions.

Discussed at:	devsummit Strassburg
Reviewed by:	bz, julian
Approved by:	julian (mentor)
Obtained from:	//depot/projects/vimage-commit2/...
X-MFC after:	never
Sponsored by:	NLnet Foundation, The FreeBSD Foundation
2008-12-10 23:12:39 +00:00
Marko Zec
f02493cbbd Unhide declarations of network stack virtualization structs from
underneath #ifdef VIMAGE blocks.

This change introduces some churn in #include ordering and nesting
throughout the network stack and drivers but is not expected to cause
any additional issues.

In the next step this will allow us to instantiate the virtualization
container structures and switch from using global variables to their
"containerized" counterparts.

Reviewed by:	bz, julian
Approved by:	julian (mentor)
Obtained from:	//depot/projects/vimage-commit2/...
X-MFC after:	never
Sponsored by:	NLnet Foundation, The FreeBSD Foundation
2008-11-28 23:30:51 +00:00
Dag-Erling Smørgrav
1ede983cc9 Retire the MALLOC and FREE macros. They are an abomination unto style(9).
MFC after:	3 months
2008-10-23 15:53:51 +00:00
Marko Zec
8b615593fc Step 1.5 of importing the network stack virtualization infrastructure
from the vimage project, as per plan established at devsummit 08/08:
http://wiki.freebsd.org/Image/Notes200808DevSummit

Introduce INIT_VNET_*() initializer macros, VNET_FOREACH() iterator
macros, and CURVNET_SET() context setting macros, all currently
resolving to NOPs.

Prepare for virtualization of selected SYSCTL objects by introducing a
family of SYSCTL_V_*() macros, currently resolving to their global
counterparts, i.e. SYSCTL_V_INT() == SYSCTL_INT().

Move selected #defines from sys/sys/vimage.h to newly introduced header
files specific to virtualized subsystems (sys/net/vnet.h,
sys/netinet/vinet.h etc.).

All the changes are verified to have zero functional impact at this
point in time by doing MD5 comparision between pre- and post-change
object files(*).

(*) netipsec/keysock.c did not validate depending on compile time options.

Implemented by:	julian, bz, brooks, zec
Reviewed by:	julian, bz, brooks, kris, rwatson, ...
Approved by:	julian (mentor)
Obtained from:	//depot/projects/vimage-commit2/...
X-MFC after:	never
Sponsored by:	NLnet Foundation, The FreeBSD Foundation
2008-10-02 15:37:58 +00:00
Andrew Thompson
89ac5af9c1 Add ng_rmnode_flags() so the caller can pass NG_QUEUE and have the node
destroyed asynchronously due to locking or other constraints.

Reviewed by:	julian
2008-09-22 19:28:18 +00:00
George V. Neville-Neil
a13c239b91 Make it simpler to build netgraph modules outside of the kernel source
tree.  This change follows similar ones in the device tree.

MFC after:	2 weeks
2008-06-24 18:49:49 +00:00
Alexander Motin
9852972bb5 Several changes breaking netgraph module ABI collected together:
- reorder structures fields (XX_refs) a bit to group fields modified
   same time together. According to my tests it gives up to 10%
   SMP performance benefit on real workload due to reduced inter-CPU
   cache trashing.
 - change q_flags from long to int as long is not really needed there and
   it's usage with atomics is argued by some people.
 - move NGF_WORKQ flag into the separate field q_flags2 as it protected by
   queue mutex instead of node writer protection used by the rest of flags.
 - move nd_work queue entry to ng_queue structure to which it is more
   related and make it STAILQ instead of TAILQ as now it is a classic FIFO.
 - remove q_node pointer from ng_queue structure as it is not really needed.
 - reimplement item queue using STAILQ instead of own equal implementation.
   As soon as BT subsystem has own item queues using ng_item.el_next update
   it also.
 - change depth field in ng_item from uintptr_t to u_int. It was made
   uintptr_t to keep ABI compatibility.

Reviewed by:	julian, emax
Tested with:	Netperf cluster
2008-04-15 21:15:32 +00:00
Alexander Motin
10e873189c Improve apply callback error reporting:
Before this patch callback returned result of the last finished call chain.
Now it returns last nonzero result from all call chain results in this request.

As soon as this improvement gives reliable error reporting, it is now possible
to remove dirty workaround in ng_socket, made to return ENOBUFS error statuses
of request-response operations. That workaround was responsible for returning
ENOBUFS errors to completely unrelated requests working at the same time
on socket.
2008-03-11 21:58:48 +00:00
Alexander Motin
81a253a4ed Implement stack protection based on GET_STACK_USAGE() macro.
This fixes system panics possible with complicated netgraph setups
and allows to avoid unneded extra queueing for stack unwrapping.
2008-01-31 08:51:48 +00:00
Gleb Smirnoff
b332b91f74 - Merge all the ng_send_fn2* functions into one - ng_send_fn2(),
removing some copy&pasted code.
- Reduce copy and paste in ng_apply_item().
- Resurrect ng_send_fn() as a valid symbol, not a define.

Reviewed by:	mav, julian
2007-11-14 11:25:58 +00:00
Alexander Motin
e088dd4c44 Implement new apply callback mechanism to handle item forwarding.
When item forwarded refence counter is incremented, when item
processed, counter decremented. When counter reaches zero,
apply handler is getting called.
Now it allows to report right connect() call status from user-level
at the right time.
2007-10-19 15:04:17 +00:00
Robert Watson
c6b2899785 Replace references to NET_CALLOUT_MPSAFE with CALLOUT_MPSAFE, and remove
definition of NET_CALLOUT_MPSAFE, which is no longer required now that
debug.mpsafenet has been removed.

The once over:	bz
Approved by:	re (kensmith)
2007-07-28 07:31:30 +00:00
Gleb Smirnoff
c6964951c6 - ng_address_ID() has already freed the message, don't do double free.
- Get error from ng_address_ID().

Reported by:	Coverity via pjd
2006-10-17 10:59:39 +00:00
Gleb Smirnoff
27e216594b Use NET_CALLOUT_MPSAFE for netgraph callout initializer. 2006-06-06 08:05:27 +00:00
Gleb Smirnoff
7383c92c9f Provide additional macros for sending netgraph items, which allow
to use non-default flags for netgraph functions. Implement current
macros via new ones.
2006-01-12 22:44:04 +00:00
Gleb Smirnoff
22b286280c Remove old debugging leftover.
Reviewed by:	julian
2006-01-12 21:03:09 +00:00
Gleb Smirnoff
4c9b591060 Some whitespace and style cleanup. 2005-11-15 10:54:20 +00:00
Gleb Smirnoff
4be5933577 Rework the ng_item queueing on nodes:
- Introduce a new flags NGQF_QREADER and NGQF_QWRITER,
    which tell how the item should be actually applied,
    overriding NGQF_READER/NGQF_WRITER flags.
  - Do not differ between pending reader or writer. Use only
    one flag that is raised, when there are pending items.
  - Schedule netgraph ISR in ng_queue_rw(), so that callers
    do not need to do this job.
  - Fix several comments.

Submitted by:	julian
2005-11-02 14:27:24 +00:00
Gleb Smirnoff
fb3eb9dace Remove ng_callout_init_mtx() macro. Mutexed callouts are incompatible
with netgraph(4).
2005-09-12 07:41:31 +00:00
Gleb Smirnoff
5e067d612f Add ng_callout_ini_mtx() macro. 2005-08-30 09:44:54 +00:00
Julian Elischer
9722b2db99 reorder a couple of cases of void __inline to __inline void
to silence warnings from compilig with -W

Submitted by:	sandvine inc
MFC after:	1 week
2005-08-08 20:08:44 +00:00
Ruslan Ermilov
0515d4fd73 Fix up the comment. 2005-08-02 20:06:48 +00:00
Gleb Smirnoff
59087d62e7 Bump NG_ABI_VERSION. A number of ABI breakeges had been done since
RELENG_5 fork.
2005-07-21 12:32:52 +00:00
Gleb Smirnoff
25f8b7d9c3 Enhance struct ng_hook - add hk_type field. This field will describe
data link type of the hook. It will be used to ease autoconfiguration
of netgraph and also to print warning messages, when incompatoble nodes
are connected together.
2005-07-21 12:31:42 +00:00
Gleb Smirnoff
8afe16d57b In the splnet times, netgraph was functional and synchronous. Nowadays,
an item may be queued and processed later. While this is OK for mbufs,
this is a problem for control messages.

In the framework:
- Add optional callback function pointer to an item. When item gets
  applied the callback is executed from ng_apply_item().
- Add new flag NG_PROGRESS. If this flag is supplied, then return
  EINPROGRESS instead of 0 in case if item failed to deliver
  synchronously and was queued.
- Honor NG_PROGRESS in ng_snd_item().

In ng_socket:
- When userland sends control message add callback to the item.
- If ng_snd_item() returns EINPROGRESS, then sleep.

This change fixes possible races in ngctl(8) scripts.

Reviewed by:	julian
Approved by:	re (scottl)
2005-07-05 17:35:20 +00:00
Gleb Smirnoff
45b719ae22 Use NG_QUEUE instead of constant. 2005-05-16 19:52:42 +00:00
Gleb Smirnoff
42282202ee - Extend interface of ng_getqblk(), so that malloc wait flags are
specified by caller.
- Change ng_send_item() interface - use 'flags' argument instead of
  boolean 'queue'.
- Extend ng_send_fn(), ng_package_data() and ng_package_msg()
  interface - add possibility to pass flags. Rename ng_send_fn() to
  ng_send_fn1(). Create macro for ng_send_fn().
- Update all macros, that use ng_package_data() and ng_package_msg().

Reviewed by:	julian
2005-05-16 17:07:03 +00:00
Gleb Smirnoff
73e872668d Make NETGRAPH_DEBUG a kernel option, so that it can't be turned off
without hacking source.

In collaboration with:	ru, julian
2005-05-16 08:25:55 +00:00
Gleb Smirnoff
1489164f6b Use uma(9) for allocating netgraph items:
- ng_getqblk() simply runs uma_zalloc().
  - ng_free_item() simply frees.
  - ngq_mtx is pushed down under NETGRAPH_DEBUG.
  - NGQF_FREE is removed.

Increase default maxalloc to 512.

Reviewed by:	julian
2005-05-14 09:25:18 +00:00
Gleb Smirnoff
aacdb11479 Since there is no way to queue a function call to node, create
ng_queue_fn() - a queue version of ng_send_fn().
2005-05-13 11:35:02 +00:00
Gleb Smirnoff
1436fff881 Plug item leak in macro NG_RESPOND_MSG. Item was leaked when destination
node couldn't be addressed.

Submitted by:	Roselyn Lee
2005-03-10 19:34:20 +00:00
Gleb Smirnoff
687809752d Make netgraph ISR and callout MPSAFE.
Reviewed by:	rwatson, ru
2005-02-12 09:52:36 +00:00
Warner Losh
c398230b64 /* -> /*- for license, minor formatting changes 2005-01-07 01:45:51 +00:00
Gleb Smirnoff
f9d9e1b4ec Mechanically rename s/ng_timeout/ng_callout/g, s/ng_untimeout/ng_uncallout/g.
This is done to keep both versions in RELENG_5 and support both APIs.

Reviewed by:	scottl
Approved by:	julian (mentor), implicitly
2004-12-01 11:56:32 +00:00
Gleb Smirnoff
30bef41b8a - Make ng_timeout() to use callout() interface instead of timeout().
- Remove callout-hacking from ng_untimeout().

Approved by:	julian (mentor)
MFC after:	1 month
2004-11-02 21:24:30 +00:00
Julian Elischer
f036d4081d Align netgraph message fields ready for 64-bit (and 128 bit :-) machines.
requires a recompile of netgraph users.
Also change the size of a field in the bluetooth code
that was waiting for the next change that needed recompiles so
it could piggyback its way in.

Submitted by:	jdp, maksim
MFC after:	2 days
2004-08-20 01:24:23 +00:00
Gleb Smirnoff
7610f57454 When making a peer of unknown type framework tries to load module
using linker_load_module(). This works OK if NGM_MKPEER message came
from userland and we have process associated with thread. But when
NGM_MKPEER was queued because target node was busy, linker_load_module()
is called from netisr thread leading to panic.
To workaround that we do not load modules by framework, instead ng_socket
loads module (if this is required) before sending NGM_MKPEER.
However, the race condition between return from NgSendMsg() and actual
creation of node still exist and needs to be solved.

PR:		kern/62789
Approved by:	julian
2004-07-27 20:30:56 +00:00
Julian Elischer
be4252b367 Slight cosmetic changes.
Also introduce a macro to be called by persistent nodes to signal their
persistence during shutdown to hide this mechanism from the node author.

Make node flags have a consistent style in naming.

Document the change.
2004-07-20 17:15:38 +00:00