Commit Graph

1002 Commits

Author SHA1 Message Date
Kris Kennaway
c637bc9203 Replace callout_init(..., 1) with callout_init(..., CALLOUT_MPSAFE) for
better grep-compliance and to standardize with the rest of the kernel.

Reviewed by:	       jhb
MFC after:	       1 week
2008-04-16 16:47:14 +00:00
Alexander Motin
9852972bb5 Several changes breaking netgraph module ABI collected together:
- reorder structures fields (XX_refs) a bit to group fields modified
   same time together. According to my tests it gives up to 10%
   SMP performance benefit on real workload due to reduced inter-CPU
   cache trashing.
 - change q_flags from long to int as long is not really needed there and
   it's usage with atomics is argued by some people.
 - move NGF_WORKQ flag into the separate field q_flags2 as it protected by
   queue mutex instead of node writer protection used by the rest of flags.
 - move nd_work queue entry to ng_queue structure to which it is more
   related and make it STAILQ instead of TAILQ as now it is a classic FIFO.
 - remove q_node pointer from ng_queue structure as it is not really needed.
 - reimplement item queue using STAILQ instead of own equal implementation.
   As soon as BT subsystem has own item queues using ng_item.el_next update
   it also.
 - change depth field in ng_item from uintptr_t to u_int. It was made
   uintptr_t to keep ABI compatibility.

Reviewed by:	julian, emax
Tested with:	Netperf cluster
2008-04-15 21:15:32 +00:00
Alexander Motin
8f9ac44aa7 Add memory barriers to the node locking operations.
Add some comments.
2008-04-09 19:03:19 +00:00
Alexander Motin
394cb30a36 Rewrite node's r/w/q-lock semantics using only atomics instead of mutex
and atomics combination. Mutex is now used only for queue protection.
Also avoid unneded extra swi scheduling calls.
2008-04-06 15:26:32 +00:00
Alexander Motin
c77b232bb6 - Account all node stats at the shape mode.
- Do not check destination hook presence, it will be done by netgraph.
- Use u_int instead of int in some places to simplify type conversions.
- Use NG_SEND_DATA_ONLY() macro instead of selfmade equivalent.
2008-03-30 07:53:51 +00:00
Alexander Motin
018fe3d10e Use new atomic_fetchadd() primitive instead of looping atomic_cmpset(). 2008-03-30 00:27:48 +00:00
Alexander Motin
f573da1a0e There is no need to erase hook->hk_node before freing hook. 2008-03-29 22:53:58 +00:00
Alexander Motin
244586d6f1 Remove ng_setisr() call from ng_dequeue(). It is useless as we any way
will never exit ngintr(), while there is some ready requests on the queue.
It was made years ago with hope of parallel queue processing by several
net threads. But even if we have several threads sometimes, we have no
rights to process queue in parallel as it will break original requests
serialization that is critically important for some setups.
2008-03-27 23:02:30 +00:00
Alexander Motin
c86d865ec8 Switch from timeval to bintime, to use 1/(2^20) of seconds instead of
microseconds. It allows to use bit shifts instead of some heavy 64bit
mul/div math operations.
2008-03-27 20:04:20 +00:00
Alexander Motin
714f558be1 Some minor code and math optimizations. 2008-03-26 21:19:03 +00:00
Alexander Motin
489290e9e9 Rewrite node to support multiple hooks, alike to ng_l2tp, to use one pair
of pptpgre and ksocket nodes for all calls between two peers. This patch
modifies node's API by adding new "session_%04x" hook names support, while
keeping backward compatibility.

Together with appropriate user-level support (by latest mpd5) it gives
huge performance benefits for case of multiple active calls between
two peers because of avoiding data duplication and extra socket processing.
On my benchmarks I have got more then 10 times speedup for the 200
simultaneous PPTP calls between two peers.
In conclusion, it allows now to build effective "clients <=> PAC <=> PNS"
setups.
2008-03-24 22:55:22 +00:00
Alexander Motin
e81de8afb0 Remove impossible (hk_peer == NULL) check from ng_address_hook().
Valid hook can't have NULL peer. Even invalid one can't, as it is resets to
deadhook, but not NULL.
2008-03-16 23:12:17 +00:00
Alexander Motin
4e7597635f Add session ID hashing to speedup incoming packets dispatch in case
of many connections working via the same tunnel. For example, in case
of full "client <-> LAC <-> LNS" setup.
2008-03-16 21:33:12 +00:00
Alexander Motin
10e873189c Improve apply callback error reporting:
Before this patch callback returned result of the last finished call chain.
Now it returns last nonzero result from all call chain results in this request.

As soon as this improvement gives reliable error reporting, it is now possible
to remove dirty workaround in ng_socket, made to return ENOBUFS error statuses
of request-response operations. That workaround was responsible for returning
ENOBUFS errors to completely unrelated requests working at the same time
on socket.
2008-03-11 21:58:48 +00:00
Alexander Motin
395adfbe34 Addition to the previous commit. Release inproc in case of memory error. 2008-03-09 11:17:00 +00:00
Alexander Motin
af63939c67 To avoid control data losses do not acknowledge recieving of control packet
if netgraph reported error while delivering to destination.
Reset 'next send' counter to the last requested by peer on ack timeout
to resend all subsequest packets after lost one again without additional hints.
2008-03-08 23:55:29 +00:00
Alexander Motin
6e7ed93017 Send only one incoming notification at a time to reduce queue
trashing and improve performance.
Remove waitflag argument from ng_ksocket_incoming2(), it means nothing
as function call was queued by netgraph.
Remove node validity check, as node validity guarantied by netgraph.
Update comments.
2008-03-07 21:12:56 +00:00
Alexander Motin
ed75521f5b Increase default queue items allocation limit from 512 to 4096 items
to avoid terrible unpredicted effects for netgraph operation of their
exhaustion while allocating control messages.
Add separate configurable 512 items limit for data items allocation
for DoS/overload protection.

Discussed with:	julian
2008-03-05 22:12:34 +00:00
Alexander Motin
cfea3f8522 Implement 128 items node name hash for faster name search.
Increase node ID hash size from 32 to 128 items.
2008-03-04 18:22:18 +00:00
Alexander Motin
b7c649d811 Fix incorrect field name. 2008-03-04 11:10:54 +00:00
Alexander Motin
dda30f129e Use more compact LIST instead of TAILQ for session hash.
Add all listening hooks into LIST to simplify searches.
Use ng_findhook() instead of own equal implementation.
2008-03-03 19:36:03 +00:00
Alexander Motin
bd500dab7f Make session ID generator to use session ID hash.
Make session ID generator thread-safe.
2008-03-02 23:26:35 +00:00
Alexander Motin
fffba935e4 Add support for the libalias redirect functionality.
Submitted by:   Vadim Goncharov <vadim_nuclight@mail.ru>
2008-03-01 17:14:02 +00:00
Alexander Motin
db3408aed0 Fix incorrect constant used in rev. 1.146 that broke node writer locking. 2008-02-25 21:24:53 +00:00
Alexander Motin
510b772284 Fix shutdown bug made by previous commit. 2008-02-24 10:13:32 +00:00
Gleb Smirnoff
150c26cb34 Use rtalloc1() instead of rtalloc_ign(). It returns a locked
rtentry. We quickly copy the fields of interest, and then
RTFREE_LOCKED(). This should be faster then lock & unlock the
rtentry twice.
2008-02-07 11:10:17 +00:00
Alexander Motin
b4d0be220a Do not use bcmp() to compare two bytes with constants. 2008-02-06 20:37:34 +00:00
Alexander Motin
f50597f5f1 Cleanup and tune ng_snd_item() function as it is one of the
most busy netgraph functions.
Tune stack protection constants to avoid division operation.
2008-02-06 18:50:40 +00:00
Alexander Motin
193f57e2c0 Prepare hooks direct pointers on setup to avoid heavy ng_findhook() calls
during operarion.
2008-02-04 19:26:53 +00:00
Alexander Motin
e632000eed Move all possible node logic out of the rcvdata() function
to the newhook()/disconnect().
Unify function names with other nodes.
2008-02-03 18:55:45 +00:00
Alexander Motin
102fe25ee0 Revert previous commit.
glebius@ noticed that it was not a bug, but undocumented feature.
2008-02-03 10:30:45 +00:00
Dmitry Morozovsky
f9773372c3 Fix one more grammo.
Noticed by:	ru
2008-02-02 08:41:53 +00:00
Dmitry Morozovsky
942fe01f61 Reword recent comment a bit. 2008-02-01 17:35:46 +00:00
Alexander Motin
d4529f987a Add comments about stack protection mechanism. 2008-02-01 11:01:15 +00:00
Alexander Motin
b1a3358ba3 Tune the message for better informativity.
Print the hook pointer as other functions do.
2008-02-01 07:25:06 +00:00
Benno Rice
db2dabf87c Band-aid recent commit by mav by replacing a variable in a CTR statement with
the variable that appears as if it should've been there.

Pointy hat to:		mav
Not tested either by:	benno
2008-02-01 07:17:26 +00:00
Alexander Motin
b2b5279b25 Implement Session-ID hashing to improve receive performance scalability
for big number of concurrent sessions.
2008-01-31 22:42:37 +00:00
Alexander Motin
e72a98f4bf Some code reformat. 2008-01-31 10:13:04 +00:00
Alexander Motin
81a253a4ed Implement stack protection based on GET_STACK_USAGE() macro.
This fixes system panics possible with complicated netgraph setups
and allows to avoid unneded extra queueing for stack unwrapping.
2008-01-31 08:51:48 +00:00
Alexander Motin
f38b370376 Avoid data copying when it is possible.
bpf_filter() is able to work directly on mbuf chain.
2008-01-28 22:37:17 +00:00
Alexander Motin
57f2b25dfa Run expire even without export hook connected.
PR:	kern/119839
2008-01-27 15:01:16 +00:00
Alexander Motin
cfcb2a4c82 Fix memory leak when export hook is not connected. 2008-01-27 09:22:10 +00:00
Alexander Motin
f704a24b20 Remove one very strange unneded if. 2008-01-27 08:52:41 +00:00
Alexander Motin
f8e5127d49 Slightly simplify code. 2008-01-27 02:04:12 +00:00
Alexander Motin
5716b3f192 Improve multilink receive performance by netgraph item reuse. 2008-01-26 22:42:47 +00:00
Alexander Motin
f9ec69904b Improve multilink xmit performance by netgraph item reuse. 2008-01-26 22:41:14 +00:00
Alexander Motin
93caf2e299 Improve multilink receive performance with fragment headers preallocation. 2008-01-26 22:39:05 +00:00
Alexander Motin
9c10d11388 Fix bundle xmit octets stats for packet-split operation mode. 2008-01-23 11:47:09 +00:00
Jeff Roberson
397c19d175 Remove explicit locking of struct file.
- Introduce a finit() which is used to initailize the fields of struct file
   in such a way that the ops vector is only valid after the data, type,
   and flags are valid.
 - Protect f_flag and f_count with atomic operations.
 - Remove the global list of all files and associated accounting.
 - Rewrite the unp garbage collection such that it no longer requires
   the global list of all files and instead uses a list of all unp sockets.
 - Mark sockets in the accept queue so we don't incorrectly gc them.

Tested by:	kris, pho
2007-12-30 01:42:15 +00:00
Alexander Motin
098ff746f8 Add support for optional "AC-Name\Service-Name" syntax at NGM_PPPOE_CONNECT
argument. It allows ppp, mpd or any other node consumer to request
connection to specified access concentrator.

Proposed by:	Alexander A. Burylov <burylov@mail.ru>
2007-12-29 19:44:41 +00:00
Alexander Motin
97b4f83bb3 Fix incorrectly placed bracket in pppoe_find_svc(). 2007-12-26 19:33:53 +00:00
Alexander Motin
b06ccdeab3 Remove some prehistoric never used defines. 2007-12-26 19:15:07 +00:00
Robert Watson
3de213cc00 Add a new 'why' argument to kdb_enter(), and a set of constants to use
for that argument.  This will allow DDB to detect the broad category of
reason why the debugger has been entered, which it can use for the
purposes of deciding which DDB script to run.

Assign approximate why values to all current consumers of the
kdb_enter() interface.
2007-12-25 17:52:02 +00:00
Alexander Motin
673f5a8b44 Add option to set packets per second limits instead of default
bits per second ones.
2007-12-19 22:50:14 +00:00
Alexander Motin
52b9b77f78 Increase control channel xmit queue to 128 packets.
Previous value 16 was too small for real LAC as temporal activity
spike cound easily overflow queue demanding tunnel disconnection due
to possible state inconsistency.
2007-12-12 19:04:30 +00:00
Julian Elischer
182ff780b4 Add ipv6 to ng_cisco node. ipv6 wasn't a reality when I wrote it..
Submitted by: Marko Zec
2007-11-30 23:27:39 +00:00
Gleb Smirnoff
b332b91f74 - Merge all the ng_send_fn2* functions into one - ng_send_fn2(),
removing some copy&pasted code.
- Reduce copy and paste in ng_apply_item().
- Resurrect ng_send_fn() as a valid symbol, not a define.

Reviewed by:	mav, julian
2007-11-14 11:25:58 +00:00
Maksim Yevmenkin
b64b9bb278 Fix locking issue in ng_btsocket_l2cap_ctloutput()
Submitted by:	Heiko Wundram (Beenic) < wundram at beenic dot net >
MFC after:	3 days
2007-10-31 16:17:20 +00:00
Maksim Yevmenkin
a6f3c1e3f3 Allow RFCOMM servers to bind to a ''wildcard'' RFCOMM channel
zero (0). Actual RFCOMM channel will be assigned after listen(2)
call is done on a RFCOMM socket bound to a ''wildcard'' RFCOMM
channel zero (0).

Address locking issues in ng_btsocket_rfcomm_bind()

Submitted by:	Heiko Wundram (Beenic) < wundram at beenic dot net >
MFC after:	1 week
2007-10-29 19:06:47 +00:00
Alexander Motin
eb4687d223 Minor debug message fix. 2007-10-28 18:05:59 +00:00
Ruslan Ermilov
857304e6f1 Fix build with NETGRAPH_DEBUG. 2007-10-19 20:09:58 +00:00
Alexander Motin
e088dd4c44 Implement new apply callback mechanism to handle item forwarding.
When item forwarded refence counter is incremented, when item
processed, counter decremented. When counter reaches zero,
apply handler is getting called.
Now it allows to report right connect() call status from user-level
at the right time.
2007-10-19 15:04:17 +00:00
Alexander Motin
98e7b7536b Split ng_pppoe_rcvdata() function into three hook-specific ones
to simplify code and reduce stack usage.
2007-10-14 09:58:22 +00:00
Alexander Motin
8cfaad5f6a Remove ng_pppoe_sendpacket() function to simplify code as it is called
as much times as it has cases inside of it.
2007-10-14 09:51:19 +00:00
Alexander Motin
702f98951d Protect struct seq with mutex.
Approved by:	glebius (mentor)
2007-10-12 04:56:26 +00:00
Alexander Motin
058ef12167 Remove one unneded assertion. It is also checked in
ng_l2tp_seq_check().

Approved by:	glebius (mentor)
2007-10-12 04:54:43 +00:00
Alexander Motin
bf741e4d08 Replace single rcvdata with 3 distinct to simplify code and
reduce stack usage.

Approved by:	mentor (glebius)
2007-10-12 04:53:23 +00:00
Alexander Motin
40097c5d9f Remove duplicate variables. 2007-10-12 04:51:30 +00:00
Alexander Motin
661e502900 Dead code removal.
Approved by:	re (kensmith), glebius (mentor)
2007-09-21 08:25:55 +00:00
Alexander Motin
99f4de905c This is optimization of ether and debug hooks determination. It
simplifies code and should speedup pppoe_findsession() function which is
called for every incoming packet.

Approved by:	re (kensmith), glebius (mentor)
2007-09-21 08:24:08 +00:00
Alexander Motin
1e7d84b055 This patch fixes thread unsafe usage of global pkt_hdr
variable. Second part is not so important, but IMO is also good.

Approved by:	re (kensmith), glebius (mentor)
2007-09-21 08:16:33 +00:00
Alexander Motin
6d67c50b39 Fix typo which brokes VJ decompression
when VJC negotiated in only one direction.

Approved by:	re (bmah), glebius (mentor)
2007-09-15 16:55:44 +00:00
Maksim Yevmenkin
d46210e60d Return EADDRNOTAVAIL instead of EDESTADDRREQ error when
listen(2) is called on improperly bound socket.

Suggested by:	Iain Hibbert
Approved by:	re (kensmith)
MFC after:	3 days
2007-08-23 16:55:22 +00:00
Alexander Motin
3fb87c2411 Add ng_send_fn() error handeling inside ng_con_nodes().
Without it some errors may left unnoticed and unhandeled
that will lead to hooks left in half-connected state.

Reviewed by:	julian@
Approved by:	re (kensmith), glebius (mentor)
2007-08-18 11:59:17 +00:00
Maksim Yevmenkin
51713b2a7b Make ng_h4(4) MPSAFE. Use similar to ng_tty(4) locking strategy.
Reconnect ng_h(4) back to the build.

Reviewed by:	kensmith
Approved by:	re (kensmith)
MFC after:	1 month
2007-08-13 17:19:28 +00:00
Robert Watson
0bf686c125 Remove the now-unused NET_{LOCK,UNLOCK,ASSERT}_GIANT() macros, which
previously conditionally acquired Giant based on debug.mpsafenet.  As that
has now been removed, they are no longer required.  Removing them
significantly simplifies error-handling in the socket layer, eliminated
quite a bit of unwinding of locking in error cases.

While here clean up the now unneeded opt_net.h, which previously was used
for the NET_WITH_GIANT kernel option.  Clean up some related gotos for
consistency.

Reviewed by:	bz, csjp
Tested by:	kris
Approved by:	re (kensmith)
2007-08-06 14:26:03 +00:00
Alexander Motin
d6fe462ac1 Add 64bit statistic counters to the ng_ppp node.
64bit counters are needed to simplify traffic accounting and
reduce system load at the big PPP concentrators.

Approved by:	re (rwatson), glebius (mentor)
2007-08-01 20:49:35 +00:00
Alexander Motin
e89c150775 This patch improves fine-grained locking for the ng_ppp node.
Till now node's transmit path was completely unprotected
and so wasn't thread safe in multilink mode. It's receive path was
declared as WRITER as the simpliest protection method but it
reduces performance when compression or encryption enabled.

Approved by:	re (rwatson), glebius (mentor)
2007-08-01 20:38:37 +00:00
Robert Watson
c6b2899785 Replace references to NET_CALLOUT_MPSAFE with CALLOUT_MPSAFE, and remove
definition of NET_CALLOUT_MPSAFE, which is no longer required now that
debug.mpsafenet has been removed.

The once over:	bz
Approved by:	re (kensmith)
2007-07-28 07:31:30 +00:00
Alexander Motin
091193febe Reduce stack usage by 256 bytes per call. It helps to avoid kernel
stack overflow in complicated traffic filtering setups.

There can be minor performance degradation for the MHLEN < len <= 256 case
due to additional buffer allocation, but it is a rare case.

Approved by:	re (rwatson), glebius (mentor)
MFC after:	1 week
2007-07-26 18:15:02 +00:00
Gleb Smirnoff
bb5ba44f82 Honor the IFF_MONITOR flag.
PR:		kern/99500
Submitted by:	Craig Leres <leres ee.lbl.gov>
Approved by:	re (kensmith)
2007-07-26 10:54:33 +00:00
Maksim Yevmenkin
08b755600f Mark ng_h4(4) as not MPSAFE and disconnect it from the build for now.
Approved by:	re (rwatson)
2007-07-10 16:38:43 +00:00
Warner Losh
02fb1cf4b9 These modules depend on usb, make that explicit
Approved by: re@
2007-06-23 04:34:38 +00:00
Matt Jacob
2a9a64c6a2 Fix various compilation warnings for gcc-4.2.
Approved by:	re (bruce)
2007-06-23 00:02:20 +00:00
Maksim Yevmenkin
280c458af8 Replace sosend() with direct call to .pru_send method on the
L2CAP socket. This is to avoid LOR with sx(9) lock in sblock()
called from sosend_generic().

Approved by:	re (kensmith)
MFC after:	1 week
2007-06-21 19:55:49 +00:00
Xin LI
8dac5046be Fix build problem caused by a set of typos.
Reported by:	tinderbox
Approved by:	re (mux)
2007-06-19 14:56:35 +00:00
Warner Losh
38b00ba0cd Finish removing usb_port.h compat macros. 2007-06-18 22:23:20 +00:00
Ruslan Ermilov
77764a595a Remove two more instances of the USBDEV() macro. 2007-06-13 12:36:01 +00:00
Alexander Motin
df01e68922 Add missing ng_uncallout() on node shutdown.
Approved by:	glebius (mentor)
2007-06-13 11:01:17 +00:00
Maksim Yevmenkin
8004e6ecc8 Catch up with USB cleanups and fix the world 2007-06-13 00:32:00 +00:00
Warner Losh
4f9929e80f Eliminate usb_thread_t. 2007-06-12 17:30:54 +00:00
Warner Losh
645016c0e4 Expand USB_ATTACH_{ERROR,SUCCESS}_RETURN inline and eliminate from
usb_port.h.  They aren't needed, and are a legacy of this code's past.
2007-06-12 15:37:19 +00:00
Warner Losh
c7b8e2f50c Silence a gcc warning in a more canonical way (evl = NULL rather than &evl).
I saw warnings here at one point on the arm build.
2007-06-11 15:29:02 +00:00
Warner Losh
387ecc9396 Expand USB_ATTACH_SETUP inline.
Kill devinfo stuff.
2007-06-09 06:53:27 +00:00
David Malone
041b706b2f Despite several examples in the kernel, the third argument of
sysctl_handle_int is not sizeof the int type you want to export.
The type must always be an int or an unsigned int.

Remove the instances where a sizeof(variable) is passed to stop
people accidently cut and pasting these examples.

In a few places this was sysctl_handle_int was being used on 64 bit
types, which would truncate the value to be exported.  In these
cases use sysctl_handle_quad to export them and change the format
to Q so that sysctl(1) can still print them.
2007-06-04 18:25:08 +00:00
Alexander Motin
c35e19c430 No need to update link queue stats when round-robin algorithm enabled.
Approved by:	glebius (mentor)
2007-06-04 13:50:09 +00:00
Gleb Smirnoff
2775748750 Partially back out rev. 1.127, to restore broken functionality. This
should be redesigned, but better enter RELENG_7 with a working ngctl(8).

Agreed by:	julian
2007-06-01 09:20:57 +00:00
Robert Watson
e1e8f51b85 Universally adopt most conventional spelling of acquire. 2007-05-27 20:50:23 +00:00
Alexander Motin
e842c54054 Add support for setmode and settarget messages.
Approved by:	glebius (mentor)
2007-05-22 12:23:39 +00:00
Alexander Motin
179f56e196 Allow node to bypass traffic while no alias address defined.
Approved by:	glebius (mentor)
2007-05-22 12:14:43 +00:00