Commit Graph

16575 Commits

Author SHA1 Message Date
Brooks Davis
bdf80fecf0 Document the fact that tunables can be set in device.hints.
Reviewed by:	wblock
MFC after:	3 days
Differential Revision:	https://reviews.freebsd.org/D3153
2015-07-23 17:27:10 +00:00
Marcel Moolenaar
be00e09818 Check the hw.proto.attach environment variable for devices that
proto(4) should attach to instead of the normal driver.

Document the variable.
2015-07-19 23:37:45 +00:00
Edward Tomasz Napierala
bd425507b7 Expand sysctl descriptions in iscsi(4) and ctl(4).
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2015-07-18 15:27:12 +00:00
Ed Schouten
50f960e60e Fix a small typo: "the the".
Spotted by:	wblock
2015-07-16 15:43:55 +00:00
Ed Schouten
707d98fe2f Implement the CloudABI random_get() system call.
The random_get() system call works similar to getentropy()/getrandom()
on OpenBSD/Linux. It fills a buffer with random data.

This change introduces a new function, read_random_uio(), that is used
to implement read() on the random devices. We can call into this
function from within the CloudABI compatibility layer.

Approved by:	secteam
Reviewed by:	jmg, markm, wblock
Obtained from:	https://github.com/NuxiNL/freebsd
Differential Revision:	https://reviews.freebsd.org/D3053
2015-07-14 18:45:15 +00:00
Christian Brueffer
9870187435 Markup fixes. 2015-07-13 15:26:03 +00:00
Christian Brueffer
647cdd025a Fix a typo and duplicate word. 2015-07-13 14:25:15 +00:00
Mark Murray
3aa77530ca * Address review (and add a bit myself).
- Tweek man page.
 - Remove all mention of RANDOM_FORTUNA. If the system owner wants YARROW or DUMMY, they ask for it, otherwise they get FORTUNA.
 - Tidy up headers a bit.
 - Tidy up declarations a bit.
 - Make static in a couple of places where needed.
 - Move Yarrow/Fortuna SYSINIT/SYSUNINIT to randomdev.c, moving us towards a single file where the algorithm context is used.
 - Get rid of random_*_process_buffer() functions. They were only used in one place each, and are better subsumed into those places.
 - Remove *_post_read() functions as they are stubs everywhere.
 - Assert against buffer size illegalities.
 - Clean up some silly code in the randomdev_read() routine.
 - Make the harvesting more consistent.
 - Make some requested argument name changes.
 - Tidy up and clarify a few comments.
 - Make some requested comment changes.
 - Make some requested macro changes.

* NOTE: the thing calling itself a 'unit test' is not yet a proper
  unit test, but it helps me ensure things work. It may be a proper
  unit test at some time in the future, but for now please don't make
  any assumptions or hold any expectations.

Differential Revision:	https://reviews.freebsd.org/D2025
Approved by:	so (/dev/random blanket)
2015-07-12 18:14:38 +00:00
Adrian Chadd
6520495abc Add an initial NUMA affinity/policy configuration for threads and processes.
This is based on work done by jeff@ and jhb@, as well as the numa.diff
patch that has been circulating when someone asks for first-touch NUMA
on -10 or -11.

* Introduce a simple set of VM policy and iterator types.
* tie the policy types into the vm_phys path for now, mirroring how
  the initial first-touch allocation work was enabled.
* add syscalls to control changing thread and process defaults.
* add a global NUMA VM domain policy.
* implement a simple cascade policy order - if a thread policy exists, use it;
  if a process policy exists, use it; use the default policy.
* processes inherit policies from their parent processes, threads inherit
  policies from their parent threads.
* add a simple tool (numactl) to query and modify default thread/process
  policities.
* add documentation for the new syscalls, for numa and for numactl.
* re-enable first touch NUMA again by default, as now policies can be
  set in a variety of methods.

This is only relevant for very specific workloads.

This doesn't pretend to be a final NUMA solution.

The previous defaults in -HEAD (with MAXMEMDOM set) can be achieved by
'sysctl vm.default_policy=rr'.

This is only relevant if MAXMEMDOM is set to something other than 1.
Ie, if you're using GENERIC or a modified kernel with non-NUMA, then
this is a glorified no-op for you.

Thank you to Norse Corp for giving me access to rather large
(for FreeBSD!) NUMA machines in order to develop and verify this.

Thank you to Dell for providing me with dual socket sandybridge
and westmere v3 hardware to do NUMA development with.

Thank you to Scott Long at Netflix for providing me with access
to the two-socket, four-domain haswell v3 hardware.

Thank you to Peter Holm for running the stress testing suite
against the NUMA branch during various stages of development!

Tested:

* MIPS (regression testing; non-NUMA)
* i386 (regression testing; non-NUMA GENERIC)
* amd64 (regression testing; non-NUMA GENERIC)
* westmere, 2 socket (thankyou norse!)
* sandy bridge, 2 socket (thankyou dell!)
* ivy bridge, 2 socket (thankyou norse!)
* westmere-EX, 4 socket / 1TB RAM (thankyou norse!)
* haswell, 2 socket (thankyou norse!)
* haswell v3, 2 socket (thankyou dell)
* haswell v3, 2x18 core (thankyou scott long / netflix!)

* Peter Holm ran a stress test suite on this work and found one
  issue, but has not been able to verify it (it doesn't look NUMA
  related, and he only saw it once over many testing runs.)

* I've tested bhyve instances running in fixed NUMA domains and cpusets;
  all seems to work correctly.

Verified:

* intel-pcm - pcm-numa.x and pcm-memory.x, whilst selecting different
  NUMA policies for processes under test.

Review:

This was reviewed through phabricator (https://reviews.freebsd.org/D2559)
as well as privately and via emails to freebsd-arch@.  The git history
with specific attributes is available at https://github.com/erikarn/freebsd/
in the NUMA branch (https://github.com/erikarn/freebsd/compare/local/adrian_numa_policy).

This has been reviewed by a number of people (stas, rpaulo, kib, ngie,
wblock) but not achieved a clear consensus.  My hope is that with further
exposure and testing more functionality can be implemented and evaluated.

Notes:

* The VM doesn't handle unbalanced domains very well, and if you have an overly
  unbalanced memory setup whilst under high memory pressure, VM page allocation
  may fail leading to a kernel panic.  This was a problem in the past, but it's
  much more easily triggered now with these tools.

* This work only controls the path through vm_phys; it doesn't yet strongly/predictably
  affect contigmalloc, KVA placement, UMA, etc.  So, driver placement of memory
  isn't really guaranteed in any way.  That's next on my plate.

Sponsored by:	Norse Corp, Inc.; Dell
2015-07-11 15:21:37 +00:00
John-Mark Gurney
f405d8eb61 some additional improvements to the documentation...
Sponsored by:	Netflix, Inc.
2015-07-11 04:20:56 +00:00
John-Mark Gurney
94b591186d yet more documentation improvements... Many changes were made to the
OCF w/o documentation...

Document the new (8+ year old) device_t way of handling things, that
_unregister_all will leave no threads in newsession, the _SYNC flag,
the requirement that a flag be specified...

Other minor changes like breaking up a wall of text into paragraphs...
2015-07-08 22:46:45 +00:00
Baptiste Daroussin
f9b9c07087 Fix typo which breaks build of manpages when WITHOUT_MANCOMPRESS is set
PR:		201153
Reported by:	Andriy Voskoboinyk <s3erios@gmail.com>
2015-07-08 22:24:55 +00:00
Patrick Kelsey
fe3ff217dd Replace use of .Po Pc with the preferred .Pq for single line
enclosures in iovctl.conf(5), iovctl(8), pci(9), and
pci_iov_schema(9).

Differential Revision: https://reviews.freebsd.org/D3000
Reviewed by: wblock
Approved by: jmallett (mentor)
2015-07-08 16:16:44 +00:00
Conrad Meyer
80f295a415 Add myself to committers-src.dot
Approved by:	markj (mentor)
2015-07-08 03:20:28 +00:00
Warner Losh
abb14e5ded The results of the vote are in. This reflects that vote. Single
line statements inside of braces is recognized as an acceptable
style.
	http://reviews.freebsd.org/V3
As always, this isn't license for wholesale change, etc.
2015-07-06 20:10:47 +00:00
Mark Johnston
363f89481b Remove a BUGS entry that was addressed by r282300. 2015-07-05 23:24:52 +00:00
Mark Johnston
4b4ad3a219 Rename the dtrace-* man pages to dtrace_* for consistency with other
subsection man pages (e.g. geom_*, mac_*, snd_*).
2015-07-05 23:23:12 +00:00
Konstantin Belousov
e6e979f5d0 Document the locking context for the directly dispatched callouts.
Cross-reference timeout(9).

Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2015-07-05 19:29:24 +00:00
Mariusz Zaborski
58c86148dd Move nvlist documentation to the FreeBSD Kernel Developer's sections.
Approved by:	pjd (mentor)
2015-07-04 10:27:30 +00:00
Julio Merino
30161d035c Add support for TEST_METADATA
Allow Makefiles to define generic metadata settings that apply to all test
programs defined by a Makefile.  The generic TEST_METADATA variable extends
the per-test program settings already supported via TEST_METADATA.<program>.

This feature will be useful to easily apply some settings to all programs
in a directory.  In particular, Kyua 0.12 will support parallel execution
of test programs and a bunch of them will need to be tagged as is_exclusive
to indicate that they cannot be run in parallel with anything else due to
their side-effects.  It will be reasonable to set this setting on whole
directories.

MFC after:	1 week
2015-07-04 02:37:36 +00:00
Julio Merino
18bafc4da9 Add missing shebang
Plain test programs are not preprocessed by the build system (as opposed to
ATF test cases, which automatically gain a shebang pointing at atf-sh), so
we must take care of providing the shebang ourselves.

I'm not sure why this was not causing problems with Kyua 0.11, but the
upcoming 0.12 release chokes on this particular issue.

MFC after:	1 week
2015-07-04 02:24:47 +00:00
Edward Tomasz Napierala
a2c782d103 Make ctl(4) appear in "man -k iscsi" results.
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
2015-07-03 16:55:08 +00:00
Marcel Moolenaar
15abbe5137 Minor update to the proto(4) man page:
1.  We now support ISA devices
2.  DMA support has been added
2015-07-03 16:20:14 +00:00
John-Mark Gurney
e08d13cf83 more word smithing wrt the crd_inject field...
We've already defined IV earlier, so no need to expand it a second
time here...
2015-07-03 01:55:06 +00:00
John-Mark Gurney
2ca5eb5d2d update the documentation of the _IV_ flags... _IV_PRESENT doesn't
mean what you think it should...  This will be fixed in the future
with a flag rename, but document what the flag really does and make
the _IV_ flags clear what their presents (or lack there of) means...

Reviewed by:	gnn, eri (both earlier version)
2015-07-03 00:37:16 +00:00
Andrew Turner
5607ec2379 Set MACHINE_CPU to arm64 when building for arm64. This is needed by the
ports tree as they check this value in a number of ports.

PR:		201259
Sponsored by:	ABT Systems Ltd
2015-07-02 13:42:57 +00:00
John-Mark Gurney
0812640475 add units to the length and count so that it's clear what they
are measured in...  Gems like: "len is the length of the buffer."
aren't useful in man pages.
2015-06-30 19:06:14 +00:00
Hiren Panchasara
5341316696 Add missing DTrace scripts that exist in the source tree to the Makefile for
install.
While here, sort the list.

Differential Revision:	D2950
Submitted by:		Jason Wolfe <j at nitrology.com>
Reviewed by:		gnn, markj, hiren
Sponsored by:		Limelight Networks
2015-06-30 18:23:08 +00:00
Mark Murray
c4f9c760c9 Updated random(4) boot/shutdown scripting.
Fix the man pages as well.

Differential Revision: https://reviews.freebsd.org/D2924
Approved by: so (delphij)
2015-06-30 17:09:41 +00:00
Mark Murray
d1b06863fb Huge cleanup of random(4) code.
* GENERAL
- Update copyright.
- Make kernel options for RANDOM_YARROW and RANDOM_DUMMY. Set
  neither to ON, which means we want Fortuna
- If there is no 'device random' in the kernel, there will be NO
  random(4) device in the kernel, and the KERN_ARND sysctl will
  return nothing. With RANDOM_DUMMY there will be a random(4) that
  always blocks.
- Repair kern.arandom (KERN_ARND sysctl). The old version went
  through arc4random(9) and was a bit weird.
- Adjust arc4random stirring a bit - the existing code looks a little
  suspect.
- Fix the nasty pre- and post-read overloading by providing explictit
  functions to do these tasks.
- Redo read_random(9) so as to duplicate random(4)'s read internals.
  This makes it a first-class citizen rather than a hack.
- Move stuff out of locked regions when it does not need to be
  there.
- Trim RANDOM_DEBUG printfs. Some are excess to requirement, some
  behind boot verbose.
- Use SYSINIT to sequence the startup.
- Fix init/deinit sysctl stuff.
- Make relevant sysctls also tunables.
- Add different harvesting "styles" to allow for different requirements
  (direct, queue, fast).
- Add harvesting of FFS atime events. This needs to be checked for
  weighing down the FS code.
- Add harvesting of slab allocator events. This needs to be checked for
  weighing down the allocator code.
- Fix the random(9) manpage.
- Loadable modules are not present for now. These will be re-engineered
  when the dust settles.
- Use macros for locks.
- Fix comments.

* src/share/man/...
- Update the man pages.

* src/etc/...
- The startup/shutdown work is done in D2924.

* src/UPDATING
- Add UPDATING announcement.

* src/sys/dev/random/build.sh
- Add copyright.
- Add libz for unit tests.

* src/sys/dev/random/dummy.c
- Remove; no longer needed. Functionality incorporated into randomdev.*.

* live_entropy_sources.c live_entropy_sources.h
- Remove; content moved.
- move content to randomdev.[ch] and optimise.

* src/sys/dev/random/random_adaptors.c src/sys/dev/random/random_adaptors.h
- Remove; plugability is no longer used. Compile-time algorithm
  selection is the way to go.

* src/sys/dev/random/random_harvestq.c src/sys/dev/random/random_harvestq.h
- Add early (re)boot-time randomness caching.

* src/sys/dev/random/randomdev_soft.c src/sys/dev/random/randomdev_soft.h
- Remove; no longer needed.

* src/sys/dev/random/uint128.h
- Provide a fake uint128_t; if a real one ever arrived, we can use
  that instead. All that is needed here is N=0, N++, N==0, and some
  localised trickery is used to manufacture a 128-bit 0ULLL.

* src/sys/dev/random/unit_test.c src/sys/dev/random/unit_test.h
- Improve unit tests; previously the testing human needed clairvoyance;
  now the test will do a basic check of compressibility. Clairvoyant
  talent is still a good idea.
- This is still a long way off a proper unit test.

* src/sys/dev/random/fortuna.c src/sys/dev/random/fortuna.h
- Improve messy union to just uint128_t.
- Remove unneeded 'static struct fortuna_start_cache'.
- Tighten up up arithmetic.
- Provide a method to allow eternal junk to be introduced; harden
  it against blatant by compress/hashing.
- Assert that locks are held correctly.
- Fix the nasty pre- and post-read overloading by providing explictit
  functions to do these tasks.
- Turn into self-sufficient module (no longer requires randomdev_soft.[ch])

* src/sys/dev/random/yarrow.c src/sys/dev/random/yarrow.h
- Improve messy union to just uint128_t.
- Remove unneeded 'staic struct start_cache'.
- Tighten up up arithmetic.
- Provide a method to allow eternal junk to be introduced; harden
  it against blatant by compress/hashing.
- Assert that locks are held correctly.
- Fix the nasty pre- and post-read overloading by providing explictit
  functions to do these tasks.
- Turn into self-sufficient module (no longer requires randomdev_soft.[ch])
- Fix some magic numbers elsewhere used as FAST and SLOW.

Differential Revision: https://reviews.freebsd.org/D2025
Reviewed by: vsevolod,delphij,rwatson,trasz,jmg
Approved by: so (delphij)
2015-06-30 17:00:45 +00:00
Maxim Konovalov
2a52334e0b o DragonFly 4.2.0 added. 2015-06-30 07:46:06 +00:00
Sean Bruno
8971c13ffa Delete the refernce to VLAN handling being disabled by default. This is
no longer the case.

PR:		118693
MFC after:	3 days
2015-06-29 17:59:00 +00:00
Christian Brueffer
d14a50d95a Fix issues that kept some of the examples from actually working.
PR:		144534
Submitted by:	Nicolas Edel
MFC after:	1 week
2015-06-29 14:41:19 +00:00
Hans Petter Selasky
38b622e199 Make the system queue header file fully usable within C++ programs by
adding macros to define class lists.

This change is backwards compatible for all use within C and C++
programs. Only C++ programs will have added support to use the queue
macros within classes. Previously the queue macros could only be used
within structures.

The queue.3 manual page has been updated to describe the new
functionality and some alphabetic sorting has been done while
at it.

Differential Revision:	https://reviews.freebsd.org/D2745
PR:			200827 (exp-run)
MFC after:		2 weeks
2015-06-28 21:06:45 +00:00
Matthew Seaman
4311777e58 Steve KArgl's commit bit taken in at his own request
With hat:     core-secretary
Approved by:	core
2015-06-28 09:03:26 +00:00
Baptiste Daroussin
af38028d11 Make all shared library a relative symlink
This makes sysroot usable for cross building, it also removes the need for
_SHLIBDIRPREFIX (keeps its definition since picobsd uses it and I have no time
to test it)

Differential Revision:	https://reviews.freebsd.org/D2920
Submitted by:	imp, adrian
Tested by:	adrian
2015-06-27 23:28:56 +00:00
Andrew Turner
61b9bd2303 Sort the cpu architectures by name rather than a combination of
alphabetical order and appending new architectures to the end of the list.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
2015-06-25 16:47:11 +00:00
Ermal Luçi
a5b789f65a ALTQ FAIRQ discipline import from DragonFLY
Differential Revision:  https://reviews.freebsd.org/D2847
Reviewed by:    glebius, wblock(manpage)
Approved by:    gnn(mentor)
Obtained from:  pfSense
Sponsored by:   Netgate
2015-06-24 19:16:41 +00:00
Kevin Lo
2907b9f461 Mention that using ports/net/malo-firmware-kmod to install the firmware. 2015-06-24 09:28:43 +00:00
Simon J. Gerraty
cc2520d2f3 Fix generation of src.conf.5
Since makeman turns all options on, we need to guard somethings from
make(showconfig)
2015-06-22 20:21:57 +00:00
Simon J. Gerraty
7090067b2a Appy := to dependent opts once value determined.
This is needed to fix output from 'make showconfig'
2015-06-22 19:01:09 +00:00
Baptiste Daroussin
a82117a6af Simplify code to install manpages and it symlinks
Use the bmake's .for multi variable loops to simplify and make the code more
readable
2015-06-21 22:42:55 +00:00
Baptiste Daroussin
5428abe13d Simplify symlinks on include files by using bmake's multivariable .for loops 2015-06-21 22:27:52 +00:00
Edward Tomasz Napierala
c6ba08803a Expand sysctls descriptions for iscsi(4) and ctl(4).
Differential Revision:	https://reviews.freebsd.org/D2876
Reviewed by:	wblock@
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
2015-06-21 14:21:38 +00:00
Simon J. Gerraty
916c1cd0ab Deal with MAKEOBJDIRPREFIX
MAKEOBJDIRPREFIX does not really fit our model.
Use it to set OBJROOT to about where user expects
and set MAKEOBJDIR to do what we want.
2015-06-20 21:48:07 +00:00
Konstantin Belousov
7705435606 The barriers, provided by _acq and _rel atomics, are acquire and
release barriers, not read and write barriers.  They fence all memory
accesses from the respective side, not limited by the kind of
operation.

Reviewed by:	jhb
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2015-06-20 17:18:46 +00:00
Baptiste Daroussin
729c67f5a7 Use .CURDIR in soelim's include path instead of SRCDIR
This fixes build when a user has defined SRCDIR in make.conf as required by
some ports
2015-06-20 00:17:57 +00:00
Baptiste Daroussin
d586a8e4ae Readd merge.awk forgotten while importing the old vi documentation directly into
the tree.

Do not rely on SRCDIR to find the merge.awk script but use .CURDIR
Fix a long standing bug making the generated documents never including the index
2015-06-20 00:13:28 +00:00
Xin LI
10b942e5d3 Fix markups and change e.g./eg. to e.g.,.
MFC after:	2 weeks
2015-06-19 21:35:56 +00:00
Xin LI
908e4e9754 Fix markups.
MFC after:	2 weeks
2015-06-19 21:35:24 +00:00