Commit Graph

413 Commits

Author SHA1 Message Date
Adrian Chadd
6520495abc Add an initial NUMA affinity/policy configuration for threads and processes.
This is based on work done by jeff@ and jhb@, as well as the numa.diff
patch that has been circulating when someone asks for first-touch NUMA
on -10 or -11.

* Introduce a simple set of VM policy and iterator types.
* tie the policy types into the vm_phys path for now, mirroring how
  the initial first-touch allocation work was enabled.
* add syscalls to control changing thread and process defaults.
* add a global NUMA VM domain policy.
* implement a simple cascade policy order - if a thread policy exists, use it;
  if a process policy exists, use it; use the default policy.
* processes inherit policies from their parent processes, threads inherit
  policies from their parent threads.
* add a simple tool (numactl) to query and modify default thread/process
  policities.
* add documentation for the new syscalls, for numa and for numactl.
* re-enable first touch NUMA again by default, as now policies can be
  set in a variety of methods.

This is only relevant for very specific workloads.

This doesn't pretend to be a final NUMA solution.

The previous defaults in -HEAD (with MAXMEMDOM set) can be achieved by
'sysctl vm.default_policy=rr'.

This is only relevant if MAXMEMDOM is set to something other than 1.
Ie, if you're using GENERIC or a modified kernel with non-NUMA, then
this is a glorified no-op for you.

Thank you to Norse Corp for giving me access to rather large
(for FreeBSD!) NUMA machines in order to develop and verify this.

Thank you to Dell for providing me with dual socket sandybridge
and westmere v3 hardware to do NUMA development with.

Thank you to Scott Long at Netflix for providing me with access
to the two-socket, four-domain haswell v3 hardware.

Thank you to Peter Holm for running the stress testing suite
against the NUMA branch during various stages of development!

Tested:

* MIPS (regression testing; non-NUMA)
* i386 (regression testing; non-NUMA GENERIC)
* amd64 (regression testing; non-NUMA GENERIC)
* westmere, 2 socket (thankyou norse!)
* sandy bridge, 2 socket (thankyou dell!)
* ivy bridge, 2 socket (thankyou norse!)
* westmere-EX, 4 socket / 1TB RAM (thankyou norse!)
* haswell, 2 socket (thankyou norse!)
* haswell v3, 2 socket (thankyou dell)
* haswell v3, 2x18 core (thankyou scott long / netflix!)

* Peter Holm ran a stress test suite on this work and found one
  issue, but has not been able to verify it (it doesn't look NUMA
  related, and he only saw it once over many testing runs.)

* I've tested bhyve instances running in fixed NUMA domains and cpusets;
  all seems to work correctly.

Verified:

* intel-pcm - pcm-numa.x and pcm-memory.x, whilst selecting different
  NUMA policies for processes under test.

Review:

This was reviewed through phabricator (https://reviews.freebsd.org/D2559)
as well as privately and via emails to freebsd-arch@.  The git history
with specific attributes is available at https://github.com/erikarn/freebsd/
in the NUMA branch (https://github.com/erikarn/freebsd/compare/local/adrian_numa_policy).

This has been reviewed by a number of people (stas, rpaulo, kib, ngie,
wblock) but not achieved a clear consensus.  My hope is that with further
exposure and testing more functionality can be implemented and evaluated.

Notes:

* The VM doesn't handle unbalanced domains very well, and if you have an overly
  unbalanced memory setup whilst under high memory pressure, VM page allocation
  may fail leading to a kernel panic.  This was a problem in the past, but it's
  much more easily triggered now with these tools.

* This work only controls the path through vm_phys; it doesn't yet strongly/predictably
  affect contigmalloc, KVA placement, UMA, etc.  So, driver placement of memory
  isn't really guaranteed in any way.  That's next on my plate.

Sponsored by:	Norse Corp, Inc.; Dell
2015-07-11 15:21:37 +00:00
Warner Losh
32c88b5252 Remove old fmake. It wasn't built by default for some time. Users that
really need it can find it in the devel/fmake port or pkg install fmake.
Note: This commit is orthogonal to the question 'can we fmake buildworld'.

Differential Revision: https://reviews.freebsd.org/D2840
2015-06-16 20:58:33 +00:00
Baptiste Daroussin
8faa40d1a0 Switch to mandoc's version of makewhatis(8), whatis(1), apropos(1) utilities.
This change among other things improve search capabilities over the manpages
allowing fine grain query.

A new build option WITHOUT_MANDOCDB has been added to keep the ancient version
of the database and the tools. The plan is to entirely remove this option before
11.0-RELEASE.

Differential Revision:	https://reviews.freebsd.org/D2603
2015-05-30 17:41:37 +00:00
Baptiste Daroussin
7b0896d882 Remove demandoc(1) from the build 2015-05-21 13:02:43 +00:00
Baptiste Daroussin
363da13804 Replace groff's soelim by soeliminate(1) renamed soelim(1) 2015-05-01 20:08:25 +00:00
Ed Maste
0a06cffc15 Add ELF Tool Chain's c++filt to the build
Differential Revision:	https://reviews.freebsd.org/D2408
Reviewed by:	brooks, imp
Sponsored by:	The FreeBSD Foundation
2015-04-30 21:30:33 +00:00
Andrew Turner
da713b9ed2 Disable truss, gprof, and lint on arm64, they don't build.
Sponsored by:	The FreeBSD Foundation
2015-04-14 10:15:58 +00:00
Ed Maste
6744fea6b3 Switch to ELF toolchain readelf(1)
ELF toolchain readelf lacked some functionality at the time other tools
(like size, strip, nm, etc.) were switched over to the ELF toolchain
versions. That has been addressed as of the last update, so we can add
it to the list.

PR:		198950 [exp-run]
Reviewed by:	bapt, imp, rpaulo
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D2156
2015-03-30 17:38:25 +00:00
Warner Losh
2aa222b153 These local variables are unused. gc them. 2015-03-12 08:32:22 +00:00
Enji Cooper
c7b6816fa9 Add MK_FILE to control whether or not to build file(1), libmagic(3), etc
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
2015-02-04 10:24:40 +00:00
Enji Cooper
11981695fc Add the following options to enable/disable several features in the base system
WITHOUT_BOOTPARAMD - bootparamd
WITHOUT_BOOTPD - bootpd
WITHOUT_FINGER - finger, fingerd
WITHOUT_FTP - ftp, ftpd
WITHOUT_INETD - inetd
WITHOUT_RBOOTD - rbootd
WITHOUT_TCP_WRAPPERS - tcpd, et al
WITHOUT_TFTP - tftp, tftp-server
WITHOUT_TIMED - timed

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
2015-02-04 10:19:32 +00:00
Enji Cooper
3f802165ba Add MK_TALK knob for building the talk and talkd
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
2015-01-25 04:37:44 +00:00
Enji Cooper
b29d6977f3 Add MK_ISCSI knob for building the iscsi initiator, iscsi daemon, kernel
modules, etc

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
2015-01-25 04:20:11 +00:00
Enji Cooper
79a86dafb7 Add MK_EE knob to control installing edit, ee, etc
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
2015-01-25 00:03:44 +00:00
Baptiste Daroussin
1fb816da82 Update to mandoc cvs version as of 20141201
- Compatiblity with existing manpages has been improved
- Now support ".so" directive with compressed manpages (which fixes a regression
we have since we have new man(1))
2014-12-02 23:24:57 +00:00
Ed Maste
3194293903 Build infrastructure for elftoolchain tools
Set WITH_ELFTOOLCHAIN_TOOLS in src.conf to use the elftoolchain version
of the following tools:

 * addr2line
 * elfcopy (strip / mcs)
 * nm
 * size
 * strings

Reviewed by:	bapt (earlier version)
Sponsored by:	The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D1224
2014-12-01 17:49:42 +00:00
Baptiste Daroussin
9215669c36 Readd protect(2) removed by mistake
Reported by:	emaste
2014-11-25 14:40:38 +00:00
Baptiste Daroussin
3e11bd9e2a Convert to usr.bin/ to LIBADD
Reduce overlinking
2014-11-25 14:29:10 +00:00
Baptiste Daroussin
2421b90f47 Add the demandoc utility from the mandoc project
This is a modern version of the deroff utility, useful for example to do
spellchecking on manpages
2014-11-22 21:11:17 +00:00
Baptiste Daroussin
4357ec0f20 Add the preconv utility from mandoc project
it recodes multibyte UNIX manual files into mandoc(1)
2014-11-22 20:55:36 +00:00
Baptiste Daroussin
c501e7de0a Revert accidently message Makefile from 274836 and connect soeliminate(1) 2014-11-22 01:27:45 +00:00
Baptiste Daroussin
68d12e1fae Add a new soeliminate(1) command
mandoc(1) does not provide an equivalent of the GNU groff's soelim(1) as an
external binary. It does provide the funcitonnality but internally.

Lots if manpages in ports uses ".so" directives to include the content of
another manpage, which works properly if the manpages are not compressed.
With compressed manpages it will fail. So we need to preprocess those manpages
with soelim(1) before compressing them.

soeliminate(1) add the minimum functionnality from soelim(1) required for that
task, in order to still be able to prepare properly those manpages in case we
ship the base system only with mandoc as a manpage renderer.

soeliminate(1) accept all the arguments from soelim(1) for compatibility but
only '-I dir' is really functionnal.

Name it soeliminate and not soelim, so groff from base or ports can still call
soelim(1) for its internal use and avoid potential incompatibilities

MFC after:	1 month
2014-11-22 01:22:03 +00:00
Devin Teske
7fc89952f1 Re-enable dpv(1,3): Introduced via r274116; temporarily disabled
shortly thereafter via r274124 until I could get the right recipe
down w/respect to SUBDIR_DEPEND.

Thanks to:      ngie, ian
Reviewed by:    ian
MFC after:      21 days
X-MFC-to:       stable/10 stable/9
X-MFC-with:     274116 274120 274121 274123 274144 274146
2014-11-06 19:28:01 +00:00
Devin Teske
39341fd090 Temporarily _disable_ compilation of dpv(3) and dpv(1).
Will revisit this to find out how to solve the ordering
issue in buildworld (potentially `make -j' specific).

Reviewed by:	shurd
2014-11-05 02:58:02 +00:00
Devin Teske
041394f38a Add new libraries/utilities for data throughput visualization.
dpv(3): dialog progress view library
dpv(1): stream data from stdin or multiple paths with dialog progress view
figpar(3): configuration file parsing library

Reviews:	D714
Reviewed by:	jelischer, shurd
Discussed at:	MeetBSD California 2014 Vendor/Dev Summit
Discussed on:	-current
MFC after:	21 days
X-MFC-to:	stable/10 stable/9
2014-11-04 23:46:01 +00:00
Marcel Moolenaar
41019211af Hook xo(1) to the build -- it's like echo, but uses libxo to
support emitting machine-readable output.

Sponsored by:	Juniper Networks, Inc.
2014-10-23 23:16:21 +00:00
Baptiste Daroussin
42148fc49d New BSDL timeout(1) utility compatible with GNU timeout
it fully passes the GNU timeout regression tests, it is written in a mostly
portable way (only signal parsing is relying on non portable structures)

Phabric:	D377
2014-07-16 09:55:36 +00:00
Ed Maste
824a909300 Rename the WITHOUT_VT_SUPPORT knob to WITHOUT_VT
The _SUPPORT knobs have a consistent meaning which differs from the
behaviour controlled by this knob.  As the knob is opt-out and has not
appeared in a release the impact should be low.

Suggested by:	imp, wblock
MFC after:	1 week
2014-06-30 00:20:12 +00:00
Gavin Atkinson
2d4f49b998 Remove csup(1) and its associated cpasswd(1) tool.
With the move by the FreeBSD Project away from CVSup as a distribution
mechanism, there is no longer a need to keep this in base.

Approved by:	mux (around a year ago), silence on -hackers
X-MFC-after:	never
2014-06-25 12:06:45 +00:00
Gavin Atkinson
47200bbd05 Add a stub send-pr that simply points people towards the online support
pages, to give people used to send-pr a bit of guidance.

MFC after:	3 days
2014-06-17 12:59:03 +00:00
Ed Maste
b410b62c85 vt fontcvt: move to usr.bin/vtfontcvt
vtfontcvt is useful for end users to convert arbitrary bitmap fonts
for use by vt(4).  It can also be used as a build tool, allowing us
to keep the source font data in the src tree rather than uuencoded
binaries.

Reviewed by:	ray, wblock (D183)
Sponsored by:	The FreeBSD Foundation
2014-06-10 18:29:45 +00:00
Julio Merino
8c7ec47a40 Move atf-sh from /usr/bin/ to /usr/libexec/
In r266650, we made libatf-c and libatf-c++ private libraries so that no
components outside of the source tree could unintendedly depend on them.

This change does the same for the "atf-sh library" by moving the atf-sh
interpreter from its public location in /usr/bin/ to the private location
in /usr/libexec/.  Our build system will ensure that our own test programs
use the right binary, but users won't be able to depend on atf-sh by
"mistake".

Committing this now to ride the UPDATING notice added with r267172 today.
2014-06-06 18:58:06 +00:00
Warner Losh
f510ecf0ee Optionally allow building the historical FreeBSD make program and
install it as fmake. This defaults to no. This should be viewed as the
first step towards evental migration of this historic code to ports
and removal from the tree.
2014-05-10 16:37:39 +00:00
Warner Losh
d03127447d Remove support for WITHOUT_BMAKE. bmake is now the only make that can
build world, so it is the only make we build or install. fmake is
still in the tree, but disconnected, and upgrades from older systems
that still have bmake has not been removed, but its state has not been
tested (it should work given how minimal the work to upgrade to bmake
is).
2014-05-06 04:22:37 +00:00
Warner Losh
c6063d0da8 Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.
2014-05-06 04:22:01 +00:00
Warner Losh
d6d1ee211a Add option WITHOUT_VI to not build/install vi. vi was the largest
binary without a knob to turn it off.
2014-04-24 23:17:40 +00:00
Marcel Moolenaar
81a44173f0 Hook mkimg(1) to the build. 2014-03-29 19:04:38 +00:00
Dimitry Andric
54ff5d7323 Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
all the SUBDIR entries in parallel, instead of serially.  Apply this
option to a selected number of Makefiles, which can greatly speed up the
build on multi-core machines, when using make -j.

This can be extended to more Makefiles later on, whenever they are
verified to work correctly with parallel building.

I tested this on a 24-core machine, with make -j48 buildworld (N = 6):

                before    stddev       after    stddev
                =======   ======       =======  ======
real time        1741.1     16.5         959.8     2.7
user time       12468.7     16.4       14393.0    16.8
sys  time        1825.0     54.8        2110.6    22.8

(user+sys)/real     8.2                   17.1

E.g. the build was approximately 45% faster in real time.  On machines
with less cores, or with lower -j settings, the speedup will not be as
impressive.  But at least you can now almost max out a machine with
buildworld!

Submitted by:	jilles
MFC after:	2 weeks
2014-03-26 22:30:38 +00:00
Warner Losh
8e6ee5a9a1 Don't build BSDL dtc if the GPL dtc is enabled. 2014-01-29 05:00:04 +00:00
Julio Merino
fba3cde907 Build and install the atf tests.
Reviewed by:	freebsd-testing
Approved by:	rpaulo (mentor)
2013-11-08 14:33:41 +00:00
Julio Merino
e01d128a42 Subsume the functionality of MK_ATF into MK_TESTS.
There is no reason to keep the two knobs separate: if tests are
enabled, the ATF libraries are required; and if tests are disabled,
the ATF libraries are not necessary.  Keeping the two just serves
to complicate the build.

Reviewed by:	freebsd-testing
Approved by:	rpaulo (mentor)
2013-11-08 14:24:47 +00:00
Nathan Whitehorn
d25d7b7d6e Disable use of compiler atomic builtins. For APR, this is limited to
architectures where they are known not to work. For SVN itself, use
the least common denominator and disable them across the board. This
allows svnlite to build and run on all FreeBSD architectures.

Approved by:	re (gjb)
2013-10-04 18:27:02 +00:00
Dag-Erling Smørgrav
56b72efe82 Remove BIND.
Approved by:	re (gjb)
2013-09-30 17:23:45 +00:00
Dag-Erling Smørgrav
fdde88bc82 Build and install drill(1).
Approved by:	re (blanket)
2013-09-22 20:30:55 +00:00
John Baldwin
55648840de Extend the support for exempting processes from being killed when swap is
exhausted.
- Add a new protect(1) command that can be used to set or revoke protection
  from arbitrary processes.  Similar to ktrace it can apply a change to all
  existing descendants of a process as well as future descendants.
- Add a new procctl(2) system call that provides a generic interface for
  control operations on processes (as opposed to the debugger-specific
  operations provided by ptrace(2)).  procctl(2) uses a combination of
  idtype_t and an id to identify the set of processes on which to operate
  similar to wait6().
- Add a PROC_SPROTECT control operation to manage the protection status
  of a set of processes.  MADV_PROTECT still works for backwards
  compatability.
- Add a p_flag2 to struct proc (and a corresponding ki_flag2 to kinfo_proc)
  the first bit of which is used to track if P_PROTECT should be inherited
  by new child processes.

Reviewed by:	kib, jilles (earlier version)
Approved by:	re (delphij)
MFC after:	1 month
2013-09-19 18:53:42 +00:00
Edward Tomasz Napierala
009ea47eb2 Bring in the new iSCSI target and initiator.
Reviewed by:	ken (parts)
Approved by:	re (delphij)
Sponsored by:	FreeBSD Foundation
2013-09-14 15:29:06 +00:00
Dag-Erling Smørgrav
9aad2c487e Hook host(1) up to the build in the LDNS case.
Approved by:	re (blanket)
2013-09-08 20:48:23 +00:00
Andrew Turner
4a90491e55 Subversion requires atomic functions we only support on arm with clang. 2013-08-19 17:44:19 +00:00
Tim Kientzle
ad4422d3cd Enable svnlite on armv6. 2013-06-29 15:31:23 +00:00
Peter Wemm
4cc6982f98 Only enable svn on amd64/ia64/sparc64/i386. 2013-06-20 02:26:32 +00:00