freebsd-nq/sys/vm
Adrian Chadd 6520495abc Add an initial NUMA affinity/policy configuration for threads and processes.
This is based on work done by jeff@ and jhb@, as well as the numa.diff
patch that has been circulating when someone asks for first-touch NUMA
on -10 or -11.

* Introduce a simple set of VM policy and iterator types.
* tie the policy types into the vm_phys path for now, mirroring how
  the initial first-touch allocation work was enabled.
* add syscalls to control changing thread and process defaults.
* add a global NUMA VM domain policy.
* implement a simple cascade policy order - if a thread policy exists, use it;
  if a process policy exists, use it; use the default policy.
* processes inherit policies from their parent processes, threads inherit
  policies from their parent threads.
* add a simple tool (numactl) to query and modify default thread/process
  policities.
* add documentation for the new syscalls, for numa and for numactl.
* re-enable first touch NUMA again by default, as now policies can be
  set in a variety of methods.

This is only relevant for very specific workloads.

This doesn't pretend to be a final NUMA solution.

The previous defaults in -HEAD (with MAXMEMDOM set) can be achieved by
'sysctl vm.default_policy=rr'.

This is only relevant if MAXMEMDOM is set to something other than 1.
Ie, if you're using GENERIC or a modified kernel with non-NUMA, then
this is a glorified no-op for you.

Thank you to Norse Corp for giving me access to rather large
(for FreeBSD!) NUMA machines in order to develop and verify this.

Thank you to Dell for providing me with dual socket sandybridge
and westmere v3 hardware to do NUMA development with.

Thank you to Scott Long at Netflix for providing me with access
to the two-socket, four-domain haswell v3 hardware.

Thank you to Peter Holm for running the stress testing suite
against the NUMA branch during various stages of development!

Tested:

* MIPS (regression testing; non-NUMA)
* i386 (regression testing; non-NUMA GENERIC)
* amd64 (regression testing; non-NUMA GENERIC)
* westmere, 2 socket (thankyou norse!)
* sandy bridge, 2 socket (thankyou dell!)
* ivy bridge, 2 socket (thankyou norse!)
* westmere-EX, 4 socket / 1TB RAM (thankyou norse!)
* haswell, 2 socket (thankyou norse!)
* haswell v3, 2 socket (thankyou dell)
* haswell v3, 2x18 core (thankyou scott long / netflix!)

* Peter Holm ran a stress test suite on this work and found one
  issue, but has not been able to verify it (it doesn't look NUMA
  related, and he only saw it once over many testing runs.)

* I've tested bhyve instances running in fixed NUMA domains and cpusets;
  all seems to work correctly.

Verified:

* intel-pcm - pcm-numa.x and pcm-memory.x, whilst selecting different
  NUMA policies for processes under test.

Review:

This was reviewed through phabricator (https://reviews.freebsd.org/D2559)
as well as privately and via emails to freebsd-arch@.  The git history
with specific attributes is available at https://github.com/erikarn/freebsd/
in the NUMA branch (https://github.com/erikarn/freebsd/compare/local/adrian_numa_policy).

This has been reviewed by a number of people (stas, rpaulo, kib, ngie,
wblock) but not achieved a clear consensus.  My hope is that with further
exposure and testing more functionality can be implemented and evaluated.

Notes:

* The VM doesn't handle unbalanced domains very well, and if you have an overly
  unbalanced memory setup whilst under high memory pressure, VM page allocation
  may fail leading to a kernel panic.  This was a problem in the past, but it's
  much more easily triggered now with these tools.

* This work only controls the path through vm_phys; it doesn't yet strongly/predictably
  affect contigmalloc, KVA placement, UMA, etc.  So, driver placement of memory
  isn't really guaranteed in any way.  That's next on my plate.

Sponsored by:	Norse Corp, Inc.; Dell
2015-07-11 15:21:37 +00:00
..
_vm_radix.h On all the architectures, avoid to preallocate the physical memory 2013-08-09 11:28:55 +00:00
default_pager.c Place VM objects on the object list when created and never remove them. 2015-05-08 19:43:37 +00:00
device_pager.c Place VM objects on the object list when created and never remove them. 2015-05-08 19:43:37 +00:00
memguard.c Add comments about CTLFLAG_RDTUN vs. TUNABLE_INT_FETCH. 2015-03-26 05:20:18 +00:00
memguard.h Replace kernel virtual address space allocation with vmem. This provides 2013-08-07 06:21:20 +00:00
phys_pager.c Place VM objects on the object list when created and never remove them. 2015-05-08 19:43:37 +00:00
pmap.h Change pmap_enter(9) interface to take flags parameter and superpage 2014-08-08 17:12:03 +00:00
redzone.c Pull in r267961 and r267973 again. Fix for issues reported will follow. 2014-06-28 03:56:17 +00:00
redzone.h
sg_pager.c Place VM objects on the object list when created and never remove them. 2015-05-08 19:43:37 +00:00
swap_pager.c o Un-inline vm_pager_get_pages(), vm_pager_get_pages_async(). 2015-06-17 22:44:27 +00:00
swap_pager.h
uma_core.c Huge cleanup of random(4) code. 2015-06-30 17:00:45 +00:00
uma_dbg.c If INVARIANTS is specified, add ctor/dtor to junk memory if they are 2015-06-25 20:44:46 +00:00
uma_dbg.h
uma_int.h Revert r281451. It causes a panic/hang early in boot for a number of 2015-04-24 17:03:53 +00:00
uma.h The vmem callback to reclaim kmem arena address space on low or 2015-05-09 20:08:36 +00:00
vm_domain.c Add an initial NUMA affinity/policy configuration for threads and processes. 2015-07-11 15:21:37 +00:00
vm_domain.h Add an initial NUMA affinity/policy configuration for threads and processes. 2015-07-11 15:21:37 +00:00
vm_extern.h Add a new file operations hook for mmap operations. File type-specific 2015-06-04 19:41:15 +00:00
vm_fault.c Make KPI of vm_pager_get_pages() more strict: if a pager changes a page 2015-06-12 11:32:20 +00:00
vm_glue.c Make KPI of vm_pager_get_pages() more strict: if a pager changes a page 2015-06-12 11:32:20 +00:00
vm_init.c Pull in r267961 and r267973 again. Fix for issues reported will follow. 2014-06-28 03:56:17 +00:00
vm_kern.c Correct a type error in kmem_unback(). Previously, kmem_unback() did not 2015-06-10 05:17:14 +00:00
vm_kern.h - Add a statically allocated memguard arena since it is needed very early 2013-08-13 22:40:43 +00:00
vm_map.c Account for the main process stack being one page below the highest 2015-07-02 15:22:13 +00:00
vm_map.h Oops. vm_map_simplify_entry() is used by mac_proc_vm_revoke_recurse(), so 2014-09-08 02:25:01 +00:00
vm_meter.c Place VM objects on the object list when created and never remove them. 2015-05-08 19:43:37 +00:00
vm_mmap.c Add a local variable initialization needed in the OBJT_DEFAULT case. 2015-07-05 22:26:19 +00:00
vm_object.c Make KPI of vm_pager_get_pages() more strict: if a pager changes a page 2015-06-12 11:32:20 +00:00
vm_object.h Provide vnode in memory map info for files on tmpfs 2015-06-02 18:37:04 +00:00
vm_page.c Retire VM_FREEPOOL_CACHE as the next step in eliminating PG_CACHE pages. 2015-06-08 04:59:32 +00:00
vm_page.h The intention of r254304 was to scan the active queue continuously. 2015-07-08 17:45:59 +00:00
vm_pageout.c The intention of r254304 was to scan the active queue continuously. 2015-07-08 17:45:59 +00:00
vm_pageout.h
vm_pager.c o Un-inline vm_pager_get_pages(), vm_pager_get_pages_async(). 2015-06-17 22:44:27 +00:00
vm_pager.h o Un-inline vm_pager_get_pages(), vm_pager_get_pages_async(). 2015-06-17 22:44:27 +00:00
vm_param.h Rename global cnt to vm_cnt to avoid shadowing. 2014-03-22 10:26:09 +00:00
vm_phys.c Add an initial NUMA affinity/policy configuration for threads and processes. 2015-07-11 15:21:37 +00:00
vm_phys.h Add an initial NUMA affinity/policy configuration for threads and processes. 2015-07-11 15:21:37 +00:00
vm_radix.c Pull in r267961 and r267973 again. Fix for issues reported will follow. 2014-06-28 03:56:17 +00:00
vm_radix.h Eliminate a redundant parameter to vm_radix_replace(). 2013-12-08 20:07:02 +00:00
vm_reserv.c Retire VM_FREEPOOL_CACHE as the next step in eliminating PG_CACHE pages. 2015-06-08 04:59:32 +00:00
vm_reserv.h Refactor vm_page_alloc()'s interactions with vm_reserv_alloc_page() and 2013-05-12 16:50:18 +00:00
vm_unix.c Implement lockless resource limits. 2015-06-10 10:48:12 +00:00
vm_zeroidle.c Pull in r267961 and r267973 again. Fix for issues reported will follow. 2014-06-28 03:56:17 +00:00
vm.h rename scheduler->swapper and SI_SUB_RUN_SCHEDULER->SI_SUB_LAST 2013-07-24 09:45:31 +00:00
vnode_pager.c Satisfy vm_object uma zone destructor requirements after r282660 when 2015-05-10 08:21:03 +00:00
vnode_pager.h Merge from projects/sendfile: 2014-11-23 12:01:52 +00:00