d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
36acfc6507
freebsd-dev/sys/kern
History
Alexander Motin 36acfc6507 Rework CPU load balancing in SCHED_ULE: 
- In sched_pickcpu() be more careful taking previous CPU on SMT systems.
Do it only if all other logical CPUs of that physical one are idle to avoid
extra resource sharing.
 - In sched_pickcpu() change general logic of CPU selection. First
look for idle CPU, sharing last level cache with previously used one,
skipping SMT CPU groups. If none found, search all CPUs for the least loaded
one, where the thread with its priority can run now. If none found, search
just for the least loaded CPU.
 - Make cpu_search() compare lowest/highest CPU load when comparing CPU
groups with equal load. That allows to differentiate 1+1 and 2+0 loads.
 - Make cpu_search() to prefer specified (previous) CPU or group if load
is equal. This improves cache affinity for more complicated topologies.
 - Randomize CPU selection if above factors are equal. Previous code tend
to prefer CPUs with lower IDs, causing unneeded collisions.
 - Rework periodic balancer in sched_balance_group(). With cpu_search()
more intelligent now, make balansing process flat, removing recursion
over the topology tree. That fixes double swap problem and makes load
distribution more even and predictable.

All together this gives 10-15% performance improvement in many tests on
CPUs with SMT, such as Core i7, for number of threads is less then number
of logical CPUs. In some tests it also gives positive effect to systems
without SMT.

Reviewed by:	jeff
Tested by:	flo, hackers@
MFC after:	1 month
Sponsored by:	iXsystems, Inc.
2012-02-27 10:31:54 +00:00
..
bus_if.m Add a new bus method, BUS_ADJUST_RESOURCE() that is intended to be a 2011-04-29 21:36:45 +00:00
capabilities.conf Add experimental support for process descriptors 2011-08-18 22:51:30 +00:00
clock_if.m
cpufreq_if.m
device_if.m Update comments for DEVICE_PROBE() to reflect that BUS_PROBE_DEFAULT is 2011-05-24 13:22:40 +00:00
genassym.sh
imgact_aout.c Do not trash the argv[0] pointer for an a.out process on amd64. 2011-06-16 22:00:59 +00:00
imgact_elf32.c
imgact_elf64.c
imgact_elf.c Finally, try to enable the nxstacks on amd64 and powerpc64 for both 64bit 2012-01-30 07:56:00 +00:00
imgact_gzip.c Fix misuse of the kernel map in miscellaneous image activators. 2012-02-17 23:47:16 +00:00
imgact_shell.c
inflate.c
init_main.c TDF_* flags should be used with td_flags field and TDP_* flags should be used 2012-01-22 11:01:36 +00:00
init_sysent.c - Add the ffclock_getcounter(), ffclock_getestimate() and ffclock_setestimate() 2011-11-21 01:26:10 +00:00
kern_acct.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
kern_alq.c
kern_clock.c Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs. 2011-11-07 15:43:11 +00:00
kern_clocksource.c Set negative quality to TSC timecounter when C3 state is enabled for Intel 2011-06-22 16:40:45 +00:00
kern_condvar.c
kern_conf.c Remove unneeded newline. It fits in 80 columns now. 2012-02-10 14:55:47 +00:00
kern_cons.c belatedly transfer copyrights from libkern/gets.c to kern_cons.c 2011-12-17 15:50:45 +00:00
kern_context.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
kern_cpu.c Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs. 2011-11-07 15:43:11 +00:00
kern_cpuset.c Add a missing curly bracket 2011-12-05 10:34:52 +00:00
kern_ctf.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
kern_descrip.c Reformat comment to be more readable in standard Xterm. 2011-11-15 01:48:53 +00:00
kern_dtrace.c Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs. 2011-11-07 06:44:47 +00:00
kern_environment.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
kern_et.c Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs. 2011-11-07 15:43:11 +00:00
kern_event.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
kern_exec.c Use shared lock for the executable vnode in the exec path after the 2012-01-19 23:03:31 +00:00
kern_exit.c Restore the return statement erronously removed in the r232048. 2012-02-24 11:02:35 +00:00
kern_fail.c Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs. 2011-11-07 06:44:47 +00:00
kern_ffclock.c Revise the sysctl handling code and restructure the hierarchy of sysctls 2011-12-01 07:19:13 +00:00
kern_fork.c Allow the parent to gather the exit status of the children reparented 2012-02-23 11:50:23 +00:00
kern_gzio.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
kern_hhook.c Staticize malloc types. 2011-04-13 11:28:46 +00:00
kern_idle.c On multi-core, multi-threaded PPC systems, it is important that the threads 2011-05-31 15:11:43 +00:00
kern_intr.c Be pedantic and change // comment to C-style one. 2012-01-16 20:42:56 +00:00
kern_jail.c Analogous to r232059, add a parameter for the ZFS file system: 2012-02-26 16:30:39 +00:00
kern_khelp.c Use the full and proper company name for Swinburne University of Technology 2011-04-12 08:13:18 +00:00
kern_kthread.c Whenever a new kernel thread is spawned, explicitly clear any CPU affinity 2012-02-04 16:49:29 +00:00
kern_ktr.c Use strchr() and strrchr(). 2012-01-02 12:12:10 +00:00
kern_ktrace.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
kern_linker.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
kern_lock.c panic: add a switch and infrastructure for stopping other CPUs in SMP case 2011-12-11 21:02:01 +00:00
kern_lockf.c Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs. 2011-11-07 06:44:47 +00:00
kern_lockstat.c
kern_loginclass.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
kern_malloc.c Eliminate stale numbers from a comment. 2011-12-07 16:27:23 +00:00
kern_mbuf.c
kern_mib.c Define two new sysctl node flags: CTLFLAG_CAPRD and CTLFLAG_CAPRW, which 2011-07-17 23:05:24 +00:00
kern_module.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
kern_mtxpool.c
kern_mutex.c panic: add a switch and infrastructure for stopping other CPUs in SMP case 2011-12-11 21:02:01 +00:00
kern_ntptime.c Introduce the sysclock_getsnapshot() and sysclock_snap2bintime() KPIs. The 2011-12-24 01:32:01 +00:00
kern_osd.c
kern_physio.c
kern_pmc.c Commit the support for removing cpumask_t and replacing it directly with 2011-05-05 14:39:14 +00:00
kern_poll.c Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs. 2011-11-07 15:43:11 +00:00
kern_priv.c
kern_proc.c Add sysctl to retrieve or set umask of another process. 2012-02-26 14:25:48 +00:00
kern_prot.c Fix a typo. (s/nessesary/necessary/) 2012-01-08 18:48:36 +00:00
kern_racct.c kern_racct: move sys/systm.h inclusion to its proper place 2011-12-12 07:46:10 +00:00
kern_rctl.c Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs. 2011-11-07 06:44:47 +00:00
kern_resource.c Change kern.proc.rlimit sysctl to: 2012-01-22 20:25:00 +00:00
kern_rmlock.c panic: add a switch and infrastructure for stopping other CPUs in SMP case 2011-12-11 21:02:01 +00:00
kern_rwlock.c panic: add a switch and infrastructure for stopping other CPUs in SMP case 2011-12-11 21:02:01 +00:00
kern_sdt.c
kern_sema.c
kern_shutdown.c Avoid to check the same cache line/variable from all the locking 2012-01-28 14:00:21 +00:00
kern_sig.c Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs. 2011-11-07 15:43:11 +00:00
kern_switch.c critical_exit: ignore td_owepreempt if kdb_active is set 2011-12-04 21:27:41 +00:00
kern_sx.c put sys/systm.h at its proper place or add it if missing 2011-12-12 10:05:13 +00:00
kern_synch.c panic: add a switch and infrastructure for stopping other CPUs in SMP case 2011-12-11 21:02:01 +00:00
kern_syscalls.c
kern_sysctl.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
kern_tc.c Add a missing break. This bug was introduced in r228856. 2012-02-10 06:30:52 +00:00
kern_thr.c Document a large number of currently undocumented sysctls. While here 2011-12-13 00:38:50 +00:00
kern_thread.c Some small fixes to CPU accounting for threads: 2012-01-03 21:03:28 +00:00
kern_time.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
kern_timeout.c Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs. 2011-11-07 06:44:47 +00:00
kern_umtx.c In revision 231989, we pass a 16-bit clock ID into kernel, however 2012-02-25 02:12:17 +00:00
kern_uuid.c Convert all users of IF_ADDR_LOCK to use new locking macros that specify 2012-01-05 19:00:36 +00:00
kern_xxx.c
ksched.c Use p4prio_to_tsprio to calculate TS priority instead of using 2011-06-07 02:50:14 +00:00
link_elf_obj.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
link_elf.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
linker_if.m
Make.tags.inc Update tags build script 2011-07-10 00:53:04 +00:00
Makefile
makesyscalls.sh - Add the ffclock_getcounter(), ffclock_getestimate() and ffclock_setestimate() 2011-11-21 01:26:10 +00:00
md4c.c
md5c.c
p1003_1b.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
posix4_mib.c Define two new sysctl node flags: CTLFLAG_CAPRD and CTLFLAG_CAPRW, which 2011-07-17 23:05:24 +00:00
sched_4bsd.c Some small fixes to CPU accounting for threads: 2012-01-03 21:03:28 +00:00
sched_ule.c Rework CPU load balancing in SCHED_ULE: 2012-02-27 10:31:54 +00:00
serdev_if.m
stack_protector.c
subr_acl_nfs4.c Remove assertion against empty NFSv4 ACLs. An empty ACL is not exactly 2011-10-05 17:29:49 +00:00
subr_acl_posix1e.c
subr_autoconf.c
subr_blist.c - Fix typos s/(more|less) then|\1 than/ 2011-12-03 15:41:37 +00:00
subr_bufring.c
subr_bus.c - Fix undefined behavior when device_get_name is null 2012-01-15 07:09:18 +00:00
subr_clock.c
subr_devstat.c Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs. 2011-11-07 15:43:11 +00:00
subr_disk.c
subr_eventhandler.c
subr_fattime.c
subr_firmware.c Do not increment the parent firmware's reference count when any other 2011-11-19 00:20:28 +00:00
subr_hash.c Convert panic()s to KASSERT()s. This is an optimisation for 2012-01-23 16:31:46 +00:00
subr_hints.c Use strchr() and strrchr(). 2012-01-02 12:12:10 +00:00
subr_kdb.c introduce cngrab/cnungrab stub calls in some places where they make sense 2011-12-17 15:11:22 +00:00
subr_kobj.c As it turns out, r186347 actually is insufficient to avoid the use of the 2011-11-15 20:11:03 +00:00
subr_lock.c panic: add a switch and infrastructure for stopping other CPUs in SMP case 2011-12-11 21:02:01 +00:00
subr_log.c Add kqueue support to /dev/klog. 2012-02-01 14:34:52 +00:00
subr_mbpool.c
subr_mchain.c - Use uint8_t for the variable x and spell the size of the variable 2012-02-06 06:03:16 +00:00
subr_module.c
subr_msgbuf.c Add a timestamp to the msgbuf output in order to determine when when 2012-02-16 05:11:35 +00:00
subr_param.c
subr_pcpu.c Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs. 2011-11-07 06:44:47 +00:00
subr_power.c
subr_prf.c Fix double vision syndrome (read: double output) when in the 2011-10-16 14:16:46 +00:00
subr_prof.c Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs. 2011-11-07 15:43:11 +00:00
subr_rman.c Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs. 2011-11-07 15:43:11 +00:00
subr_rtc.c Core structure and functions to support a feed-forward clock within the kernel. 2011-11-19 14:10:16 +00:00
subr_sbuf.c r222015 introduced a new assertion that the size of a fixed-length sbuf 2011-08-19 08:29:10 +00:00
subr_scanf.c Xen netback driver rewrite. 2012-01-26 16:35:09 +00:00
subr_sglist.c
subr_sleepqueue.c Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs. 2011-11-07 15:43:11 +00:00
subr_smp.c Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs. 2011-11-07 15:43:11 +00:00
subr_stack.c Constify stack argument for functions that don't modify it. 2011-11-16 19:06:55 +00:00
subr_syscall.c The PTRACESTOP() macro is used only once. Inline the only use and remove 2012-02-11 14:49:25 +00:00
subr_taskqueue.c Ensure that ta_pending doesn't overflow u_short by capping its value at USHRT_MAX. 2011-09-15 08:42:06 +00:00
subr_trap.c Assert that exiting process does not return to usermode. 2011-10-03 16:58:58 +00:00
subr_turnstile.c Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs. 2011-11-07 15:43:11 +00:00
subr_uio.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
subr_unit.c
subr_witness.c Convert the per-interface address list lock from a mutex to a reader/writer 2012-01-09 19:34:12 +00:00
sys_capability.c Revisit the capability failure trace points. The initial implementation 2011-10-18 07:28:58 +00:00
sys_generic.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
sys_pipe.c Fix fchmod() and fchown() on fifos. 2012-02-26 15:14:29 +00:00
sys_procdesc.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
sys_process.c Allow the parent to gather the exit status of the children reparented 2012-02-23 11:50:23 +00:00
sys_socket.c Add the fo_chown and fo_chmod methods to struct fileops and use them 2011-08-16 20:07:47 +00:00
syscalls.c - Add the ffclock_getcounter(), ffclock_getestimate() and ffclock_setestimate() 2011-11-21 01:26:10 +00:00
syscalls.master - Add the ffclock_getcounter(), ffclock_getestimate() and ffclock_setestimate() 2011-11-21 01:26:10 +00:00
systrace_args.c - Add the ffclock_getcounter(), ffclock_getestimate() and ffclock_setestimate() 2011-11-21 01:26:10 +00:00
sysv_ipc.c
sysv_msg.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
sysv_sem.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
sysv_shm.c Close a race due to dropping of the map lock between creating a map entry 2012-02-19 00:28:49 +00:00
tty_compat.c
tty_info.c Fix whitespace inconsistencies in TTY code. 2012-02-06 18:15:46 +00:00
tty_inq.c Use strchr() and strrchr(). 2012-01-02 12:12:10 +00:00
tty_outq.c Fix whitespace inconsistencies in the TTY layer and its drivers owned by me. 2011-06-26 18:26:20 +00:00
tty_pts.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
tty_tty.c
tty_ttydisc.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
tty.c Also call the low-level driver if ->c_iflag & (IXON|IXOFF|IXANY) changes. 2012-02-26 20:56:49 +00:00
uipc_accf.c
uipc_cow.c
uipc_debug.c
uipc_domain.c Catch up with r195837 (2.5 years ago) which renamed net_add_domain() to domain_add(). 2012-02-23 17:47:19 +00:00
uipc_mbuf2.c
uipc_mbuf.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
uipc_mqueue.c Use strchr() and strrchr(). 2012-01-02 12:12:10 +00:00
uipc_sem.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
uipc_shm.c Correct an error of omission in the implementation of the truncation 2012-01-08 20:09:26 +00:00
uipc_sockbuf.c Document a large number of currently undocumented sysctls. While here 2011-12-13 00:38:50 +00:00
uipc_socket.c Add SO_PROTOCOL/SO_PROTOTYPE socket SOL_SOCKET-level option to get the 2012-02-26 13:55:43 +00:00
uipc_syscalls.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
uipc_usrreq.c When detaching an unix domain socket, uipc_detach() checks 2012-02-25 10:15:41 +00:00
vfs_acl.c In order to maximize the re-usability of kernel code in user space this 2011-09-16 13:58:51 +00:00
vfs_aio.c Add 32-bit compat code for AIO kevent flags introduced in revision 230857. 2012-02-05 04:49:31 +00:00
vfs_bio.c Fix typo. 2012-02-26 19:10:14 +00:00
vfs_cache.c o Reduce chances for integer overflow. 2012-02-25 12:06:40 +00:00
vfs_cluster.c Unbreak detection of the async mode for clustered writes after r231075. 2012-02-08 15:07:19 +00:00
vfs_default.c Existing VOP_VPTOCNP() interface has a fatal flow that is critical for 2011-11-19 07:50:49 +00:00
vfs_export.c
vfs_extattr.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
vfs_hash.c
vfs_init.c Modify vfs_register() to use a hash calculation 2011-09-13 21:01:26 +00:00
vfs_lookup.c Avoid double-unlock or double unreference for ndp->ni_dvp when the vnode dp 2012-01-01 18:45:59 +00:00
vfs_mount.c Analogous to r230407 a separate path buffer in vfs_mount.c is required 2012-02-05 10:59:50 +00:00
vfs_mountroot.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
vfs_subr.c When detaching an unix domain socket, uipc_detach() checks 2012-02-25 10:15:41 +00:00
vfs_syscalls.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
vfs_vnops.c Fix found places where uio_resid is truncated to int. 2012-02-21 01:05:12 +00:00
vnode_if.src Add 5 spare VOPs as placeholders to avoid breaking the KBI in the future 2012-01-06 20:06:45 +00:00