d/freebsd-nq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
f897951965
freebsd-nq/sys/kern/kern_thread.c

1265 lines
32 KiB
C
Raw Normal View History

/* -> /*- for copyright notices, minor format tweaks as necessary
2005-01-06 23:35:40 +00:00
/*-
sys/kern: adoption of SPDX licensing ID tags. Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts.
2017-11-27 15:20:12 +00:00
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
*
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
* Copyright (C) 2001 Julian Elischer <julian@freebsd.org>.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice(s), this list of conditions and the following disclaimer as
s/Muliple/Multiple Removed whitespace at EOL and EOF.
2004-01-10 18:34:01 +00:00
* the first lines of this file unmodified other than the possible
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
* addition of one or more copyright notices.
* 2. Redistributions in binary form must reproduce the above copyright
* notice(s), this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
* SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
* DAMAGE.
*/
Introduce some WITNESS improvements: - Speedup the lock orderings lookup modifying the witness graph from a linked tree to a matrix. A table lookup caches the lock orderings in order to make a O(1) access for them. Any witness object has an unique index withing this lookup cache table. - Reduce the lock contention on w_mtx acquiring it only when the LOR actually happens and not in a sane case. In order to do this don't totally flush lock lists (per-CPU spinlocks list and per-thread sleeplocks list) but check for ll_count anytime we need to have to verify allocations sanity. - Introduce the function witness_thread_exit() in the witness namespace which should verify a thread doesn't hold any witness occurrence why exiting. - Rename the sysctl debug.witness.graphs into debug.witness.fullgraph and add debug.witness.badstacks which prints out stacks for LOR revealed. This is implemented using the stack(9) support, which makes WITNESS to be dependent by the STACK option or by the DDB (including STACK) option. - Fix style(9) for src/sys/kern/subr_witness.c The hash table approach has been developed by Ilya Maykov on the behalf of Isilon Systems which kindly released the patch. Jeff Roberson, ported the patch to -CURRENT and fixed w_mtx contention, on the behalf of Nokia. Submitted by: Ilya Maykov <ivmaykov at gmail dot com> (Isilon Systems), jeff Sponsored by: Nokia
2008-08-13 18:24:22 +00:00
#include "opt_witness.h"
Inform hwpmc(4) of a thread's impending demise prior to invoking sched_throw(). Debugging help: fabient Review and testing by: fabient
2009-10-25 04:34:47 +00:00
#include "opt_hwpmc_hooks.h"
Introduce some WITNESS improvements: - Speedup the lock orderings lookup modifying the witness graph from a linked tree to a matrix. A table lookup caches the lock orderings in order to make a O(1) access for them. Any witness object has an unique index withing this lookup cache table. - Reduce the lock contention on w_mtx acquiring it only when the LOR actually happens and not in a sane case. In order to do this don't totally flush lock lists (per-CPU spinlocks list and per-thread sleeplocks list) but check for ll_count anytime we need to have to verify allocations sanity. - Introduce the function witness_thread_exit() in the witness namespace which should verify a thread doesn't hold any witness occurrence why exiting. - Rename the sysctl debug.witness.graphs into debug.witness.fullgraph and add debug.witness.badstacks which prints out stacks for LOR revealed. This is implemented using the stack(9) support, which makes WITNESS to be dependent by the STACK option or by the DDB (including STACK) option. - Fix style(9) for src/sys/kern/subr_witness.c The hash table approach has been developed by Ilya Maykov on the behalf of Isilon Systems which kindly released the patch. Jeff Roberson, ported the patch to -CURRENT and fixed w_mtx contention, on the behalf of Nokia. Submitted by: Ilya Maykov <ivmaykov at gmail dot com> (Isilon Systems), jeff Sponsored by: Nokia
2008-08-13 18:24:22 +00:00
Use __FBSDID().
2003-06-11 00:56:59 +00:00
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
#include <sys/param.h>
#include <sys/systm.h>
#include <sys/kernel.h>
#include <sys/lock.h>
#include <sys/mutex.h>
#include <sys/proc.h>
Add a rangelock implementation, intended to be used to range-locking the i/o regions of the vnode data space. The implementation is quite simple-minded, it uses the list of the lock requests, ordered by arrival time. Each request may be for read or for write. The implementation is fair FIFO. MFC after: 2 month
2012-05-30 16:06:38 +00:00
#include <sys/rangelock.h>
1. Count last time slice, this intends to fix "calcru: runtime went backwards" bug for threaded process. 2. Add comment about possible logical problem with scheduler. MFC after: 3 days
2006-03-14 04:00:21 +00:00
#include <sys/resourcevar.h>
Implement the DTrace sched provider. This implementation aims to be compatible with the sched provider implemented by Solaris and its open- source derivatives. Full documentation of the sched provider can be found on Oracle's DTrace wiki pages. Note that for compatibility with scripts originally written for Solaris, serveral probes are defined that will never fire. These probes are defined to fire when Solaris-specific features perform certain actions. As these features are not present in FreeBSD, the probes can never fire. Also, I have added a two probes that are not defined in Solaris, lend-pri and load-change. These probes have been added to make it possible to collect schedgraph data with DTrace. Finally, a few probes are defined in Solaris to take a cpuinfo_t * argument. As it was not immediately clear to me how to translate that to FreeBSD, currently those probes are passed NULL in place of a cpuinfo_t *. Sponsored by: Sandvine Incorporated MFC after: 2 weeks
2012-05-15 01:30:25 +00:00
#include <sys/sdt.h>
Shuffle some code around.
2004-06-11 17:48:20 +00:00
#include <sys/smp.h>
- Implement a mechanism for allowing schedulers to place scheduler dependant data in the scheduler independant structures (proc, ksegrp, kse, thread). - Implement unused stubs for this mechanism in sched_4bsd. Approved by: re Reviewed by: luigi, trb Tested on: x86, alpha
2002-11-21 01:22:38 +00:00
#include <sys/sched.h>
Switch the sleep/wakeup and condition variable implementations to use the sleep queue interface: - Sleep queues attempt to merge some of the benefits of both sleep queues and condition variables. Having sleep qeueus in a hash table avoids having to allocate a queue head for each wait channel. Thus, struct cv has shrunk down to just a single char * pointer now. However, the hash table does not hold threads directly, but queue heads. This means that once you have located a queue in the hash bucket, you no longer have to walk the rest of the hash chain looking for threads. Instead, you have a list of all the threads sleeping on that wait channel. - Outside of the sleepq code and the sleep/cv code the kernel no longer differentiates between cv's and sleep/wakeup. For example, calls to abortsleep() and cv_abort() are replaced with a call to sleepq_abort(). Thus, the TDF_CVWAITQ flag is removed. Also, calls to unsleep() and cv_waitq_remove() have been replaced with calls to sleepq_remove(). - The sched_sleep() function no longer accepts a priority argument as sleep's no longer inherently bump the priority. Instead, this is soley a propery of msleep() which explicitly calls sched_prio() before blocking. - The TDF_ONSLEEPQ flag has been dropped as it was never used. The associated TDF_SET_ONSLEEPQ and TDF_CLR_ON_SLEEPQ macros have also been dropped and replaced with a single explicit clearing of td_wchan. TD_SET_ONSLEEPQ() would really have only made sense if it had taken the wait channel and message as arguments anyway. Now that that only happens in one place, a macro would be overkill.
2004-02-27 18:52:44 +00:00
#include <sys/sleepqueue.h>
Refactor select to reduce contention and hide internal implementation details from consumers. - Track individual selecters on a per-descriptor basis such that there are no longer collisions and after sleeping for events only those descriptors which triggered events must be rescaned. - Protect the selinfo (per descriptor) structure with a mtx pool mutex. mtx pool mutexes were chosen to preserve api compatibility with existing code which does nothing but bzero() to setup selinfo structures. - Use a per-thread wait channel rather than a global wait channel. - Hide select implementation details in a seltd structure which is opaque to the rest of the kernel. - Provide a 'selsocket' interface for those kernel consumers who wish to select on a socket when they have no fd so they no longer have to be aware of select implementation details. Tested by: kris Reviewed on: arch
2007-12-16 06:21:20 +00:00
#include <sys/selinfo.h>
Call kern_thr_exit() instead of duplicating it. This code is missing the racct_subr() call from kern_thr_exit() and would require further code duplication in future changes. Reviewed by: kib MFC after: 1 week
2015-12-29 23:16:20 +00:00
#include <sys/syscallsubr.h>
In preparation for switching linuxulator to the use the native 1:1 threads add a hook for cleaning thread resources before the thread die. Differential Revision: https://reviews.freebsd.org/D1038
2015-05-24 14:51:29 +00:00
#include <sys/sysent.h>
Add an implementation of turnstiles and change the sleep mutex code to use turnstiles to implement blocking isntead of implementing a thread queue directly. These turnstiles are somewhat similar to those used in Solaris 7 as described in Solaris Internals but are also different. Turnstiles do not come out of a fixed-sized pool. Rather, each thread is assigned a turnstile when it is created that it frees when it is destroyed. When a thread blocks on a lock, it donates its turnstile to that lock to serve as queue of blocked threads. The queue associated with a given lock is found by a lookup in a simple hash table. The turnstile itself is protected by a lock associated with its entry in the hash table. This means that sched_lock is no longer needed to contest on a mutex. Instead, sched_lock is only used when manipulating run queues or thread priorities. Turnstiles also implement priority propagation inherently. Currently turnstiles only support mutexes. Eventually, however, turnstiles may grow two queue's to support a non-sleepable reader/writer lock implementation. For more details, see the comments in sys/turnstile.h and kern/subr_turnstile.c. The two primary advantages from the turnstile code include: 1) the size of struct mutex shrinks by four pointers as it no longer stores the thread queue linkages directly, and 2) less contention on sched_lock in SMP systems including the ability for multiple CPUs to contend on different locks simultaneously (not that this last detail is necessarily that much of a big win). Note that 1) means that this commit is a kernel ABI breaker, so don't mix old modules with a new kernel and vice versa. Tested on: i386 SMP, sparc64 SMP, alpha SMP
2003-11-11 22:07:29 +00:00
#include <sys/turnstile.h>
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
#include <sys/ktr.h>
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
#include <sys/rwlock.h>
Allocate umtx_q from heap instead of stack, this avoids page fault panic in kernel under heavy swapping.
2005-03-05 09:15:03 +00:00
#include <sys/umtx.h>
All these files need sys/vmmeter.h, but now they got it implicitly included via sys/pcpu.h.
2017-04-17 17:07:00 +00:00
#include <sys/vmmeter.h>
Add cpuset, an api for thread to cpu binding and cpu resource grouping and assignment. - Add a reference to a struct cpuset in each thread that is inherited from the thread that created it. - Release the reference when the thread is destroyed. - Add prototypes for syscalls and macros for manipulating cpusets in sys/cpuset.h - Add syscalls to create, get, and set new numbered cpusets: cpuset(), cpuset_{get,set}id() - Add syscalls for getting and setting affinity masks for cpusets or individual threads: cpuid_{get,set}affinity() - Add types for the 'level' and 'which' parameters for the cpuset. This will permit expansion of the api to cover cpu masks for other objects identifiable with an id_t integer. For example, IRQs and Jails may be coming soon. - The root set 0 contains all valid cpus. All thread initially belong to cpuset 1. This permits migrating all threads off of certain cpus to reserve them for special applications. Sponsored by: Nokia Discussed with: arch, rwatson, brooks, davidxu, deischen Reviewed by: antoine
2008-03-02 07:39:22 +00:00
#include <sys/cpuset.h>
Inform hwpmc(4) of a thread's impending demise prior to invoking sched_throw(). Debugging help: fabient Review and testing by: fabient
2009-10-25 04:34:47 +00:00
#ifdef HWPMC_HOOKS
#include <sys/pmckern.h>
#endif
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
Add new fields to process-related data structures: - td_ar to struct thread, which holds the in-progress audit record during a system call. - p_au to struct proc, which holds per-process audit state, such as the audit identifier, audit terminal, and process audit masks. In the earlier implementation, td_ar was added to the zero'd section of struct thread. In order to facilitate merging to RELENG_6, it has been moved to the end of the data structure, requiring explicit initalization in the thread constructor. Much help from: wsalamon Obtained from: TrustedBSD Project
2006-02-02 00:37:05 +00:00
#include <security/audit/audit.h>
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
#include <vm/vm.h>
Migrate the thread stack management functions from the machine-dependent to the machine-independent parts of the VM. At the same time, this introduces vm object locking for the non-i386 platforms. Two details: 1. KSTACK_GUARD has been removed in favor of KSTACK_GUARD_PAGES. The different machine-dependent implementations used various combinations of KSTACK_GUARD and KSTACK_GUARD_PAGES. To disable guard page, set KSTACK_GUARD_PAGES to 0. 2. Remove the (unnecessary) clearing of PG_ZERO in vm_thread_new. In 5.x, (but not 4.x,) PG_ZERO can only be set if VM_ALLOC_ZERO is passed to vm_page_alloc() or vm_page_grab().
2003-06-14 23:23:55 +00:00
#include <vm/vm_extern.h>
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
#include <vm/uma.h>
- Adds event handlers for process_ctor,process_dtor, process_init, process_fini, thread_ctor, thread_dtor, thread_init, thread_fini. This will allow us to extend dynamically areas in proc/thread for dtrace ;-) Reviewed by: rwatson
2007-11-15 14:20:07 +00:00
#include <sys/eventhandler.h>
ia64 does not have the same degree of stealth include file nesting, so it needs an explicit #include <machine/frame.h> to get 'struct trapframe'. The fact that it needs this at this level is rather bogus but it will not compile without it.
2002-07-17 23:43:55 +00:00
Add asserts to verify stability of struct proc and struct thread layouts. Some notes: - Only i386 and amd64 layouts are checked, other Tier-1 (or close to it) architectures would benefit from the same check. - Unconditional enabling of the asserts depend on the stability of locks memory layout. If locks are optimized to avoid bloat when some debugging or profiling features turned off, it makes sense to only assert layout for production configs. Reviewed by: badger, emaste, jhb, vangyzen Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Differential revision: https://reviews.freebsd.org/D10526
2017-04-27 21:24:50 +00:00
/*
* Asserts below verify the stability of struct thread and struct proc
* layout, as exposed by KBI to modules. On head, the KBI is allowed
* to drift, change to the structures must be accompanied by the
* assert update.
*
* On the stable branches after KBI freeze, conditions must not be
* violated. Typically new fields are moved to the end of the
* structures.
*/
#ifdef __amd64__
Implement 'domainset', a cpuset based NUMA policy mechanism. This allows userspace to control NUMA policy administratively and programmatically. Implement domainset based iterators in the page layer. Remove the now legacy numa_* syscalls. Cleanup some header polution created by having seq.h in proc.h. Reviewed by: markj, kib Discussed with: alc Tested by: pho Sponsored by: Netflix, Dell/EMC Isilon Differential Revision: https://reviews.freebsd.org/D13403
2018-01-12 22:48:23 +00:00
_Static_assert(offsetof(struct thread, td_flags) == 0xfc,
Add asserts to verify stability of struct proc and struct thread layouts. Some notes: - Only i386 and amd64 layouts are checked, other Tier-1 (or close to it) architectures would benefit from the same check. - Unconditional enabling of the asserts depend on the stability of locks memory layout. If locks are optimized to avoid bloat when some debugging or profiling features turned off, it makes sense to only assert layout for production configs. Reviewed by: badger, emaste, jhb, vangyzen Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Differential revision: https://reviews.freebsd.org/D10526
2017-04-27 21:24:50 +00:00
"struct thread KBI td_flags");
Implement 'domainset', a cpuset based NUMA policy mechanism. This allows userspace to control NUMA policy administratively and programmatically. Implement domainset based iterators in the page layer. Remove the now legacy numa_* syscalls. Cleanup some header polution created by having seq.h in proc.h. Reviewed by: markj, kib Discussed with: alc Tested by: pho Sponsored by: Netflix, Dell/EMC Isilon Differential Revision: https://reviews.freebsd.org/D13403
2018-01-12 22:48:23 +00:00
_Static_assert(offsetof(struct thread, td_pflags) == 0x104,
Add asserts to verify stability of struct proc and struct thread layouts. Some notes: - Only i386 and amd64 layouts are checked, other Tier-1 (or close to it) architectures would benefit from the same check. - Unconditional enabling of the asserts depend on the stability of locks memory layout. If locks are optimized to avoid bloat when some debugging or profiling features turned off, it makes sense to only assert layout for production configs. Reviewed by: badger, emaste, jhb, vangyzen Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Differential revision: https://reviews.freebsd.org/D10526
2017-04-27 21:24:50 +00:00
"struct thread KBI td_pflags");
Account the size of the vslock-ed memory by the thread. Assert that all such memory is unwired on return to usermode. The count of the wired memory will be used to detect the copyout mode. Tested by: pho (as part of the larger patch) Sponsored by: The FreeBSD Foundation MFC after: 1 week
2018-03-24 13:51:27 +00:00
_Static_assert(offsetof(struct thread, td_frame) == 0x470,
Add asserts to verify stability of struct proc and struct thread layouts. Some notes: - Only i386 and amd64 layouts are checked, other Tier-1 (or close to it) architectures would benefit from the same check. - Unconditional enabling of the asserts depend on the stability of locks memory layout. If locks are optimized to avoid bloat when some debugging or profiling features turned off, it makes sense to only assert layout for production configs. Reviewed by: badger, emaste, jhb, vangyzen Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Differential revision: https://reviews.freebsd.org/D10526
2017-04-27 21:24:50 +00:00
"struct thread KBI td_frame");
Account the size of the vslock-ed memory by the thread. Assert that all such memory is unwired on return to usermode. The count of the wired memory will be used to detect the copyout mode. Tested by: pho (as part of the larger patch) Sponsored by: The FreeBSD Foundation MFC after: 1 week
2018-03-24 13:51:27 +00:00
_Static_assert(offsetof(struct thread, td_emuldata) == 0x518,
Add asserts to verify stability of struct proc and struct thread layouts. Some notes: - Only i386 and amd64 layouts are checked, other Tier-1 (or close to it) architectures would benefit from the same check. - Unconditional enabling of the asserts depend on the stability of locks memory layout. If locks are optimized to avoid bloat when some debugging or profiling features turned off, it makes sense to only assert layout for production configs. Reviewed by: badger, emaste, jhb, vangyzen Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Differential revision: https://reviews.freebsd.org/D10526
2017-04-27 21:24:50 +00:00
"struct thread KBI td_emuldata");
_Static_assert(offsetof(struct proc, p_flag) == 0xb0,
"struct proc KBI p_flag");
_Static_assert(offsetof(struct proc, p_pid) == 0xbc,
"struct proc KBI p_pid");
_Static_assert(offsetof(struct proc, p_filemon) == 0x3d0,
"struct proc KBI p_filemon");
_Static_assert(offsetof(struct proc, p_comm) == 0x3e0,
"struct proc KBI p_comm");
_Static_assert(offsetof(struct proc, p_emuldata) == 0x4b8,
"struct proc KBI p_emuldata");
#endif
#ifdef __i386__
Implement 'domainset', a cpuset based NUMA policy mechanism. This allows userspace to control NUMA policy administratively and programmatically. Implement domainset based iterators in the page layer. Remove the now legacy numa_* syscalls. Cleanup some header polution created by having seq.h in proc.h. Reviewed by: markj, kib Discussed with: alc Tested by: pho Sponsored by: Netflix, Dell/EMC Isilon Differential Revision: https://reviews.freebsd.org/D13403
2018-01-12 22:48:23 +00:00
_Static_assert(offsetof(struct thread, td_flags) == 0x98,
Add asserts to verify stability of struct proc and struct thread layouts. Some notes: - Only i386 and amd64 layouts are checked, other Tier-1 (or close to it) architectures would benefit from the same check. - Unconditional enabling of the asserts depend on the stability of locks memory layout. If locks are optimized to avoid bloat when some debugging or profiling features turned off, it makes sense to only assert layout for production configs. Reviewed by: badger, emaste, jhb, vangyzen Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Differential revision: https://reviews.freebsd.org/D10526
2017-04-27 21:24:50 +00:00
"struct thread KBI td_flags");
Implement 'domainset', a cpuset based NUMA policy mechanism. This allows userspace to control NUMA policy administratively and programmatically. Implement domainset based iterators in the page layer. Remove the now legacy numa_* syscalls. Cleanup some header polution created by having seq.h in proc.h. Reviewed by: markj, kib Discussed with: alc Tested by: pho Sponsored by: Netflix, Dell/EMC Isilon Differential Revision: https://reviews.freebsd.org/D13403
2018-01-12 22:48:23 +00:00
_Static_assert(offsetof(struct thread, td_pflags) == 0xa0,
Add asserts to verify stability of struct proc and struct thread layouts. Some notes: - Only i386 and amd64 layouts are checked, other Tier-1 (or close to it) architectures would benefit from the same check. - Unconditional enabling of the asserts depend on the stability of locks memory layout. If locks are optimized to avoid bloat when some debugging or profiling features turned off, it makes sense to only assert layout for production configs. Reviewed by: badger, emaste, jhb, vangyzen Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Differential revision: https://reviews.freebsd.org/D10526
2017-04-27 21:24:50 +00:00
"struct thread KBI td_pflags");
Account the size of the vslock-ed memory by the thread. Assert that all such memory is unwired on return to usermode. The count of the wired memory will be used to detect the copyout mode. Tested by: pho (as part of the larger patch) Sponsored by: The FreeBSD Foundation MFC after: 1 week
2018-03-24 13:51:27 +00:00
_Static_assert(offsetof(struct thread, td_frame) == 0x2e8,
Add asserts to verify stability of struct proc and struct thread layouts. Some notes: - Only i386 and amd64 layouts are checked, other Tier-1 (or close to it) architectures would benefit from the same check. - Unconditional enabling of the asserts depend on the stability of locks memory layout. If locks are optimized to avoid bloat when some debugging or profiling features turned off, it makes sense to only assert layout for production configs. Reviewed by: badger, emaste, jhb, vangyzen Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Differential revision: https://reviews.freebsd.org/D10526
2017-04-27 21:24:50 +00:00
"struct thread KBI td_frame");
Account the size of the vslock-ed memory by the thread. Assert that all such memory is unwired on return to usermode. The count of the wired memory will be used to detect the copyout mode. Tested by: pho (as part of the larger patch) Sponsored by: The FreeBSD Foundation MFC after: 1 week
2018-03-24 13:51:27 +00:00
_Static_assert(offsetof(struct thread, td_emuldata) == 0x334,
Add asserts to verify stability of struct proc and struct thread layouts. Some notes: - Only i386 and amd64 layouts are checked, other Tier-1 (or close to it) architectures would benefit from the same check. - Unconditional enabling of the asserts depend on the stability of locks memory layout. If locks are optimized to avoid bloat when some debugging or profiling features turned off, it makes sense to only assert layout for production configs. Reviewed by: badger, emaste, jhb, vangyzen Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Differential revision: https://reviews.freebsd.org/D10526
2017-04-27 21:24:50 +00:00
"struct thread KBI td_emuldata");
_Static_assert(offsetof(struct proc, p_flag) == 0x68,
"struct proc KBI p_flag");
_Static_assert(offsetof(struct proc, p_pid) == 0x74,
"struct proc KBI p_pid");
_Static_assert(offsetof(struct proc, p_filemon) == 0x27c,
"struct proc KBI p_filemon");
_Static_assert(offsetof(struct proc, p_comm) == 0x288,
"struct proc KBI p_comm");
_Static_assert(offsetof(struct proc, p_emuldata) == 0x314,
"struct proc KBI p_emuldata");
#endif
Implement the DTrace sched provider. This implementation aims to be compatible with the sched provider implemented by Solaris and its open- source derivatives. Full documentation of the sched provider can be found on Oracle's DTrace wiki pages. Note that for compatibility with scripts originally written for Solaris, serveral probes are defined that will never fire. These probes are defined to fire when Solaris-specific features perform certain actions. As these features are not present in FreeBSD, the probes can never fire. Also, I have added a two probes that are not defined in Solaris, lend-pri and load-change. These probes have been added to make it possible to collect schedgraph data with DTrace. Finally, a few probes are defined in Solaris to take a cpuinfo_t * argument. As it was not immediately clear to me how to translate that to FreeBSD, currently those probes are passed NULL in place of a cpuinfo_t *. Sponsored by: Sandvine Incorporated MFC after: 2 weeks
2012-05-15 01:30:25 +00:00
SDT_PROVIDER_DECLARE(proc);
dtrace sdt: remove the ugly sname parameter of SDT_PROBE_DEFINE In its stead use the Solaris / illumos approach of emulating '-' (dash) in probe names with '__' (two consecutive underscores). Reviewed by: markj MFC after: 3 weeks
2013-11-26 08:46:27 +00:00
SDT_PROBE_DEFINE(proc, , , lwp__exit);
Implement the DTrace sched provider. This implementation aims to be compatible with the sched provider implemented by Solaris and its open- source derivatives. Full documentation of the sched provider can be found on Oracle's DTrace wiki pages. Note that for compatibility with scripts originally written for Solaris, serveral probes are defined that will never fire. These probes are defined to fire when Solaris-specific features perform certain actions. As these features are not present in FreeBSD, the probes can never fire. Also, I have added a two probes that are not defined in Solaris, lend-pri and load-change. These probes have been added to make it possible to collect schedgraph data with DTrace. Finally, a few probes are defined in Solaris to take a cpuinfo_t * argument. As it was not immediately clear to me how to translate that to FreeBSD, currently those probes are passed NULL in place of a cpuinfo_t *. Sponsored by: Sandvine Incorporated MFC after: 2 weeks
2012-05-15 01:30:25 +00:00
Make KSE a kernel option, turned on by default in all GENERIC kernel configs except sun4v (which doesn't process signals properly with KSE). Reviewed by: davidxu@
2006-10-26 21:42:22 +00:00
/*
* thread related storage.
*/
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
static uma_zone_t thread_zone;
- Split the struct kse into struct upcall and struct kse. struct kse will soon be visible only to schedulers. This greatly simplifies much the KSE code. Submitted by: davidxu
2003-02-17 05:14:26 +00:00
TAILQ_HEAD(, thread) zombie_threads = TAILQ_HEAD_INITIALIZER(zombie_threads);
Fix some entries in the locks static table of witness. In particular: - smp_tlb_mtx is no longer used, so it is axed. - smp rendezvous lock isn't really a leaf spin-mutex. Its bad placement in the table, however, has been the source of a false positive LOR reporting with the dt_lock. However, smp rendezvous lock would have had sched_lock there for older lock, so it wasn't still a leaf lock. - allpmaps is only used in ia32 architecture, so it is inserted in the appropriate stub. Addictionally: - kse_zombie_lock is no longer present, so its definition is axed out. - zombie_lock doesn't need to have an exported symbol, so just let's it be declared as static. Tested by: kris Approved by: jeff (mentor) Approved by: re
2007-09-20 20:38:43 +00:00
static struct mtx zombie_lock;
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
MTX_SYSINIT(zombie_lock, &zombie_lock, "zombie lock", MTX_SPIN);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
Solve a complex exit race introduced with thread_lock: - Add a count of exiting threads, p_exitthreads, to struct proc. - Increment p_exithreads when we set the deadthread in thread_exit(). - When we thread_stash() a deadthread use an atomic to drop the count. - Spin until the p_exithreads count reaches 0 in thread_wait(). - Lock the last exiting thread momentarily to be certain that it has exited cpu_throw(). - Restructure thread_wait(). It does not need a loop as there will only ever be one thread. Tested by: moose@opera.com Reported by: kris, moose@opera.com
2007-06-12 07:24:46 +00:00
static void thread_zombie(struct thread *);
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
static int thread_unsuspend_one(struct thread *td, struct proc *p,
bool boundary);
Solve a complex exit race introduced with thread_lock: - Add a count of exiting threads, p_exitthreads, to struct proc. - Increment p_exithreads when we set the deadthread in thread_exit(). - When we thread_stash() a deadthread use an atomic to drop the count. - Spin until the p_exithreads count reaches 0 in thread_wait(). - Lock the last exiting thread momentarily to be certain that it has exited cpu_throw(). - Restructure thread_wait(). It does not need a loop as there will only ever be one thread. Tested by: moose@opera.com Reported by: kris, moose@opera.com
2007-06-12 07:24:46 +00:00
MFp4: The unit number allocator reuses ID too fast, this may hide bugs in other code, add a ring buffer to delay freeing a thread ID.
2010-12-09 05:16:20 +00:00
#define TID_BUFFER_SIZE 1024
Assign thread IDs to kernel threads. The purpose of the thread ID (tid) is twofold: 1. When a 1:1 or M:N threaded process dumps core, we need to put the register state of each of its kernel threads in the core file. This can only be done by differentiating the pid field in the respective note. For this we need the tid. 2. When thread support is present for remote debugging the kernel with gdb(1), threads need to be identified by an integer due to limitations in the remote protocol. This requires having a tid. To minimize the impact of having thread IDs, threads that are created as part of a fork (i.e. the initial thread in a process) will inherit the process ID (i.e. tid=pid). Subsequent threads will have IDs larger than PID_MAX to avoid interference with the pid allocation algorithm. The assignment of tids is handled by thread_new_tid(). The thread ID allocation algorithm has been written with 3 assumptions in mind: 1. IDs need to be created as fast a possible, 2. Reuse of IDs may happen instantaneously, 3. Someone else will write a better algorithm.
2004-04-03 15:59:13 +00:00
struct mtx tid_lock;
Use subr_unit to allocate thread ID's with. Tested by: davidxu
2005-03-18 12:34:14 +00:00
static struct unrhdr *tid_unrhdr;
MFp4: The unit number allocator reuses ID too fast, this may hide bugs in other code, add a ring buffer to delay freeing a thread ID.
2010-12-09 05:16:20 +00:00
static lwpid_t tid_buffer[TID_BUFFER_SIZE];
static int tid_head, tid_tail;
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
static MALLOC_DEFINE(M_TIDHASH, "tidhash", "thread hash");
struct tidhashhead *tidhashtbl;
u_long tidhash;
struct rwlock tidhash_lock;
Introduce EVENTHANDLER_LIST and some users. This introduces a facility to EVENTHANDLER(9) for explicitly defining a reference to an event handler list. This is useful since previously all invokers of events had to do a locked traversal of the global list of event handler lists in order to find the appropriate event handler list. By keeping a pointer to the appropriate list an invoker can avoid this traversal completely. The pointer is initialized with SYSINIT(9) during the eventhandler stage. Users registering interest in events do not need to know if the event is backed by such a list, since the list is added to the global list of lists. As with lists that are not pre-defined it is safe to register for the events before the list has been created. This converts the process_* and thread_* events to using the new facility, as these are events whose locked traversals end up showing up significantly in ports build workflows (and presumably other workflows with many short lived threads/procs). It may be advantageous to convert other events to using the new facility. The el_flags field is now unused, but leave it be so that this revision can be MFC'd. Reviewed by: bdrewery, markj, mjg Approved by: rstone (mentor) In collaboration with: ian MFC after: 4 weeks Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D12814
2017-11-09 22:51:48 +00:00
EVENTHANDLER_LIST_DEFINE(thread_ctor);
EVENTHANDLER_LIST_DEFINE(thread_dtor);
EVENTHANDLER_LIST_DEFINE(thread_init);
EVENTHANDLER_LIST_DEFINE(thread_fini);
MFp4: The unit number allocator reuses ID too fast, this may hide bugs in other code, add a ring buffer to delay freeing a thread ID.
2010-12-09 05:16:20 +00:00
static lwpid_t
tid_alloc(void)
{
lwpid_t tid;
tid = alloc_unr(tid_unrhdr);
if (tid != -1)
return (tid);
mtx_lock(&tid_lock);
if (tid_head == tid_tail) {
mtx_unlock(&tid_lock);
return (-1);
}
Make the updates of the tid ring buffer' head and tail pointers explicit by moving them into separate statements from the buffer element accesses. Requested by: jhb MFC after: 3 days
2012-09-26 09:25:11 +00:00
tid = tid_buffer[tid_head];
tid_head = (tid_head + 1) % TID_BUFFER_SIZE;
MFp4: The unit number allocator reuses ID too fast, this may hide bugs in other code, add a ring buffer to delay freeing a thread ID.
2010-12-09 05:16:20 +00:00
mtx_unlock(&tid_lock);
return (tid);
}
static void
tid_free(lwpid_t tid)
{
lwpid_t tmp_tid = -1;
mtx_lock(&tid_lock);
if ((tid_tail + 1) % TID_BUFFER_SIZE == tid_head) {
Make the updates of the tid ring buffer' head and tail pointers explicit by moving them into separate statements from the buffer element accesses. Requested by: jhb MFC after: 3 days
2012-09-26 09:25:11 +00:00
tmp_tid = tid_buffer[tid_head];
tid_head = (tid_head + 1) % TID_BUFFER_SIZE;
MFp4: The unit number allocator reuses ID too fast, this may hide bugs in other code, add a ring buffer to delay freeing a thread ID.
2010-12-09 05:16:20 +00:00
}
Make the updates of the tid ring buffer' head and tail pointers explicit by moving them into separate statements from the buffer element accesses. Requested by: jhb MFC after: 3 days
2012-09-26 09:25:11 +00:00
tid_buffer[tid_tail] = tid;
tid_tail = (tid_tail + 1) % TID_BUFFER_SIZE;
MFp4: The unit number allocator reuses ID too fast, this may hide bugs in other code, add a ring buffer to delay freeing a thread ID.
2010-12-09 05:16:20 +00:00
mtx_unlock(&tid_lock);
if (tmp_tid != -1)
free_unr(tid_unrhdr, tmp_tid);
}
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
Unbreak the KSE code. Keep track of zobie threads using the Per-CPU storage during the context switch. Rearrange thread cleanups to avoid problems with Giant. Clean threads when freed or when recycled. Approved by: re (jhb)
2002-12-10 02:33:45 +00:00
* Prepare a thread for use.
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
*/
* Add a "how" argument to uma_zone constructors and initialization functions so that they know whether the allocation is supposed to be able to sleep or not. * Allow uma_zone constructors and initialation functions to return either success or error. Almost all of the ones in the tree currently return success unconditionally, but mbuf is a notable exception: the packet zone constructor wants to be able to fail if it cannot suballocate an mbuf cluster, and the mbuf allocators want to be able to fail in general in a MAC kernel if the MAC mbuf initializer fails. This fixes the panics people are seeing when they run out of memory for mbuf clusters. * Allow debug.nosleepwithlocks on WITNESS to be disabled, without changing the default. Both bmilekic and jeff have reviewed the changes made to make failable zone allocations work.
2004-08-02 00:18:36 +00:00
static int
thread_ctor(void *mem, int size, void *arg, int flags)
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
{
struct thread *td;
td = (struct thread *)mem;
Completely redo thread states. Reviewed by: davidxu@freebsd.org
2002-09-11 08:13:56 +00:00
td->td_state = TDS_INACTIVE;
Various small style fixes.
2004-09-22 15:24:33 +00:00
td->td_oncpu = NOCPU;
Add a comment explaining td_critnest's initial state and its life from that point on, as it happens relatively indirectly, and in a codepath the casual reader may not be acquainted with or find obvious. Glanced at by: jhb
2004-06-09 14:06:44 +00:00
MFp4: The unit number allocator reuses ID too fast, this may hide bugs in other code, add a ring buffer to delay freeing a thread ID.
2010-12-09 05:16:20 +00:00
td->td_tid = tid_alloc();
Sleeping is not allowed in uma->fini
2005-03-19 08:22:13 +00:00
Add a comment explaining td_critnest's initial state and its life from that point on, as it happens relatively indirectly, and in a codepath the casual reader may not be acquainted with or find obvious. Glanced at by: jhb
2004-06-09 14:06:44 +00:00
/*
* Note that td_critnest begins life as 1 because the thread is not
* running and is thereby implicitly waiting to be on the receiving
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
* end of a context switch.
Add a comment explaining td_critnest's initial state and its life from that point on, as it happens relatively indirectly, and in a codepath the casual reader may not be acquainted with or find obvious. Glanced at by: jhb
2004-06-09 14:06:44 +00:00
*/
Set td_critnest to 1 when setting up a thread since it is a MI field with MI values. This ensures that td_critnest for a newly fork'd thread is always valid. Requested by: bde (a long time ago)
2003-08-04 20:28:20 +00:00
td->td_critnest = 1;
MFp4: It is possible a lower priority thread lending priority to higher priority thread, in old code, it is ignored, however the lending should always be recorded, add field td_lend_user_pri to fix the problem, if a thread does not have borrowed priority, its value is PRI_MAX. MFC after: 1 week
2010-12-09 02:42:02 +00:00
td->td_lend_user_pri = PRI_MAX;
Introduce EVENTHANDLER_LIST and some users. This introduces a facility to EVENTHANDLER(9) for explicitly defining a reference to an event handler list. This is useful since previously all invokers of events had to do a locked traversal of the global list of event handler lists in order to find the appropriate event handler list. By keeping a pointer to the appropriate list an invoker can avoid this traversal completely. The pointer is initialized with SYSINIT(9) during the eventhandler stage. Users registering interest in events do not need to know if the event is backed by such a list, since the list is added to the global list of lists. As with lists that are not pre-defined it is safe to register for the events before the list has been created. This converts the process_* and thread_* events to using the new facility, as these are events whose locked traversals end up showing up significantly in ports build workflows (and presumably other workflows with many short lived threads/procs). It may be advantageous to convert other events to using the new facility. The el_flags field is now unused, but leave it be so that this revision can be MFC'd. Reviewed by: bdrewery, markj, mjg Approved by: rstone (mentor) In collaboration with: ian MFC after: 4 weeks Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D12814
2017-11-09 22:51:48 +00:00
EVENTHANDLER_DIRECT_INVOKE(thread_ctor, td);
Add new fields to process-related data structures: - td_ar to struct thread, which holds the in-progress audit record during a system call. - p_au to struct proc, which holds per-process audit state, such as the audit identifier, audit terminal, and process audit masks. In the earlier implementation, td_ar was added to the zero'd section of struct thread. In order to facilitate merging to RELENG_6, it has been moved to the end of the data structure, requiring explicit initalization in the thread constructor. Much help from: wsalamon Obtained from: TrustedBSD Project
2006-02-02 00:37:05 +00:00
#ifdef AUDIT
audit_thread_alloc(td);
#endif
This is initial version of POSIX priority mutex support, a new userland mutex structure is added as following: struct umutex { __lwpid_t m_owner; uint32_t m_flags; uint32_t m_ceilings[2]; uint32_t m_spare[4]; }; The m_owner represents owner thread, it is a thread id, in non-contested case, userland can simply use atomic_cmpset_int to lock the mutex, if the mutex is contested, high order bit will be set, and userland should do locking and unlocking via kernel syscall. Flag UMUTEX_PRIO_INHERIT represents pthread's PTHREAD_PRIO_INHERIT mutex, which when contention happens, kernel should do priority propagating. Flag UMUTEX_PRIO_PROTECT indicates it is pthread's PTHREAD_PRIO_PROTECT mutex, userland should initialize m_owner to contested state UMUTEX_CONTESTED, then atomic_cmpset_int will be failure and kernel syscall should be invoked to do locking, this becauses for such a mutex, kernel should always boost the thread's priority before it can lock the mutex, m_ceilings is used by PTHREAD_PRIO_PROTECT mutex, the first element is used to boost thread's priority when it locked the mutex, second element is used when the mutex is unlocked, the PTHREAD_PRIO_PROTECT mutex's link list is kept in userland, the m_ceiling[1] is managed by thread library so kernel needn't allocate memory to keep the link list, when such a mutex is unlocked, kernel reset m_owner to UMUTEX_CONTESTED. Flag USYNC_PROCESS_SHARED indicate if the synchronization object is process shared, if the flag is not set, it saves a vm_map_lookup() call. The umtx chain is still used as a sleep queue, when a thread is blocked on PTHREAD_PRIO_INHERIT mutex, a umtx_pi is allocated to support priority propagating, it is dynamically allocated and reference count is used, it is not optimized but works well in my tests, while the umtx chain has its own locking protocol, the priority propagating protocol are all protected by sched_lock because priority propagating function is called with sched_lock held from scheduler. No visible performance degradation is found which these changes. Some parameter names in _umtx_op syscall are renamed.
2006-08-28 04:24:51 +00:00
umtx_thread_alloc(td);
* Add a "how" argument to uma_zone constructors and initialization functions so that they know whether the allocation is supposed to be able to sleep or not. * Allow uma_zone constructors and initialation functions to return either success or error. Almost all of the ones in the tree currently return success unconditionally, but mbuf is a notable exception: the packet zone constructor wants to be able to fail if it cannot suballocate an mbuf cluster, and the mbuf allocators want to be able to fail in general in a MAC kernel if the MAC mbuf initializer fails. This fixes the panics people are seeing when they run out of memory for mbuf clusters. * Allow debug.nosleepwithlocks on WITNESS to be disabled, without changing the default. Both bmilekic and jeff have reviewed the changes made to make failable zone allocations work.
2004-08-02 00:18:36 +00:00
return (0);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
/*
* Reclaim a thread after use.
*/
static void
thread_dtor(void *mem, int size, void *arg)
{
Assign thread IDs to kernel threads. The purpose of the thread ID (tid) is twofold: 1. When a 1:1 or M:N threaded process dumps core, we need to put the register state of each of its kernel threads in the core file. This can only be done by differentiating the pid field in the respective note. For this we need the tid. 2. When thread support is present for remote debugging the kernel with gdb(1), threads need to be identified by an integer due to limitations in the remote protocol. This requires having a tid. To minimize the impact of having thread IDs, threads that are created as part of a fork (i.e. the initial thread in a process) will inherit the process ID (i.e. tid=pid). Subsequent threads will have IDs larger than PID_MAX to avoid interference with the pid allocation algorithm. The assignment of tids is handled by thread_new_tid(). The thread ID allocation algorithm has been written with 3 assumptions in mind: 1. IDs need to be created as fast a possible, 2. Reuse of IDs may happen instantaneously, 3. Someone else will write a better algorithm.
2004-04-03 15:59:13 +00:00
struct thread *td;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
td = (struct thread *)mem;
#ifdef INVARIANTS
/* Verify that this thread is in a safe state to free. */
switch (td->td_state) {
Completely redo thread states. Reviewed by: davidxu@freebsd.org
2002-09-11 08:13:56 +00:00
case TDS_INHIBITED:
case TDS_RUNNING:
case TDS_CAN_RUN:
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
case TDS_RUNQ:
/*
* We must never unlink a thread that is in one of
* these states, because it is currently active.
*/
panic("bad state for thread unlinking");
/* NOTREACHED */
Completely redo thread states. Reviewed by: davidxu@freebsd.org
2002-09-11 08:13:56 +00:00
case TDS_INACTIVE:
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
break;
default:
panic("bad thread state");
/* NOTREACHED */
}
#endif
When GC'ing a thread, assert that it has no active audit record. This should not happen, but with this assert, brueffer and I would not have spent 45 minutes trying to figure out why he wasn't seeing audit records with the audit version in CVS. Obtained from: TrustedBSD Project
2006-02-05 21:06:09 +00:00
#ifdef AUDIT
audit_thread_free(td);
#endif
Update ZFS from version 6 to 13 and bring some FreeBSD-specific changes. This bring huge amount of changes, I'll enumerate only user-visible changes: - Delegated Administration Allows regular users to perform ZFS operations, like file system creation, snapshot creation, etc. - L2ARC Level 2 cache for ZFS - allows to use additional disks for cache. Huge performance improvements mostly for random read of mostly static content. - slog Allow to use additional disks for ZFS Intent Log to speed up operations like fsync(2). - vfs.zfs.super_owner Allows regular users to perform privileged operations on files stored on ZFS file systems owned by him. Very careful with this one. - chflags(2) Not all the flags are supported. This still needs work. - ZFSBoot Support to boot off of ZFS pool. Not finished, AFAIK. Submitted by: dfr - Snapshot properties - New failure modes Before if write requested failed, system paniced. Now one can select from one of three failure modes: - panic - panic on write error - wait - wait for disk to reappear - continue - serve read requests if possible, block write requests - Refquota, refreservation properties Just quota and reservation properties, but don't count space consumed by children file systems, clones and snapshots. - Sparse volumes ZVOLs that don't reserve space in the pool. - External attributes Compatible with extattr(2). - NFSv4-ACLs Not sure about the status, might not be complete yet. Submitted by: trasz - Creation-time properties - Regression tests for zpool(8) command. Obtained from: OpenSolaris
2008-11-17 20:49:29 +00:00
/* Free all OSD associated to this thread. */
osd_thread_exit(td);
Do not leak mount references for dying threads. Thread might create a condition for delayed SU cleanup, which creates a reference to the mount point in td_su, but exit without returning through userret(), e.g. when terminating due to single-threading or process exit. In this case, td_su reference is not dropped and mount point cannot be freed. Handle the situation by clearing td_su also in the thread destructor and in exit1(). softdep_ast_cleanup() has to receive the thread as argument, since e.g. thread destructor is executed in different context. Reported and tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2017-02-25 10:38:18 +00:00
td_softdep_cleanup(td);
MPASS(td->td_su == NULL);
Update ZFS from version 6 to 13 and bring some FreeBSD-specific changes. This bring huge amount of changes, I'll enumerate only user-visible changes: - Delegated Administration Allows regular users to perform ZFS operations, like file system creation, snapshot creation, etc. - L2ARC Level 2 cache for ZFS - allows to use additional disks for cache. Huge performance improvements mostly for random read of mostly static content. - slog Allow to use additional disks for ZFS Intent Log to speed up operations like fsync(2). - vfs.zfs.super_owner Allows regular users to perform privileged operations on files stored on ZFS file systems owned by him. Very careful with this one. - chflags(2) Not all the flags are supported. This still needs work. - ZFSBoot Support to boot off of ZFS pool. Not finished, AFAIK. Submitted by: dfr - Snapshot properties - New failure modes Before if write requested failed, system paniced. Now one can select from one of three failure modes: - panic - panic on write error - wait - wait for disk to reappear - continue - serve read requests if possible, block write requests - Refquota, refreservation properties Just quota and reservation properties, but don't count space consumed by children file systems, clones and snapshots. - Sparse volumes ZVOLs that don't reserve space in the pool. - External attributes Compatible with extattr(2). - NFSv4-ACLs Not sure about the status, might not be complete yet. Submitted by: trasz - Creation-time properties - Regression tests for zpool(8) command. Obtained from: OpenSolaris
2008-11-17 20:49:29 +00:00
Introduce EVENTHANDLER_LIST and some users. This introduces a facility to EVENTHANDLER(9) for explicitly defining a reference to an event handler list. This is useful since previously all invokers of events had to do a locked traversal of the global list of event handler lists in order to find the appropriate event handler list. By keeping a pointer to the appropriate list an invoker can avoid this traversal completely. The pointer is initialized with SYSINIT(9) during the eventhandler stage. Users registering interest in events do not need to know if the event is backed by such a list, since the list is added to the global list of lists. As with lists that are not pre-defined it is safe to register for the events before the list has been created. This converts the process_* and thread_* events to using the new facility, as these are events whose locked traversals end up showing up significantly in ports build workflows (and presumably other workflows with many short lived threads/procs). It may be advantageous to convert other events to using the new facility. The el_flags field is now unused, but leave it be so that this revision can be MFC'd. Reviewed by: bdrewery, markj, mjg Approved by: rstone (mentor) In collaboration with: ian MFC after: 4 weeks Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D12814
2017-11-09 22:51:48 +00:00
EVENTHANDLER_DIRECT_INVOKE(thread_dtor, td);
MFp4: The unit number allocator reuses ID too fast, this may hide bugs in other code, add a ring buffer to delay freeing a thread ID.
2010-12-09 05:16:20 +00:00
tid_free(td->td_tid);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
/*
* Initialize type-stable parts of a thread (when newly created).
*/
* Add a "how" argument to uma_zone constructors and initialization functions so that they know whether the allocation is supposed to be able to sleep or not. * Allow uma_zone constructors and initialation functions to return either success or error. Almost all of the ones in the tree currently return success unconditionally, but mbuf is a notable exception: the packet zone constructor wants to be able to fail if it cannot suballocate an mbuf cluster, and the mbuf allocators want to be able to fail in general in a MAC kernel if the MAC mbuf initializer fails. This fixes the panics people are seeing when they run out of memory for mbuf clusters. * Allow debug.nosleepwithlocks on WITNESS to be disabled, without changing the default. Both bmilekic and jeff have reviewed the changes made to make failable zone allocations work.
2004-08-02 00:18:36 +00:00
static int
thread_init(void *mem, int size, int flags)
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
{
Allocate TIDs in thread_init() and deallocate them in thread_fini(). The overhead of unconditionally allocating TIDs (and likewise, unconditionally deallocating them), is amortized across multiple thread creations by the way UMA makes it possible to have type-stable storage. Previously the cost was kept down by having threads created as part of a fork operation use the process' PID as the TID. While this had some nice properties, it also introduced complexity in the way TIDs were allocated. Most importantly, by using the type-stable storage that UMA gives us this was also unnecessary. This change affects how core dumps are created and in particular how the PRSTATUS notes are dumped. Since we don't have a thread with a TID equalling the PID, we now need a different way to preserve the old and previous behavior. We do this by having the given thread (i.e. the thread passed to the core dump code in td) dump it's state first and fill in pr_pid with the actual PID. All other threads will have pr_pid contain their TIDs. The upshot of all this is that the debugger will now likely select the right LWP (=TID) as the initial thread. Credits to: julian@ for spotting how we can utilize UMA. Thanks to: all who provided julian@ with test results.
2004-06-26 18:58:22 +00:00
struct thread *td;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
td = (struct thread *)mem;
Allocate TIDs in thread_init() and deallocate them in thread_fini(). The overhead of unconditionally allocating TIDs (and likewise, unconditionally deallocating them), is amortized across multiple thread creations by the way UMA makes it possible to have type-stable storage. Previously the cost was kept down by having threads created as part of a fork operation use the process' PID as the TID. While this had some nice properties, it also introduced complexity in the way TIDs were allocated. Most importantly, by using the type-stable storage that UMA gives us this was also unnecessary. This change affects how core dumps are created and in particular how the PRSTATUS notes are dumped. Since we don't have a thread with a TID equalling the PID, we now need a different way to preserve the old and previous behavior. We do this by having the given thread (i.e. the thread passed to the core dump code in td) dump it's state first and fill in pr_pid with the actual PID. All other threads will have pr_pid contain their TIDs. The upshot of all this is that the debugger will now likely select the right LWP (=TID) as the initial thread. Credits to: julian@ for spotting how we can utilize UMA. Thanks to: all who provided julian@ with test results.
2004-06-26 18:58:22 +00:00
Switch the sleep/wakeup and condition variable implementations to use the sleep queue interface: - Sleep queues attempt to merge some of the benefits of both sleep queues and condition variables. Having sleep qeueus in a hash table avoids having to allocate a queue head for each wait channel. Thus, struct cv has shrunk down to just a single char * pointer now. However, the hash table does not hold threads directly, but queue heads. This means that once you have located a queue in the hash bucket, you no longer have to walk the rest of the hash chain looking for threads. Instead, you have a list of all the threads sleeping on that wait channel. - Outside of the sleepq code and the sleep/cv code the kernel no longer differentiates between cv's and sleep/wakeup. For example, calls to abortsleep() and cv_abort() are replaced with a call to sleepq_abort(). Thus, the TDF_CVWAITQ flag is removed. Also, calls to unsleep() and cv_waitq_remove() have been replaced with calls to sleepq_remove(). - The sched_sleep() function no longer accepts a priority argument as sleep's no longer inherently bump the priority. Instead, this is soley a propery of msleep() which explicitly calls sched_prio() before blocking. - The TDF_ONSLEEPQ flag has been dropped as it was never used. The associated TDF_SET_ONSLEEPQ and TDF_CLR_ON_SLEEPQ macros have also been dropped and replaced with a single explicit clearing of td_wchan. TD_SET_ONSLEEPQ() would really have only made sense if it had taken the wait channel and message as arguments anyway. Now that that only happens in one place, a macro would be overkill.
2004-02-27 18:52:44 +00:00
td->td_sleepqueue = sleepq_alloc();
Add an implementation of turnstiles and change the sleep mutex code to use turnstiles to implement blocking isntead of implementing a thread queue directly. These turnstiles are somewhat similar to those used in Solaris 7 as described in Solaris Internals but are also different. Turnstiles do not come out of a fixed-sized pool. Rather, each thread is assigned a turnstile when it is created that it frees when it is destroyed. When a thread blocks on a lock, it donates its turnstile to that lock to serve as queue of blocked threads. The queue associated with a given lock is found by a lookup in a simple hash table. The turnstile itself is protected by a lock associated with its entry in the hash table. This means that sched_lock is no longer needed to contest on a mutex. Instead, sched_lock is only used when manipulating run queues or thread priorities. Turnstiles also implement priority propagation inherently. Currently turnstiles only support mutexes. Eventually, however, turnstiles may grow two queue's to support a non-sleepable reader/writer lock implementation. For more details, see the comments in sys/turnstile.h and kern/subr_turnstile.c. The two primary advantages from the turnstile code include: 1) the size of struct mutex shrinks by four pointers as it no longer stores the thread queue linkages directly, and 2) less contention on sched_lock in SMP systems including the ability for multiple CPUs to contend on different locks simultaneously (not that this last detail is necessarily that much of a big win). Note that 1) means that this commit is a kernel ABI breaker, so don't mix old modules with a new kernel and vice versa. Tested on: i386 SMP, sparc64 SMP, alpha SMP
2003-11-11 22:07:29 +00:00
td->td_turnstile = turnstile_alloc();
Add a rangelock implementation, intended to be used to range-locking the i/o regions of the vnode data space. The implementation is quite simple-minded, it uses the list of the lock requests, ordered by arrival time. Each request may be for read or for write. The implementation is fair FIFO. MFC after: 2 month
2012-05-30 16:06:38 +00:00
td->td_rlqe = NULL;
Introduce EVENTHANDLER_LIST and some users. This introduces a facility to EVENTHANDLER(9) for explicitly defining a reference to an event handler list. This is useful since previously all invokers of events had to do a locked traversal of the global list of event handler lists in order to find the appropriate event handler list. By keeping a pointer to the appropriate list an invoker can avoid this traversal completely. The pointer is initialized with SYSINIT(9) during the eventhandler stage. Users registering interest in events do not need to know if the event is backed by such a list, since the list is added to the global list of lists. As with lists that are not pre-defined it is safe to register for the events before the list has been created. This converts the process_* and thread_* events to using the new facility, as these are events whose locked traversals end up showing up significantly in ports build workflows (and presumably other workflows with many short lived threads/procs). It may be advantageous to convert other events to using the new facility. The el_flags field is now unused, but leave it be so that this revision can be MFC'd. Reviewed by: bdrewery, markj, mjg Approved by: rstone (mentor) In collaboration with: ian MFC after: 4 weeks Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D12814
2017-11-09 22:51:48 +00:00
EVENTHANDLER_DIRECT_INVOKE(thread_init, td);
This is initial version of POSIX priority mutex support, a new userland mutex structure is added as following: struct umutex { __lwpid_t m_owner; uint32_t m_flags; uint32_t m_ceilings[2]; uint32_t m_spare[4]; }; The m_owner represents owner thread, it is a thread id, in non-contested case, userland can simply use atomic_cmpset_int to lock the mutex, if the mutex is contested, high order bit will be set, and userland should do locking and unlocking via kernel syscall. Flag UMUTEX_PRIO_INHERIT represents pthread's PTHREAD_PRIO_INHERIT mutex, which when contention happens, kernel should do priority propagating. Flag UMUTEX_PRIO_PROTECT indicates it is pthread's PTHREAD_PRIO_PROTECT mutex, userland should initialize m_owner to contested state UMUTEX_CONTESTED, then atomic_cmpset_int will be failure and kernel syscall should be invoked to do locking, this becauses for such a mutex, kernel should always boost the thread's priority before it can lock the mutex, m_ceilings is used by PTHREAD_PRIO_PROTECT mutex, the first element is used to boost thread's priority when it locked the mutex, second element is used when the mutex is unlocked, the PTHREAD_PRIO_PROTECT mutex's link list is kept in userland, the m_ceiling[1] is managed by thread library so kernel needn't allocate memory to keep the link list, when such a mutex is unlocked, kernel reset m_owner to UMUTEX_CONTESTED. Flag USYNC_PROCESS_SHARED indicate if the synchronization object is process shared, if the flag is not set, it saves a vm_map_lookup() call. The umtx chain is still used as a sleep queue, when a thread is blocked on PTHREAD_PRIO_INHERIT mutex, a umtx_pi is allocated to support priority propagating, it is dynamically allocated and reference count is used, it is not optimized but works well in my tests, while the umtx chain has its own locking protocol, the priority propagating protocol are all protected by sched_lock because priority propagating function is called with sched_lock held from scheduler. No visible performance degradation is found which these changes. Some parameter names in _umtx_op syscall are renamed.
2006-08-28 04:24:51 +00:00
umtx_thread_init(td);
Fix for the panic("vm_thread_new: kstack allocation failed") and silent NULL pointer dereference in the i386 and sparc64 pmap_pinit() when the kmem_alloc_nofault() failed to allocate address space. Both functions now return error instead of panicing or dereferencing NULL. As consequence, vmspace_exec() and vmspace_unshare() returns the errno int. struct vmspace arg was added to vm_forkproc() to avoid dealing with failed allocation when most of the fork1() job is already done. The kernel stack for the thread is now set up in the thread_alloc(), that itself may return NULL. Also, allocation of the first process thread is performed in the fork1() to properly deal with stack allocation failure. proc_linkup() is separated into proc_linkup() called from fork1(), and proc_linkup0(), that is used to set up the kernel process (was known as swapper). In collaboration with: Peter Holm Reviewed by: jhb
2007-11-05 11:36:16 +00:00
td->td_kstack = 0;
Initialize td_sel in the thread_init(). Struct thread is not zeroed on the initial allocation, but seltdinit() assumes that td_sel is NULL or a valid pointer. Note that thread_fini()/seltdfini() also relies on this, but correctly resets td_sel to NULL. Submitted by: luke.tw@gmail.com PR: 199518 MFC after: 1 week
2015-04-18 17:21:12 +00:00
td->td_sel = NULL;
* Add a "how" argument to uma_zone constructors and initialization functions so that they know whether the allocation is supposed to be able to sleep or not. * Allow uma_zone constructors and initialation functions to return either success or error. Almost all of the ones in the tree currently return success unconditionally, but mbuf is a notable exception: the packet zone constructor wants to be able to fail if it cannot suballocate an mbuf cluster, and the mbuf allocators want to be able to fail in general in a MAC kernel if the MAC mbuf initializer fails. This fixes the panics people are seeing when they run out of memory for mbuf clusters. * Allow debug.nosleepwithlocks on WITNESS to be disabled, without changing the default. Both bmilekic and jeff have reviewed the changes made to make failable zone allocations work.
2004-08-02 00:18:36 +00:00
return (0);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
/*
* Tear down type-stable parts of a thread (just before being discarded).
*/
static void
thread_fini(void *mem, int size)
{
Allocate TIDs in thread_init() and deallocate them in thread_fini(). The overhead of unconditionally allocating TIDs (and likewise, unconditionally deallocating them), is amortized across multiple thread creations by the way UMA makes it possible to have type-stable storage. Previously the cost was kept down by having threads created as part of a fork operation use the process' PID as the TID. While this had some nice properties, it also introduced complexity in the way TIDs were allocated. Most importantly, by using the type-stable storage that UMA gives us this was also unnecessary. This change affects how core dumps are created and in particular how the PRSTATUS notes are dumped. Since we don't have a thread with a TID equalling the PID, we now need a different way to preserve the old and previous behavior. We do this by having the given thread (i.e. the thread passed to the core dump code in td) dump it's state first and fill in pr_pid with the actual PID. All other threads will have pr_pid contain their TIDs. The upshot of all this is that the debugger will now likely select the right LWP (=TID) as the initial thread. Credits to: julian@ for spotting how we can utilize UMA. Thanks to: all who provided julian@ with test results.
2004-06-26 18:58:22 +00:00
struct thread *td;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
td = (struct thread *)mem;
Introduce EVENTHANDLER_LIST and some users. This introduces a facility to EVENTHANDLER(9) for explicitly defining a reference to an event handler list. This is useful since previously all invokers of events had to do a locked traversal of the global list of event handler lists in order to find the appropriate event handler list. By keeping a pointer to the appropriate list an invoker can avoid this traversal completely. The pointer is initialized with SYSINIT(9) during the eventhandler stage. Users registering interest in events do not need to know if the event is backed by such a list, since the list is added to the global list of lists. As with lists that are not pre-defined it is safe to register for the events before the list has been created. This converts the process_* and thread_* events to using the new facility, as these are events whose locked traversals end up showing up significantly in ports build workflows (and presumably other workflows with many short lived threads/procs). It may be advantageous to convert other events to using the new facility. The el_flags field is now unused, but leave it be so that this revision can be MFC'd. Reviewed by: bdrewery, markj, mjg Approved by: rstone (mentor) In collaboration with: ian MFC after: 4 weeks Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D12814
2017-11-09 22:51:48 +00:00
EVENTHANDLER_DIRECT_INVOKE(thread_fini, td);
Add a rangelock implementation, intended to be used to range-locking the i/o regions of the vnode data space. The implementation is quite simple-minded, it uses the list of the lock requests, ordered by arrival time. Each request may be for read or for write. The implementation is fair FIFO. MFC after: 2 month
2012-05-30 16:06:38 +00:00
rlqentry_free(td->td_rlqe);
Add an implementation of turnstiles and change the sleep mutex code to use turnstiles to implement blocking isntead of implementing a thread queue directly. These turnstiles are somewhat similar to those used in Solaris 7 as described in Solaris Internals but are also different. Turnstiles do not come out of a fixed-sized pool. Rather, each thread is assigned a turnstile when it is created that it frees when it is destroyed. When a thread blocks on a lock, it donates its turnstile to that lock to serve as queue of blocked threads. The queue associated with a given lock is found by a lookup in a simple hash table. The turnstile itself is protected by a lock associated with its entry in the hash table. This means that sched_lock is no longer needed to contest on a mutex. Instead, sched_lock is only used when manipulating run queues or thread priorities. Turnstiles also implement priority propagation inherently. Currently turnstiles only support mutexes. Eventually, however, turnstiles may grow two queue's to support a non-sleepable reader/writer lock implementation. For more details, see the comments in sys/turnstile.h and kern/subr_turnstile.c. The two primary advantages from the turnstile code include: 1) the size of struct mutex shrinks by four pointers as it no longer stores the thread queue linkages directly, and 2) less contention on sched_lock in SMP systems including the ability for multiple CPUs to contend on different locks simultaneously (not that this last detail is necessarily that much of a big win). Note that 1) means that this commit is a kernel ABI breaker, so don't mix old modules with a new kernel and vice versa. Tested on: i386 SMP, sparc64 SMP, alpha SMP
2003-11-11 22:07:29 +00:00
turnstile_free(td->td_turnstile);
Switch the sleep/wakeup and condition variable implementations to use the sleep queue interface: - Sleep queues attempt to merge some of the benefits of both sleep queues and condition variables. Having sleep qeueus in a hash table avoids having to allocate a queue head for each wait channel. Thus, struct cv has shrunk down to just a single char * pointer now. However, the hash table does not hold threads directly, but queue heads. This means that once you have located a queue in the hash bucket, you no longer have to walk the rest of the hash chain looking for threads. Instead, you have a list of all the threads sleeping on that wait channel. - Outside of the sleepq code and the sleep/cv code the kernel no longer differentiates between cv's and sleep/wakeup. For example, calls to abortsleep() and cv_abort() are replaced with a call to sleepq_abort(). Thus, the TDF_CVWAITQ flag is removed. Also, calls to unsleep() and cv_waitq_remove() have been replaced with calls to sleepq_remove(). - The sched_sleep() function no longer accepts a priority argument as sleep's no longer inherently bump the priority. Instead, this is soley a propery of msleep() which explicitly calls sched_prio() before blocking. - The TDF_ONSLEEPQ flag has been dropped as it was never used. The associated TDF_SET_ONSLEEPQ and TDF_CLR_ON_SLEEPQ macros have also been dropped and replaced with a single explicit clearing of td_wchan. TD_SET_ONSLEEPQ() would really have only made sense if it had taken the wait channel and message as arguments anyway. Now that that only happens in one place, a macro would be overkill.
2004-02-27 18:52:44 +00:00
sleepq_free(td->td_sleepqueue);
This is initial version of POSIX priority mutex support, a new userland mutex structure is added as following: struct umutex { __lwpid_t m_owner; uint32_t m_flags; uint32_t m_ceilings[2]; uint32_t m_spare[4]; }; The m_owner represents owner thread, it is a thread id, in non-contested case, userland can simply use atomic_cmpset_int to lock the mutex, if the mutex is contested, high order bit will be set, and userland should do locking and unlocking via kernel syscall. Flag UMUTEX_PRIO_INHERIT represents pthread's PTHREAD_PRIO_INHERIT mutex, which when contention happens, kernel should do priority propagating. Flag UMUTEX_PRIO_PROTECT indicates it is pthread's PTHREAD_PRIO_PROTECT mutex, userland should initialize m_owner to contested state UMUTEX_CONTESTED, then atomic_cmpset_int will be failure and kernel syscall should be invoked to do locking, this becauses for such a mutex, kernel should always boost the thread's priority before it can lock the mutex, m_ceilings is used by PTHREAD_PRIO_PROTECT mutex, the first element is used to boost thread's priority when it locked the mutex, second element is used when the mutex is unlocked, the PTHREAD_PRIO_PROTECT mutex's link list is kept in userland, the m_ceiling[1] is managed by thread library so kernel needn't allocate memory to keep the link list, when such a mutex is unlocked, kernel reset m_owner to UMUTEX_CONTESTED. Flag USYNC_PROCESS_SHARED indicate if the synchronization object is process shared, if the flag is not set, it saves a vm_map_lookup() call. The umtx chain is still used as a sleep queue, when a thread is blocked on PTHREAD_PRIO_INHERIT mutex, a umtx_pi is allocated to support priority propagating, it is dynamically allocated and reference count is used, it is not optimized but works well in my tests, while the umtx chain has its own locking protocol, the priority propagating protocol are all protected by sched_lock because priority propagating function is called with sched_lock held from scheduler. No visible performance degradation is found which these changes. Some parameter names in _umtx_op syscall are renamed.
2006-08-28 04:24:51 +00:00
umtx_thread_fini(td);
Refactor select to reduce contention and hide internal implementation details from consumers. - Track individual selecters on a per-descriptor basis such that there are no longer collisions and after sleeping for events only those descriptors which triggered events must be rescaned. - Protect the selinfo (per descriptor) structure with a mtx pool mutex. mtx pool mutexes were chosen to preserve api compatibility with existing code which does nothing but bzero() to setup selinfo structures. - Use a per-thread wait channel rather than a global wait channel. - Hide select implementation details in a seltd structure which is opaque to the rest of the kernel. - Provide a 'selsocket' interface for those kernel consumers who wish to select on a socket when they have no fd so they no longer have to be aware of select implementation details. Tested by: kris Reviewed on: arch
2007-12-16 06:21:20 +00:00
seltdfini(td);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
- Split the struct kse into struct upcall and struct kse. struct kse will soon be visible only to schedulers. This greatly simplifies much the KSE code. Submitted by: davidxu
2003-02-17 05:14:26 +00:00
Move thread related code from kern_proc.c to kern_thread.c. Add code to free KSEs and KSEGRPs on exit. Sort KSE prototypes in proc.h. Add the missing kse_exit() syscall. ksetest now does not leak KSEs and KSEGRPS. Submitted by: (parts) davidxu
2002-10-24 08:46:34 +00:00
/*
- Split the struct kse into struct upcall and struct kse. struct kse will soon be visible only to schedulers. This greatly simplifies much the KSE code. Submitted by: davidxu
2003-02-17 05:14:26 +00:00
* For a newly created process,
* link up all the structures and its initial threads etc.
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
* called from:
Remove ia64. This includes: o All directories named *ia64* o All files named *ia64* o All ia64-specific code guarded by __ia64__ o All ia64-specific makefile logic o Mention of ia64 in comments and documentation This excludes: o Everything under contrib/ o Everything under crypto/ o sys/xen/interface o sys/sys/elf_common.h Discussed at: BSDcan
2014-07-07 00:27:09 +00:00
* {arch}/{arch}/machdep.c {arch}_init(), init386() etc.
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
* proc_dtor() (should go away)
* proc_init()
Move thread related code from kern_proc.c to kern_thread.c. Add code to free KSEs and KSEGRPs on exit. Sort KSE prototypes in proc.h. Add the missing kse_exit() syscall. ksetest now does not leak KSEs and KSEGRPS. Submitted by: (parts) davidxu
2002-10-24 08:46:34 +00:00
*/
Fix for the panic("vm_thread_new: kstack allocation failed") and silent NULL pointer dereference in the i386 and sparc64 pmap_pinit() when the kmem_alloc_nofault() failed to allocate address space. Both functions now return error instead of panicing or dereferencing NULL. As consequence, vmspace_exec() and vmspace_unshare() returns the errno int. struct vmspace arg was added to vm_forkproc() to avoid dealing with failed allocation when most of the fork1() job is already done. The kernel stack for the thread is now set up in the thread_alloc(), that itself may return NULL. Also, allocation of the first process thread is performed in the fork1() to properly deal with stack allocation failure. proc_linkup() is separated into proc_linkup() called from fork1(), and proc_linkup0(), that is used to set up the kernel process (was known as swapper). In collaboration with: Peter Holm Reviewed by: jhb
2007-11-05 11:36:16 +00:00
void
proc_linkup0(struct proc *p, struct thread *td)
{
TAILQ_INIT(&p->p_threads); /* all threads in proc */
proc_linkup(p, td);
}
Move thread related code from kern_proc.c to kern_thread.c. Add code to free KSEs and KSEGRPs on exit. Sort KSE prototypes in proc.h. Add the missing kse_exit() syscall. ksetest now does not leak KSEs and KSEGRPS. Submitted by: (parts) davidxu
2002-10-24 08:46:34 +00:00
void
Make KSE a kernel option, turned on by default in all GENERIC kernel configs except sun4v (which doesn't process signals properly with KSE). Reviewed by: davidxu@
2006-10-26 21:42:22 +00:00
proc_linkup(struct proc *p, struct thread *td)
Move thread related code from kern_proc.c to kern_thread.c. Add code to free KSEs and KSEGRPs on exit. Sort KSE prototypes in proc.h. Add the missing kse_exit() syscall. ksetest now does not leak KSEs and KSEGRPS. Submitted by: (parts) davidxu
2002-10-24 08:46:34 +00:00
{
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
1. Change prototype of trapsignal and sendsig to use ksiginfo_t *, most changes in MD code are trivial, before this change, trapsignal and sendsig use discrete parameters, now they uses member fields of ksiginfo_t structure. For sendsig, this change allows us to pass POSIX realtime signal value to user code. 2. Remove cpu_thread_siginfo, it is no longer needed because we now always generate ksiginfo_t data and feed it to libpthread. 3. Add p_sigqueue to proc structure to hold shared signals which were blocked by all threads in the proc. 4. Add td_sigqueue to thread structure to hold all signals delivered to thread. 5. i386 and amd64 now return POSIX standard si_code, other arches will be fixed. 6. In this sigqueue implementation, pending signal set is kept as before, an extra siginfo list holds additional siginfo_t data for signals. kernel code uses psignal() still behavior as before, it won't be failed even under memory pressure, only exception is when deleting a signal, we should call sigqueue_delete to remove signal from sigqueue but not SIGDELSET. Current there is no kernel code will deliver a signal with additional data, so kernel should be as stable as before, a ksiginfo can carry more information, for example, allow signal to be delivered but throw away siginfo data if memory is not enough. SIGKILL and SIGSTOP have fast path in sigqueue_add, because they can not be caught or masked. The sigqueue() syscall allows user code to queue a signal to target process, if resource is unavailable, EAGAIN will be returned as specification said. Just before thread exits, signal queue memory will be freed by sigqueue_flush. Current, all signals are allowed to be queued, not only realtime signals. Earlier patch reviewed by: jhb, deischen Tested on: i386, amd64
2005-10-14 12:43:47 +00:00
sigqueue_init(&p->p_sigqueue, p);
Now SIGCHLD is always queued.
2005-12-09 02:27:55 +00:00
p->p_ksi = ksiginfo_alloc(1);
if (p->p_ksi != NULL) {
/* XXX p_ksi may be null if ksiginfo zone is not ready */
p->p_ksi->ksi_flags = KSI_EXT | KSI_INS;
Add support for queueing SIGCHLD same as other UNIX systems did. For each child process whose status has been changed, a SIGCHLD instance is queued, if the signal is stilling pending, and process changed status several times, signal information is updated to reflect latest process status. If wait() returns because the status of a child process is available, pending SIGCHLD signal associated with the child process is discarded. Any other pending SIGCHLD signals remain pending. The signal information is allocated at the same time when proc structure is allocated, if process signal queue is fully filled or there is a memory shortage, it can still send the signal to process. There is a booting time tunable kern.sigqueue.queue_sigchild which can control the behavior, setting it to zero disables the SIGCHLD queueing feature, the tunable will be removed if the function is proved that it is stable enough. Tested on: i386 (SMP and UP)
2005-11-08 09:09:26 +00:00
}
Last step to make mq_notify conform to POSIX standard, If the process has successfully attached a notification request to the message queue via a queue descriptor, file closing should remove the attachment.
2005-11-30 05:12:03 +00:00
LIST_INIT(&p->p_mqnotifier);
Move thread related code from kern_proc.c to kern_thread.c. Add code to free KSEs and KSEGRPs on exit. Sort KSE prototypes in proc.h. Add the missing kse_exit() syscall. ksetest now does not leak KSEs and KSEGRPS. Submitted by: (parts) davidxu
2002-10-24 08:46:34 +00:00
p->p_numthreads = 0;
Make KSE a kernel option, turned on by default in all GENERIC kernel configs except sun4v (which doesn't process signals properly with KSE). Reviewed by: davidxu@
2006-10-26 21:42:22 +00:00
thread_link(td, p);
Move thread related code from kern_proc.c to kern_thread.c. Add code to free KSEs and KSEGRPs on exit. Sort KSE prototypes in proc.h. Add the missing kse_exit() syscall. ksetest now does not leak KSEs and KSEGRPS. Submitted by: (parts) davidxu
2002-10-24 08:46:34 +00:00
}
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
* Initialize global thread allocation resources.
*/
void
threadinit(void)
{
Use subr_unit to allocate thread ID's with. Tested by: davidxu
2005-03-18 12:34:14 +00:00
mtx_init(&tid_lock, "TID lock", NULL, MTX_DEF);
Add a sysctl kern.pid_max, which limits the maximum pid the system is allowed to allocate, and corresponding tunable with the same name. Note that existing processes with higher pids are left intact. MFC after: 1 week
2012-08-15 15:56:21 +00:00
/*
Fix grammar. Submitted by: jh MFC after: 1 week
2012-08-16 13:01:56 +00:00
* pid_max cannot be greater than PID_MAX.
Add a sysctl kern.pid_max, which limits the maximum pid the system is allowed to allocate, and corresponding tunable with the same name. Note that existing processes with higher pids are left intact. MFC after: 1 week
2012-08-15 15:56:21 +00:00
* leave one number for thread0.
*/
give thread0 the tid 100000 and bumpt the others to start at 100001 MFC after: 1 week
2007-12-22 04:56:48 +00:00
tid_unrhdr = new_unrhdr(PID_MAX + 2, INT_MAX, &tid_lock);
Use subr_unit to allocate thread ID's with. Tested by: davidxu
2005-03-18 12:34:14 +00:00
- Implement a mechanism for allowing schedulers to place scheduler dependant data in the scheduler independant structures (proc, ksegrp, kse, thread). - Implement unused stubs for this mechanism in sched_4bsd. Approved by: re Reviewed by: luigi, trb Tested on: x86, alpha
2002-11-21 01:22:38 +00:00
thread_zone = uma_zcreate("THREAD", sched_sizeof_thread(),
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
thread_ctor, thread_dtor, thread_init, thread_fini,
Bump struct thread alignment to 32. This gives additional bits to use in locking primitives which store the lock thread pointer in the lock value. Discussed with: kib
2017-02-07 17:03:22 +00:00
32 - 1, UMA_ZONE_NOFREE);
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
tidhashtbl = hashinit(maxproc / 2, M_TIDHASH, &tidhash);
rw_init(&tidhash_lock, "tidhash");
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
/*
Solve a complex exit race introduced with thread_lock: - Add a count of exiting threads, p_exitthreads, to struct proc. - Increment p_exithreads when we set the deadthread in thread_exit(). - When we thread_stash() a deadthread use an atomic to drop the count. - Spin until the p_exithreads count reaches 0 in thread_wait(). - Lock the last exiting thread momentarily to be certain that it has exited cpu_throw(). - Restructure thread_wait(). It does not need a loop as there will only ever be one thread. Tested by: moose@opera.com Reported by: kris, moose@opera.com
2007-06-12 07:24:46 +00:00
* Place an unused thread on the zombie list.
Threading cleanup.. part 2 of several. Make part of John Birrell's KSE patch permanent.. Specifically, remove: Any reference of the ksegrp structure. This feature was never fully utilised and made things overly complicated. All code in the scheduler that tried to make threaded programs fair to unthreaded programs. Libpthread processes will already do this to some extent and libthr processes already disable it. Also: Since this makes such a big change to the scheduler(s), take the opportunity to rename some structures and elements that had to be moved anyhow. This makes the code a lot more readable. The ULE scheduler compiles again but I have no idea if it works. The 4bsd scheduler still reqires a little cleaning and some functions that now do ALMOST nothing will go away, but I thought I'd do that as a separate commit. Tested by David Xu, and Dan Eischen using libthr and libpthread.
2006-12-06 06:34:57 +00:00
* Use the slpq as that must be unused by now.
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
*/
void
Solve a complex exit race introduced with thread_lock: - Add a count of exiting threads, p_exitthreads, to struct proc. - Increment p_exithreads when we set the deadthread in thread_exit(). - When we thread_stash() a deadthread use an atomic to drop the count. - Spin until the p_exithreads count reaches 0 in thread_wait(). - Lock the last exiting thread momentarily to be certain that it has exited cpu_throw(). - Restructure thread_wait(). It does not need a loop as there will only ever be one thread. Tested by: moose@opera.com Reported by: kris, moose@opera.com
2007-06-12 07:24:46 +00:00
thread_zombie(struct thread *td)
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
{
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
mtx_lock_spin(&zombie_lock);
Threading cleanup.. part 2 of several. Make part of John Birrell's KSE patch permanent.. Specifically, remove: Any reference of the ksegrp structure. This feature was never fully utilised and made things overly complicated. All code in the scheduler that tried to make threaded programs fair to unthreaded programs. Libpthread processes will already do this to some extent and libthr processes already disable it. Also: Since this makes such a big change to the scheduler(s), take the opportunity to rename some structures and elements that had to be moved anyhow. This makes the code a lot more readable. The ULE scheduler compiles again but I have no idea if it works. The 4bsd scheduler still reqires a little cleaning and some functions that now do ALMOST nothing will go away, but I thought I'd do that as a separate commit. Tested by David Xu, and Dan Eischen using libthr and libpthread.
2006-12-06 06:34:57 +00:00
TAILQ_INSERT_HEAD(&zombie_threads, td, td_slpq);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
mtx_unlock_spin(&zombie_lock);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
Solve a complex exit race introduced with thread_lock: - Add a count of exiting threads, p_exitthreads, to struct proc. - Increment p_exithreads when we set the deadthread in thread_exit(). - When we thread_stash() a deadthread use an atomic to drop the count. - Spin until the p_exithreads count reaches 0 in thread_wait(). - Lock the last exiting thread momentarily to be certain that it has exited cpu_throw(). - Restructure thread_wait(). It does not need a loop as there will only ever be one thread. Tested by: moose@opera.com Reported by: kris, moose@opera.com
2007-06-12 07:24:46 +00:00
/*
* Release a thread that has exited after cpu_throw().
*/
void
thread_stash(struct thread *td)
{
atomic_subtract_rel_int(&td->td_proc->p_exitthreads, 1);
thread_zombie(td);
}
Add kernel support needed for the KSE-aware libpthread: - Use ucontext_t's to store KSE thread state. - Synthesize state for the UTS upon each upcall, rather than saving and copying a trapframe. - Deliver signals to KSE-aware processes via upcall. - Rename kse mailbox structure fields to be more BSD-like. - Store the UTS's stack in struct proc in a stack_t. Reviewed by: bde, deischen, julian Approved by: -arch
2002-09-16 19:26:48 +00:00
/*
Remove kernel support for M:N threading. While the KSE project was quite successful in bringing threading to FreeBSD, the M:N approach taken by the kse library was never developed to its full potential. Backwards compatibility will be provided via libmap.conf for dynamically linked binaries and static binaries will be broken.
2008-03-12 10:12:01 +00:00
* Reap zombie resources.
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
*/
void
thread_reap(void)
{
Move thread related code from kern_proc.c to kern_thread.c. Add code to free KSEs and KSEGRPs on exit. Sort KSE prototypes in proc.h. Add the missing kse_exit() syscall. ksetest now does not leak KSEs and KSEGRPS. Submitted by: (parts) davidxu
2002-10-24 08:46:34 +00:00
struct thread *td_first, *td_next;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
- Split the struct kse into struct upcall and struct kse. struct kse will soon be visible only to schedulers. This greatly simplifies much the KSE code. Submitted by: davidxu
2003-02-17 05:14:26 +00:00
* Don't even bother to lock if none at this instant,
Rewrite subr_sleepqueue.c use of callouts to not depend on the specifics of callout KPI. Esp., do not depend on the exact interface of callout_stop(9) return values. The main change is that instead of requiring precise callouts, code maintains absolute time to wake up. Callouts now should ensure that a wake occurs at the requested moment, but we can tolerate both run-away callout, and callout_stop(9) lying about running callout either way. As consequence, it removes the constant source of the bugs where sleepq_check_timeout() causes uninterruptible thread state where the thread is detached from CPU, see e.g. r234952 and r296320. Patch also removes dual meaning of the TDF_TIMEOUT flag, making code (IMO much) simpler to reason about. Tested by: pho Reviewed by: jhb Sponsored by: The FreeBSD Foundation MFC after: 1 month Differential revision: https://reviews.freebsd.org/D7137
2016-07-28 09:09:55 +00:00
* we really don't care about the next instant.
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
*/
Make KSE a kernel option, turned on by default in all GENERIC kernel configs except sun4v (which doesn't process signals properly with KSE). Reviewed by: davidxu@
2006-10-26 21:42:22 +00:00
if (!TAILQ_EMPTY(&zombie_threads)) {
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
mtx_lock_spin(&zombie_lock);
Move thread related code from kern_proc.c to kern_thread.c. Add code to free KSEs and KSEGRPs on exit. Sort KSE prototypes in proc.h. Add the missing kse_exit() syscall. ksetest now does not leak KSEs and KSEGRPS. Submitted by: (parts) davidxu
2002-10-24 08:46:34 +00:00
td_first = TAILQ_FIRST(&zombie_threads);
if (td_first)
TAILQ_INIT(&zombie_threads);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
mtx_unlock_spin(&zombie_lock);
Move thread related code from kern_proc.c to kern_thread.c. Add code to free KSEs and KSEGRPs on exit. Sort KSE prototypes in proc.h. Add the missing kse_exit() syscall. ksetest now does not leak KSEs and KSEGRPS. Submitted by: (parts) davidxu
2002-10-24 08:46:34 +00:00
while (td_first) {
Threading cleanup.. part 2 of several. Make part of John Birrell's KSE patch permanent.. Specifically, remove: Any reference of the ksegrp structure. This feature was never fully utilised and made things overly complicated. All code in the scheduler that tried to make threaded programs fair to unthreaded programs. Libpthread processes will already do this to some extent and libthr processes already disable it. Also: Since this makes such a big change to the scheduler(s), take the opportunity to rename some structures and elements that had to be moved anyhow. This makes the code a lot more readable. The ULE scheduler compiles again but I have no idea if it works. The 4bsd scheduler still reqires a little cleaning and some functions that now do ALMOST nothing will go away, but I thought I'd do that as a separate commit. Tested by David Xu, and Dan Eischen using libthr and libpthread.
2006-12-06 06:34:57 +00:00
td_next = TAILQ_NEXT(td_first, td_slpq);
Generalised support for copy-on-write structures shared by threads. Thread credentials are maintained as follows: each thread has a pointer to creds and a reference on them. The pointer is compared with proc's creds on userspace<->kernel boundary and updated if needed. This patch introduces a counter which can be compared instead, so that more structures can use this scheme without adding more comparisons on the boundary.
2015-06-10 10:43:59 +00:00
thread_cow_free(td_first);
Move thread related code from kern_proc.c to kern_thread.c. Add code to free KSEs and KSEGRPs on exit. Sort KSE prototypes in proc.h. Add the missing kse_exit() syscall. ksetest now does not leak KSEs and KSEGRPS. Submitted by: (parts) davidxu
2002-10-24 08:46:34 +00:00
thread_free(td_first);
td_first = td_next;
}
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
}
/*
* Allocate a thread.
*/
struct thread *
Reintroduce the r196640, after fixing the problem with my testing. Remove the altkstacks, instead instantiate threads with kernel stack allocated with the right size from the start. For the thread that has kernel stack cached, verify that requested stack size is equial to the actual, and reallocate the stack if sizes differ [1]. This fixes the bug introduced by r173361 that was committed several days after r173004 and consisted of kthread_add(9) ignoring the non-default kernel stack size. Also, r173361 removed the caching of the kernel stacks for a non-first thread in the process. Introduce separate kernel stack cache that keeps some limited amount of preallocated kernel stacks to lower the latency of thread allocation. Add vm_lowmem handler to prune the cache on low memory condition. This way, system with reasonable amount of the threads get lower latency of thread creation, while still not exhausting significant portion of KVA for unused kstacks. Submitted by: peter [1] Discussed with: jhb, julian, peter Reviewed by: jhb Tested by: pho (and retested according to new test scenarious) MFC after: 1 week
2009-09-01 11:41:51 +00:00
thread_alloc(int pages)
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
{
Fix for the panic("vm_thread_new: kstack allocation failed") and silent NULL pointer dereference in the i386 and sparc64 pmap_pinit() when the kmem_alloc_nofault() failed to allocate address space. Both functions now return error instead of panicing or dereferencing NULL. As consequence, vmspace_exec() and vmspace_unshare() returns the errno int. struct vmspace arg was added to vm_forkproc() to avoid dealing with failed allocation when most of the fork1() job is already done. The kernel stack for the thread is now set up in the thread_alloc(), that itself may return NULL. Also, allocation of the first process thread is performed in the fork1() to properly deal with stack allocation failure. proc_linkup() is separated into proc_linkup() called from fork1(), and proc_linkup0(), that is used to set up the kernel process (was known as swapper). In collaboration with: Peter Holm Reviewed by: jhb
2007-11-05 11:36:16 +00:00
struct thread *td;
Make KSE a kernel option, turned on by default in all GENERIC kernel configs except sun4v (which doesn't process signals properly with KSE). Reviewed by: davidxu@
2006-10-26 21:42:22 +00:00
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
thread_reap(); /* check if any zombies to get */
Fix for the panic("vm_thread_new: kstack allocation failed") and silent NULL pointer dereference in the i386 and sparc64 pmap_pinit() when the kmem_alloc_nofault() failed to allocate address space. Both functions now return error instead of panicing or dereferencing NULL. As consequence, vmspace_exec() and vmspace_unshare() returns the errno int. struct vmspace arg was added to vm_forkproc() to avoid dealing with failed allocation when most of the fork1() job is already done. The kernel stack for the thread is now set up in the thread_alloc(), that itself may return NULL. Also, allocation of the first process thread is performed in the fork1() to properly deal with stack allocation failure. proc_linkup() is separated into proc_linkup() called from fork1(), and proc_linkup0(), that is used to set up the kernel process (was known as swapper). In collaboration with: Peter Holm Reviewed by: jhb
2007-11-05 11:36:16 +00:00
td = (struct thread *)uma_zalloc(thread_zone, M_WAITOK);
KASSERT(td->td_kstack == 0, ("thread_alloc got thread with kstack"));
Reintroduce the r196640, after fixing the problem with my testing. Remove the altkstacks, instead instantiate threads with kernel stack allocated with the right size from the start. For the thread that has kernel stack cached, verify that requested stack size is equial to the actual, and reallocate the stack if sizes differ [1]. This fixes the bug introduced by r173361 that was committed several days after r173004 and consisted of kthread_add(9) ignoring the non-default kernel stack size. Also, r173361 removed the caching of the kernel stacks for a non-first thread in the process. Introduce separate kernel stack cache that keeps some limited amount of preallocated kernel stacks to lower the latency of thread allocation. Add vm_lowmem handler to prune the cache on low memory condition. This way, system with reasonable amount of the threads get lower latency of thread creation, while still not exhausting significant portion of KVA for unused kstacks. Submitted by: peter [1] Discussed with: jhb, julian, peter Reviewed by: jhb Tested by: pho (and retested according to new test scenarious) MFC after: 1 week
2009-09-01 11:41:51 +00:00
if (!vm_thread_new(td, pages)) {
Fix for the panic("vm_thread_new: kstack allocation failed") and silent NULL pointer dereference in the i386 and sparc64 pmap_pinit() when the kmem_alloc_nofault() failed to allocate address space. Both functions now return error instead of panicing or dereferencing NULL. As consequence, vmspace_exec() and vmspace_unshare() returns the errno int. struct vmspace arg was added to vm_forkproc() to avoid dealing with failed allocation when most of the fork1() job is already done. The kernel stack for the thread is now set up in the thread_alloc(), that itself may return NULL. Also, allocation of the first process thread is performed in the fork1() to properly deal with stack allocation failure. proc_linkup() is separated into proc_linkup() called from fork1(), and proc_linkup0(), that is used to set up the kernel process (was known as swapper). In collaboration with: Peter Holm Reviewed by: jhb
2007-11-05 11:36:16 +00:00
uma_zfree(thread_zone, td);
return (NULL);
}
o Rename cpu_thread_setup() to cpu_thread_alloc() to better communicate that it relates to (is called by) thread_alloc() o Add cpu_thread_free() which is called from thread_free() to counter-act cpu_thread_alloc(). i386: Have cpu_thread_free() call cpu_thread_clean() to preserve behaviour. ia64: Have cpu_thread_free() call mtx_destroy() for the mutex initialized in cpu_thread_alloc(). PR: ia64/118024
2007-11-14 20:21:54 +00:00
cpu_thread_alloc(td);
Fix for the panic("vm_thread_new: kstack allocation failed") and silent NULL pointer dereference in the i386 and sparc64 pmap_pinit() when the kmem_alloc_nofault() failed to allocate address space. Both functions now return error instead of panicing or dereferencing NULL. As consequence, vmspace_exec() and vmspace_unshare() returns the errno int. struct vmspace arg was added to vm_forkproc() to avoid dealing with failed allocation when most of the fork1() job is already done. The kernel stack for the thread is now set up in the thread_alloc(), that itself may return NULL. Also, allocation of the first process thread is performed in the fork1() to properly deal with stack allocation failure. proc_linkup() is separated into proc_linkup() called from fork1(), and proc_linkup0(), that is used to set up the kernel process (was known as swapper). In collaboration with: Peter Holm Reviewed by: jhb
2007-11-05 11:36:16 +00:00
return (td);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
Reintroduce the r196640, after fixing the problem with my testing. Remove the altkstacks, instead instantiate threads with kernel stack allocated with the right size from the start. For the thread that has kernel stack cached, verify that requested stack size is equial to the actual, and reallocate the stack if sizes differ [1]. This fixes the bug introduced by r173361 that was committed several days after r173004 and consisted of kthread_add(9) ignoring the non-default kernel stack size. Also, r173361 removed the caching of the kernel stacks for a non-first thread in the process. Introduce separate kernel stack cache that keeps some limited amount of preallocated kernel stacks to lower the latency of thread allocation. Add vm_lowmem handler to prune the cache on low memory condition. This way, system with reasonable amount of the threads get lower latency of thread creation, while still not exhausting significant portion of KVA for unused kstacks. Submitted by: peter [1] Discussed with: jhb, julian, peter Reviewed by: jhb Tested by: pho (and retested according to new test scenarious) MFC after: 1 week
2009-09-01 11:41:51 +00:00
int
thread_alloc_stack(struct thread *td, int pages)
{
KASSERT(td->td_kstack == 0,
("thread_alloc_stack called on a thread with kstack"));
if (!vm_thread_new(td, pages))
return (0);
cpu_thread_alloc(td);
return (1);
}
Allocate KSEs and KSEGRPs separatly and remove them from the proc structure. next step is to allow > 1 to be allocated per process. This would give multi-processor threads. (when the rest of the infrastructure is in place) While doing this I noticed libkvm and sys/kern/kern_proc.c:fill_kinfo_proc are diverging more than they should.. corrective action needed soon.
2002-09-15 23:52:25 +00:00
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
* Deallocate a thread.
*/
void
thread_free(struct thread *td)
{
- Implement a new mechanism for resetting lock profiling. We now guarantee that all cpus have acknowledged the cleared enable int by scheduling the resetting thread on each cpu in succession. Since all lock profiling happens within a critical section this guarantees that all cpus have left lock profiling before we clear the datastructures. - Assert that the per-thread queue of locks lock profiling is aware of is clear on thread exit. There were several cases where this was not true that slows lock profiling and leaks information. - Remove all objects from all lists before clearing any per-cpu information in reset. Lock profiling objects can migrate between per-cpu caches and previously these migrated objects could be zero'd before they'd been removed Discussed with: attilio Sponsored by: Nokia
2009-03-15 06:41:47 +00:00
lock_profile_thread_exit(td);
- Restore the NULL check for td_cpuset. This can happen if a partially constructed thread was torn down as is the case when we fail to allocate a kernel stack.
2008-03-19 06:20:21 +00:00
if (td->td_cpuset)
cpuset_rel(td->td_cpuset);
Add cpuset, an api for thread to cpu binding and cpu resource grouping and assignment. - Add a reference to a struct cpuset in each thread that is inherited from the thread that created it. - Release the reference when the thread is destroyed. - Add prototypes for syscalls and macros for manipulating cpusets in sys/cpuset.h - Add syscalls to create, get, and set new numbered cpusets: cpuset(), cpuset_{get,set}id() - Add syscalls for getting and setting affinity masks for cpusets or individual threads: cpuid_{get,set}affinity() - Add types for the 'level' and 'which' parameters for the cpuset. This will permit expansion of the api to cover cpu masks for other objects identifiable with an id_t integer. For example, IRQs and Jails may be coming soon. - The root set 0 contains all valid cpus. All thread initially belong to cpuset 1. This permits migrating all threads off of certain cpus to reserve them for special applications. Sponsored by: Nokia Discussed with: arch, rwatson, brooks, davidxu, deischen Reviewed by: antoine
2008-03-02 07:39:22 +00:00
td->td_cpuset = NULL;
o Rename cpu_thread_setup() to cpu_thread_alloc() to better communicate that it relates to (is called by) thread_alloc() o Add cpu_thread_free() which is called from thread_free() to counter-act cpu_thread_alloc(). i386: Have cpu_thread_free() call cpu_thread_clean() to preserve behaviour. ia64: Have cpu_thread_free() call mtx_destroy() for the mutex initialized in cpu_thread_alloc(). PR: ia64/118024
2007-11-14 20:21:54 +00:00
cpu_thread_free(td);
Fix for the panic("vm_thread_new: kstack allocation failed") and silent NULL pointer dereference in the i386 and sparc64 pmap_pinit() when the kmem_alloc_nofault() failed to allocate address space. Both functions now return error instead of panicing or dereferencing NULL. As consequence, vmspace_exec() and vmspace_unshare() returns the errno int. struct vmspace arg was added to vm_forkproc() to avoid dealing with failed allocation when most of the fork1() job is already done. The kernel stack for the thread is now set up in the thread_alloc(), that itself may return NULL. Also, allocation of the first process thread is performed in the fork1() to properly deal with stack allocation failure. proc_linkup() is separated into proc_linkup() called from fork1(), and proc_linkup0(), that is used to set up the kernel process (was known as swapper). In collaboration with: Peter Holm Reviewed by: jhb
2007-11-05 11:36:16 +00:00
if (td->td_kstack != 0)
vm_thread_dispose(td);
Rewrite subr_sleepqueue.c use of callouts to not depend on the specifics of callout KPI. Esp., do not depend on the exact interface of callout_stop(9) return values. The main change is that instead of requiring precise callouts, code maintains absolute time to wake up. Callouts now should ensure that a wake occurs at the requested moment, but we can tolerate both run-away callout, and callout_stop(9) lying about running callout either way. As consequence, it removes the constant source of the bugs where sleepq_check_timeout() causes uninterruptible thread state where the thread is detached from CPU, see e.g. r234952 and r296320. Patch also removes dual meaning of the TDF_TIMEOUT flag, making code (IMO much) simpler to reason about. Tested by: pho Reviewed by: jhb Sponsored by: The FreeBSD Foundation MFC after: 1 month Differential revision: https://reviews.freebsd.org/D7137
2016-07-28 09:09:55 +00:00
callout_drain(&td->td_slpcallout);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
uma_zfree(thread_zone, td);
}
Generalised support for copy-on-write structures shared by threads. Thread credentials are maintained as follows: each thread has a pointer to creds and a reference on them. The pointer is compared with proc's creds on userspace<->kernel boundary and updated if needed. This patch introduces a counter which can be compared instead, so that more structures can use this scheme without adding more comparisons on the boundary.
2015-06-10 10:43:59 +00:00
void
thread_cow_get_proc(struct thread *newtd, struct proc *p)
{
PROC_LOCK_ASSERT(p, MA_OWNED);
newtd->td_ucred = crhold(p->p_ucred);
Implement lockless resource limits. Use the same scheme implemented to manage credentials. Code needing to look at process's credentials (as opposed to thred's) is provided with *_proc variants of relevant functions. Places which possibly had to take the proc lock anyway still use the proc pointer to access limits.
2015-06-10 10:48:12 +00:00
newtd->td_limit = lim_hold(p->p_limit);
Generalised support for copy-on-write structures shared by threads. Thread credentials are maintained as follows: each thread has a pointer to creds and a reference on them. The pointer is compared with proc's creds on userspace<->kernel boundary and updated if needed. This patch introduces a counter which can be compared instead, so that more structures can use this scheme without adding more comparisons on the boundary.
2015-06-10 10:43:59 +00:00
newtd->td_cowgen = p->p_cowgen;
}
void
thread_cow_get(struct thread *newtd, struct thread *td)
{
newtd->td_ucred = crhold(td->td_ucred);
Implement lockless resource limits. Use the same scheme implemented to manage credentials. Code needing to look at process's credentials (as opposed to thred's) is provided with *_proc variants of relevant functions. Places which possibly had to take the proc lock anyway still use the proc pointer to access limits.
2015-06-10 10:48:12 +00:00
newtd->td_limit = lim_hold(td->td_limit);
Generalised support for copy-on-write structures shared by threads. Thread credentials are maintained as follows: each thread has a pointer to creds and a reference on them. The pointer is compared with proc's creds on userspace<->kernel boundary and updated if needed. This patch introduces a counter which can be compared instead, so that more structures can use this scheme without adding more comparisons on the boundary.
2015-06-10 10:43:59 +00:00
newtd->td_cowgen = td->td_cowgen;
}
void
thread_cow_free(struct thread *td)
{
Get rid of lim_update_thread and cred_update_thread. Their primary use was in thread_cow_update to free up old resources. Freeing had to be done with proc lock held and _cow_ funcs already knew how to free old structs.
2015-07-16 14:30:11 +00:00
if (td->td_ucred != NULL)
Generalised support for copy-on-write structures shared by threads. Thread credentials are maintained as follows: each thread has a pointer to creds and a reference on them. The pointer is compared with proc's creds on userspace<->kernel boundary and updated if needed. This patch introduces a counter which can be compared instead, so that more structures can use this scheme without adding more comparisons on the boundary.
2015-06-10 10:43:59 +00:00
crfree(td->td_ucred);
Get rid of lim_update_thread and cred_update_thread. Their primary use was in thread_cow_update to free up old resources. Freeing had to be done with proc lock held and _cow_ funcs already knew how to free old structs.
2015-07-16 14:30:11 +00:00
if (td->td_limit != NULL)
Implement lockless resource limits. Use the same scheme implemented to manage credentials. Code needing to look at process's credentials (as opposed to thred's) is provided with *_proc variants of relevant functions. Places which possibly had to take the proc lock anyway still use the proc pointer to access limits.
2015-06-10 10:48:12 +00:00
lim_free(td->td_limit);
Generalised support for copy-on-write structures shared by threads. Thread credentials are maintained as follows: each thread has a pointer to creds and a reference on them. The pointer is compared with proc's creds on userspace<->kernel boundary and updated if needed. This patch introduces a counter which can be compared instead, so that more structures can use this scheme without adding more comparisons on the boundary.
2015-06-10 10:43:59 +00:00
}
void
thread_cow_update(struct thread *td)
{
struct proc *p;
Get rid of lim_update_thread and cred_update_thread. Their primary use was in thread_cow_update to free up old resources. Freeing had to be done with proc lock held and _cow_ funcs already knew how to free old structs.
2015-07-16 14:30:11 +00:00
struct ucred *oldcred;
struct plimit *oldlimit;
Generalised support for copy-on-write structures shared by threads. Thread credentials are maintained as follows: each thread has a pointer to creds and a reference on them. The pointer is compared with proc's creds on userspace<->kernel boundary and updated if needed. This patch introduces a counter which can be compared instead, so that more structures can use this scheme without adding more comparisons on the boundary.
2015-06-10 10:43:59 +00:00
p = td->td_proc;
Get rid of lim_update_thread and cred_update_thread. Their primary use was in thread_cow_update to free up old resources. Freeing had to be done with proc lock held and _cow_ funcs already knew how to free old structs.
2015-07-16 14:30:11 +00:00
oldcred = NULL;
oldlimit = NULL;
Generalised support for copy-on-write structures shared by threads. Thread credentials are maintained as follows: each thread has a pointer to creds and a reference on them. The pointer is compared with proc's creds on userspace<->kernel boundary and updated if needed. This patch introduces a counter which can be compared instead, so that more structures can use this scheme without adding more comparisons on the boundary.
2015-06-10 10:43:59 +00:00
PROC_LOCK(p);
Get rid of lim_update_thread and cred_update_thread. Their primary use was in thread_cow_update to free up old resources. Freeing had to be done with proc lock held and _cow_ funcs already knew how to free old structs.
2015-07-16 14:30:11 +00:00
if (td->td_ucred != p->p_ucred) {
oldcred = td->td_ucred;
td->td_ucred = crhold(p->p_ucred);
}
if (td->td_limit != p->p_limit) {
oldlimit = td->td_limit;
td->td_limit = lim_hold(p->p_limit);
}
Generalised support for copy-on-write structures shared by threads. Thread credentials are maintained as follows: each thread has a pointer to creds and a reference on them. The pointer is compared with proc's creds on userspace<->kernel boundary and updated if needed. This patch introduces a counter which can be compared instead, so that more structures can use this scheme without adding more comparisons on the boundary.
2015-06-10 10:43:59 +00:00
td->td_cowgen = p->p_cowgen;
PROC_UNLOCK(p);
Get rid of lim_update_thread and cred_update_thread. Their primary use was in thread_cow_update to free up old resources. Freeing had to be done with proc lock held and _cow_ funcs already knew how to free old structs.
2015-07-16 14:30:11 +00:00
if (oldcred != NULL)
crfree(oldcred);
if (oldlimit != NULL)
lim_free(oldlimit);
Generalised support for copy-on-write structures shared by threads. Thread credentials are maintained as follows: each thread has a pointer to creds and a reference on them. The pointer is compared with proc's creds on userspace<->kernel boundary and updated if needed. This patch introduces a counter which can be compared instead, so that more structures can use this scheme without adding more comparisons on the boundary.
2015-06-10 10:43:59 +00:00
}
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
* Discard the current thread and exit from its context.
Shuffle some code around.
2004-06-11 17:48:20 +00:00
* Always called with scheduler locked.
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
*
* Because we can't free a thread while we're operating under its context,
Unbreak the KSE code. Keep track of zobie threads using the Per-CPU storage during the context switch. Rearrange thread cleanups to avoid problems with Giant. Clean threads when freed or when recycled. Approved by: re (jhb)
2002-12-10 02:33:45 +00:00
* push the current thread into our CPU's deadthread holder. This means
* we needn't worry about someone else grabbing our context before we
Remove kernel support for M:N threading. While the KSE project was quite successful in bringing threading to FreeBSD, the M:N approach taken by the kse library was never developed to its full potential. Backwards compatibility will be provided via libmap.conf for dynamically linked binaries and static binaries will be broken.
2008-03-12 10:12:01 +00:00
* do a cpu_throw().
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
*/
void
thread_exit(void)
{
Some small fixes to CPU accounting for threads: - Only initialize the per-cpu switchticks and switchtime in sched_throw() for the very first context switch on APs during boot. This avoids a small gap between the middle of thread_exit() and sched_throw() where time is not accounted to any thread. - In thread_exit(), update the timestamp bookkeeping to track the changes to mi_switch() introduced by td_rux so that the code once again matches the comment claiming it is mimicing mi_switch(). Specifically, only update the per-thread stats directly and depend on ruxagg() to update p_rux rather than adjusting p_rux directly. While here, move the timestamp bookkeeping as late in the function as possible. Reviewed by: bde, kib MFC after: 1 week
2012-01-03 21:03:28 +00:00
uint64_t runtime, new_switchtime;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
struct thread *td;
- Move rusage from being per-process in struct pstats to per-thread in td_ru. This removes the requirement for per-process synchronization in statclock() and mi_switch(). This was previously supported by sched_lock which is going away. All modifications to rusage are now done in the context of the owning thread. reads proceed without locks. - Aggregate exiting threads rusage in thread_exit() such that the exiting thread's rusage is not lost. - Provide a new routine, rufetch() to fetch an aggregate of all rusage structures from all threads in a process. This routine must be used in any place requiring a rusage from a process prior to it's exit. The exited process's rusage is still available via p_ru. - Aggregate tick statistics only on demand via rufetch() or when a thread exits. Tick statistics are kept in the thread and protected by sched_lock until it exits. Initial patch by: attilio Reviewed by: attilio, bde (some objections), arch (mostly silent)
2007-06-01 01:12:45 +00:00
struct thread *td2;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
struct proc *p;
A suspended thread can, in fact, be swapped out. Thus, thread_unsuspend_one() needs to optionally wakeup the swapper. Since we hold the thread lock for that entire function, however, we have to push that requirement up into the caller. Found by: rwatson
2008-08-22 16:15:58 +00:00
int wakeup_swapper;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
td = curthread;
p = td->td_proc;
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
PROC_SLOCK_ASSERT(p, MA_OWNED);
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
mtx_assert(&Giant, MA_NOTOWNED);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
PROC_LOCK_ASSERT(p, MA_OWNED);
Fix crack-smoking code that was panicing on the quad xeon: - If either of proc or kse are NULL during thread_exit(), then the kernel is going to fault because parts of the function assume they aren't NULL. Instead, just assert they aren't NULL (as well as the kse group) and assume they are in all of the code. It doesn't make sense for them to be NULL here anyways. - Move the PROC_UNLOCK(p) up above clearing td_proc, etc. since otherwise we will panic if the proc's lock is contested. Submitted by: jhb@freebsd.org
2002-08-29 19:49:53 +00:00
KASSERT(p != NULL, ("thread exiting without a process"));
In thread_exit(), include more information about the thread/process context in the KTR trace record. In particular, include the same information as passed for mi_switch() and fork_exit() KTR trace records.
2004-08-06 22:06:14 +00:00
CTR3(KTR_PROC, "thread_exit: thread %p (pid %ld, %s)", td,
A bunch more files that should probably print out a thread name instead of a process name.
2007-11-14 06:51:33 +00:00
(long)p->p_pid, td->td_name);
actually implement proc:::lwp-exit probe MFC after: 4 days
2017-03-11 15:47:27 +00:00
SDT_PROBE0(proc, , , lwp__exit);
1. Change prototype of trapsignal and sendsig to use ksiginfo_t *, most changes in MD code are trivial, before this change, trapsignal and sendsig use discrete parameters, now they uses member fields of ksiginfo_t structure. For sendsig, this change allows us to pass POSIX realtime signal value to user code. 2. Remove cpu_thread_siginfo, it is no longer needed because we now always generate ksiginfo_t data and feed it to libpthread. 3. Add p_sigqueue to proc structure to hold shared signals which were blocked by all threads in the proc. 4. Add td_sigqueue to thread structure to hold all signals delivered to thread. 5. i386 and amd64 now return POSIX standard si_code, other arches will be fixed. 6. In this sigqueue implementation, pending signal set is kept as before, an extra siginfo list holds additional siginfo_t data for signals. kernel code uses psignal() still behavior as before, it won't be failed even under memory pressure, only exception is when deleting a signal, we should call sigqueue_delete to remove signal from sigqueue but not SIGDELSET. Current there is no kernel code will deliver a signal with additional data, so kernel should be as stable as before, a ksiginfo can carry more information, for example, allow signal to be delivered but throw away siginfo data if memory is not enough. SIGKILL and SIGSTOP have fast path in sigqueue_add, because they can not be caught or masked. The sigqueue() syscall allows user code to queue a signal to target process, if resource is unavailable, EAGAIN will be returned as specification said. Just before thread exits, signal queue memory will be freed by sigqueue_flush. Current, all signals are allowed to be queued, not only realtime signals. Earlier patch reviewed by: jhb, deischen Tested on: i386, amd64
2005-10-14 12:43:47 +00:00
KASSERT(TAILQ_EMPTY(&td->td_sigqueue.sq_list), ("signal pending"));
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
When exiting a thread, submit any pending record. Today, we don't audit thread exit, but should that happen, this will prevent unhappiness, as the thread exit system call will never return, and hence not commit the record. Pointed out by/with: cognet Obtained from: TrustedBSD Project
2006-02-06 01:51:08 +00:00
#ifdef AUDIT
AUDIT_SYSCALL_EXIT(0, td);
#endif
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
/*
* drop FPU & debug register state storage, or any other
* architecture specific resources that
* would not be on a new untouched process.
*/
Remove XXX comments from kern_thread.c. In one case, there is no reason for it in modern times. In the other case, expand the comment stating instead of doubting. Reviewed by: jhb Sponsored by: The FreeBSD Foundation MFC after: 1 week Approved by: re (hrs) X-Differential revision: https://reviews.freebsd.org/D6731
2016-06-16 12:01:11 +00:00
cpu_thread_exit(td);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
Fix crack-smoking code that was panicing on the quad xeon: - If either of proc or kse are NULL during thread_exit(), then the kernel is going to fault because parts of the function assume they aren't NULL. Instead, just assert they aren't NULL (as well as the kse group) and assume they are in all of the code. It doesn't make sense for them to be NULL here anyways. - Move the PROC_UNLOCK(p) up above clearing td_proc, etc. since otherwise we will panic if the proc's lock is contested. Submitted by: jhb@freebsd.org
2002-08-29 19:49:53 +00:00
/*
Use UMA as a complex object allocator. The process allocator now caches and hands out complete process structures *including substructures* . i.e. it get's the process structure with the first thread (and soon KSE) already allocated and attached, all in one hit. For the average non threaded program (non KSE that is) the allocated thread and its stack remain attached to the process, even when the process is unused and in the process cache. This saves having to allocate and attach it later, effectively bringing us (hopefully) close to the efficiency of pre-KSE systems where these were a single structure. Reviewed by: davidxu@freebsd.org, peter@freebsd.org
2002-09-06 07:00:37 +00:00
* The last thread is left attached to the process
* So that the whole bundle gets recycled. Skip
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
* all this stuff if we never had threads.
* EXIT clears all sign of other threads when
* it goes to single threading, so the last thread always
* takes the short path.
Fix crack-smoking code that was panicing on the quad xeon: - If either of proc or kse are NULL during thread_exit(), then the kernel is going to fault because parts of the function assume they aren't NULL. Instead, just assert they aren't NULL (as well as the kse group) and assume they are in all of the code. It doesn't make sense for them to be NULL here anyways. - Move the PROC_UNLOCK(p) up above clearing td_proc, etc. since otherwise we will panic if the proc's lock is contested. Submitted by: jhb@freebsd.org
2002-08-29 19:49:53 +00:00
*/
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
if (p->p_flag & P_HADTHREADS) {
if (p->p_numthreads > 1) {
Right now, thread_single(SINGLE_EXIT) returns after the p_numthreads reaches 1. The p_numthreads counter is decremented in thread_exit() by a call to thread_unlink(). This means that the exiting threads may still execute on other CPUs when thread_single(SINGLE_EXIT) returns. As result, vmspace could be destroyed while paging structures are still used on other CPUs by exiting threads. Delay the return from thread_single(SINGLE_EXIT) until all threads are really destroyed by thread_stash() after the last switch out. The p_exitthreads counter already provides the required mechanism, move the wait from the thread_wait() (which is called from wait(2) code) into thread_single(). Reported by: many (as "panic: pmap active <addr>") Reviewed by: alc, jhb Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-09-03 08:18:07 +00:00
atomic_add_int(&td->td_proc->p_exitthreads, 1);
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
thread_unlink(td);
- Move rusage from being per-process in struct pstats to per-thread in td_ru. This removes the requirement for per-process synchronization in statclock() and mi_switch(). This was previously supported by sched_lock which is going away. All modifications to rusage are now done in the context of the owning thread. reads proceed without locks. - Aggregate exiting threads rusage in thread_exit() such that the exiting thread's rusage is not lost. - Provide a new routine, rufetch() to fetch an aggregate of all rusage structures from all threads in a process. This routine must be used in any place requiring a rusage from a process prior to it's exit. The exited process's rusage is still available via p_ru. - Aggregate tick statistics only on demand via rufetch() or when a thread exits. Tick statistics are kept in the thread and protected by sched_lock until it exits. Initial patch by: attilio Reviewed by: attilio, bde (some objections), arch (mostly silent)
2007-06-01 01:12:45 +00:00
td2 = FIRST_THREAD_IN_PROC(p);
sched_exit_thread(td2, td);
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
/*
* The test below is NOT true if we are the
Fix typo in comment. MFC after: 3 days
2010-05-04 06:06:01 +00:00
* sole exiting thread. P_STOPPED_SINGLE is unset
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
* in exit1() after it is the only survivor.
*/
if (P_SHOULDSTOP(p) == P_STOPPED_SINGLE) {
if (p->p_numthreads == p->p_suspcount) {
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
thread_lock(p->p_singlethread);
A suspended thread can, in fact, be swapped out. Thus, thread_unsuspend_one() needs to optionally wakeup the swapper. Since we hold the thread lock for that entire function, however, we have to push that requirement up into the caller. Found by: rwatson
2008-08-22 16:15:58 +00:00
wakeup_swapper = thread_unsuspend_one(
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
p->p_singlethread, p, false);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
thread_unlock(p->p_singlethread);
A suspended thread can, in fact, be swapped out. Thus, thread_unsuspend_one() needs to optionally wakeup the swapper. Since we hold the thread lock for that entire function, however, we have to push that requirement up into the caller. Found by: rwatson
2008-08-22 16:15:58 +00:00
if (wakeup_swapper)
kick_proc0();
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
}
Use UMA as a complex object allocator. The process allocator now caches and hands out complete process structures *including substructures* . i.e. it get's the process structure with the first thread (and soon KSE) already allocated and attached, all in one hit. For the average non threaded program (non KSE that is) the allocated thread and its stack remain attached to the process, even when the process is unused and in the process cache. This saves having to allocate and attach it later, effectively bringing us (hopefully) close to the efficiency of pre-KSE systems where these were a single structure. Reviewed by: davidxu@freebsd.org, peter@freebsd.org
2002-09-06 07:00:37 +00:00
}
Round out the facilty for a 'bound' thread to loan out its KSE in specific situations. The owner thread must be blocked, and the borrower can not proceed back to user space with the borrowed KSE. The borrower will return the KSE on the next context switch where teh owner wants it back. This removes a lot of possible race conditions and deadlocks. It is consceivable that the borrower should inherit the priority of the owner too. that's another discussion and would be simple to do. Also, as part of this, the "preallocatd spare thread" is attached to the thread doing a syscall rather than the KSE. This removes the need to lock the scheduler when we want to access it, as it's now "at hand". DDB now shows a lot mor info for threaded proceses though it may need some optimisation to squeeze it all back into 80 chars again. (possible JKH project) Upcalls are now "bound" threads, but "KSE Lending" now means that other completing syscalls can be completed using that KSE before the upcall finally makes it back to the UTS. (getting threads OUT OF THE KERNEL is one of the highest priorities in the KSE system.) The upcall when it happens will present all the completed syscalls to the KSE for selection.
2002-10-09 02:33:36 +00:00
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
PCPU_SET(deadthread, td);
} else {
/*
* The last thread is exiting.. but not through exit()
*/
panic ("thread_exit: Last thread exiting on its own");
Let SA process work under ULE scheduler, originally it would panic kernel. Reviewed by: jeff
2003-08-26 11:33:15 +00:00
}
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
}
Inform hwpmc(4) of a thread's impending demise prior to invoking sched_throw(). Debugging help: fabient Review and testing by: fabient
2009-10-25 04:34:47 +00:00
#ifdef HWPMC_HOOKS
/*
* If this thread is part of a process that is being tracked by hwpmc(4),
* inform the module of the thread's impending exit.
*/
if (PMC_PROC_IS_USING_PMCS(td->td_proc))
PMC_SWITCH_CONTEXT(td, PMC_FN_CSW_OUT);
#endif
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
PROC_UNLOCK(p);
The process spin lock currently has the following distinct uses: - Threads lifetime cycle, in particular, counting of the threads in the process, and interlocking with process mutex and thread lock. The main reason of this is that turnstile locks are after thread locks, so you e.g. cannot unlock blockable mutex (think process mutex) while owning thread lock. - Virtual and profiling itimers, since the timers activation is done from the clock interrupt context. Replace the p_slock by p_itimmtx and PROC_ITIMLOCK(). - Profiling code (profil(2)), for similar reason. Replace the p_slock by p_profmtx and PROC_PROFLOCK(). - Resource usage accounting. Need for the spinlock there is subtle, my understanding is that spinlock blocks context switching for the current thread, which prevents td_runtime and similar fields from changing (updates are done at the mi_switch()). Replace the p_slock by p_statmtx and PROC_STATLOCK(). The split is done mostly for code clarity, and should not affect scalability. Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-11-26 14:10:00 +00:00
PROC_STATLOCK(p);
thread_lock(td);
PROC_SUNLOCK(p);
Some small fixes to CPU accounting for threads: - Only initialize the per-cpu switchticks and switchtime in sched_throw() for the very first context switch on APs during boot. This avoids a small gap between the middle of thread_exit() and sched_throw() where time is not accounted to any thread. - In thread_exit(), update the timestamp bookkeeping to track the changes to mi_switch() introduced by td_rux so that the code once again matches the comment claiming it is mimicing mi_switch(). Specifically, only update the per-thread stats directly and depend on ruxagg() to update p_rux rather than adjusting p_rux directly. While here, move the timestamp bookkeeping as late in the function as possible. Reviewed by: bde, kib MFC after: 1 week
2012-01-03 21:03:28 +00:00
/* Do the same timestamp bookkeeping that mi_switch() would do. */
new_switchtime = cpu_ticks();
runtime = new_switchtime - PCPU_GET(switchtime);
td->td_runtime += runtime;
td->td_incruntime += runtime;
PCPU_SET(switchtime, new_switchtime);
PCPU_SET(switchticks, ticks);
- Remove 'struct vmmeter' from 'struct pcpu', leaving only global vmmeter in place. To do per-cpu stats, convert all fields that previously were maintained in the vmmeters that sit in pcpus to counter(9). - Since some vmmeter stats may be touched at very early stages of boot, before we have set up UMA and we can do counter_u64_alloc(), provide an early counter mechanism: o Leave one spare uint64_t in struct pcpu, named pc_early_dummy_counter. o Point counter(9) fields of vmmeter to pcpu[0].pc_early_dummy_counter, so that at early stages of boot, before counters are allocated we already point to a counter that can be safely written to. o For sparc64 that required a whole dummy pcpu[MAXCPU] array. Further related changes: - Don't include vmmeter.h into pcpu.h. - vm.stats.vm.v_swappgsout and vm.stats.vm.v_swappgsin changed to 64-bit, to match kernel representation. - struct vmmeter hidden under _KERNEL, and only vmstat(1) is an exclusion. This is based on benno@'s 4-year old patch: https://lists.freebsd.org/pipermail/freebsd-arch/2013-July/014471.html Reviewed by: kib, gallatin, marius, lidl Differential Revision: https://reviews.freebsd.org/D10156
2017-04-17 17:34:47 +00:00
VM_CNT_INC(v_swtch);
Some small fixes to CPU accounting for threads: - Only initialize the per-cpu switchticks and switchtime in sched_throw() for the very first context switch on APs during boot. This avoids a small gap between the middle of thread_exit() and sched_throw() where time is not accounted to any thread. - In thread_exit(), update the timestamp bookkeeping to track the changes to mi_switch() introduced by td_rux so that the code once again matches the comment claiming it is mimicing mi_switch(). Specifically, only update the per-thread stats directly and depend on ruxagg() to update p_rux rather than adjusting p_rux directly. While here, move the timestamp bookkeeping as late in the function as possible. Reviewed by: bde, kib MFC after: 1 week
2012-01-03 21:03:28 +00:00
/* Save our resource usage in our process. */
td->td_ru.ru_nvcsw++;
Fix the double counting of the last process thread td_incruntime on exit, that is done once in thread_exit() and the second time in proc_reap(), by clearing td_incruntime. Use the opportunity to revert to the pre-RUSAGE_THREAD exporting of ruxagg() instead of ruxagg_locked() and use it from thread_exit(). Diagnosed and tested by: neel MFC after: 3 days
2010-05-24 10:23:49 +00:00
ruxagg(p, td);
Some small fixes to CPU accounting for threads: - Only initialize the per-cpu switchticks and switchtime in sched_throw() for the very first context switch on APs during boot. This avoids a small gap between the middle of thread_exit() and sched_throw() where time is not accounted to any thread. - In thread_exit(), update the timestamp bookkeeping to track the changes to mi_switch() introduced by td_rux so that the code once again matches the comment claiming it is mimicing mi_switch(). Specifically, only update the per-thread stats directly and depend on ruxagg() to update p_rux rather than adjusting p_rux directly. While here, move the timestamp bookkeeping as late in the function as possible. Reviewed by: bde, kib MFC after: 1 week
2012-01-03 21:03:28 +00:00
rucollect(&p->p_ru, &td->td_ru);
The process spin lock currently has the following distinct uses: - Threads lifetime cycle, in particular, counting of the threads in the process, and interlocking with process mutex and thread lock. The main reason of this is that turnstile locks are after thread locks, so you e.g. cannot unlock blockable mutex (think process mutex) while owning thread lock. - Virtual and profiling itimers, since the timers activation is done from the clock interrupt context. Replace the p_slock by p_itimmtx and PROC_ITIMLOCK(). - Profiling code (profil(2)), for similar reason. Replace the p_slock by p_profmtx and PROC_PROFLOCK(). - Resource usage accounting. Need for the spinlock there is subtle, my understanding is that spinlock blocks context switching for the current thread, which prevents td_runtime and similar fields from changing (updates are done at the mi_switch()). Replace the p_slock by p_statmtx and PROC_STATLOCK(). The split is done mostly for code clarity, and should not affect scalability. Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-11-26 14:10:00 +00:00
PROC_STATUNLOCK(p);
Some small fixes to CPU accounting for threads: - Only initialize the per-cpu switchticks and switchtime in sched_throw() for the very first context switch on APs during boot. This avoids a small gap between the middle of thread_exit() and sched_throw() where time is not accounted to any thread. - In thread_exit(), update the timestamp bookkeeping to track the changes to mi_switch() introduced by td_rux so that the code once again matches the comment claiming it is mimicing mi_switch(). Specifically, only update the per-thread stats directly and depend on ruxagg() to update p_rux rather than adjusting p_rux directly. While here, move the timestamp bookkeeping as late in the function as possible. Reviewed by: bde, kib MFC after: 1 week
2012-01-03 21:03:28 +00:00
Increase the amount of data exported by KTR in the KTR_RUNQ setting. This extra data is needed to really follow what is going on in the threaded case.
2004-08-09 18:21:12 +00:00
td->td_state = TDS_INACTIVE;
Introduce some WITNESS improvements: - Speedup the lock orderings lookup modifying the witness graph from a linked tree to a matrix. A table lookup caches the lock orderings in order to make a O(1) access for them. Any witness object has an unique index withing this lookup cache table. - Reduce the lock contention on w_mtx acquiring it only when the LOR actually happens and not in a sane case. In order to do this don't totally flush lock lists (per-CPU spinlocks list and per-thread sleeplocks list) but check for ll_count anytime we need to have to verify allocations sanity. - Introduce the function witness_thread_exit() in the witness namespace which should verify a thread doesn't hold any witness occurrence why exiting. - Rename the sysctl debug.witness.graphs into debug.witness.fullgraph and add debug.witness.badstacks which prints out stacks for LOR revealed. This is implemented using the stack(9) support, which makes WITNESS to be dependent by the STACK option or by the DDB (including STACK) option. - Fix style(9) for src/sys/kern/subr_witness.c The hash table approach has been developed by Ilya Maykov on the behalf of Isilon Systems which kindly released the patch. Jeff Roberson, ported the patch to -CURRENT and fixed w_mtx contention, on the behalf of Nokia. Submitted by: Ilya Maykov <ivmaykov at gmail dot com> (Isilon Systems), jeff Sponsored by: Nokia
2008-08-13 18:24:22 +00:00
#ifdef WITNESS
witness_thread_exit(td);
#endif
Increase the amount of data exported by KTR in the KTR_RUNQ setting. This extra data is needed to really follow what is going on in the threaded case.
2004-08-09 18:21:12 +00:00
CTR1(KTR_PROC, "thread_exit: cpu_throw() thread %p", td);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
sched_throw(td);
Commit a partial lazy thread switch mechanism for i386. it isn't as lazy as it could be and can do with some more cleanup. Currently its under options LAZY_SWITCH. What this does is avoid %cr3 reloads for short context switches that do not involve another user process. ie: we can take an interrupt, switch to a kthread and return to the user without explicitly flushing the tlb. However, this isn't as exciting as it could be, the interrupt overhead is still high and too much blocks on Giant still. There are some debug sysctls, for stats and for an on/off switch. The main problem with doing this has been "what if the process that you're running on exits while we're borrowing its address space?" - in this case we use an IPI to give it a kick when we're about to reclaim the pmap. Its not compiled in unless you add the LAZY_SWITCH option. I want to fix a few more things and get some more feedback before turning it on by default. This is NOT a replacement for Bosko's lazy interrupt stuff. This was more meant for the kthread case, while his was for interrupts. Mine helps a little for interrupts, but his helps a lot more. The stats are enabled with options SWTCH_OPTIM_STATS - this has been a pseudo-option for years, I just added a bunch of stuff to it. One non-trivial change was to select a new thread before calling cpu_switch() in the first place. This allows us to catch the silly case of doing a cpu_switch() to the current process. This happens uncomfortably often. This simplifies a bit of the asm code in cpu_switch (no longer have to call choosethread() in the middle). This has been implemented on i386 and (thanks to jake) sparc64. The others will come soon. This is actually seperate to the lazy switch stuff. Glanced at by: jake, jhb
2003-04-02 23:53:30 +00:00
panic("I'm a teapot!");
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/* NOTREACHED */
}
s/Muliple/Multiple Removed whitespace at EOL and EOF.
2004-01-10 18:34:01 +00:00
/*
Unbreak the KSE code. Keep track of zobie threads using the Per-CPU storage during the context switch. Rearrange thread cleanups to avoid problems with Giant. Clean threads when freed or when recycled. Approved by: re (jhb)
2002-12-10 02:33:45 +00:00
* Do any thread specific cleanups that may be needed in wait()
Push Giant down a little further: - no longer serialize on Giant for thread_single*() and family in fork, exit and exec - thread_wait() is mpsafe, assert no Giant - reduce scope of Giant in exit to not cover thread_wait and just do vm_waitproc(). - assert that thread_single() family are not called with Giant - remove the DROP/PICKUP_GIANT macros from thread_single() family - assert that thread_suspend_check() s not called with Giant - remove manual drop_giant hack in thread_suspend_check since we know it isn't held. - remove the DROP/PICKUP_GIANT macros from thread_suspend_check() family - mark kse_create() mpsafe
2004-03-13 22:31:39 +00:00
* called with Giant, proc and schedlock not held.
Unbreak the KSE code. Keep track of zobie threads using the Per-CPU storage during the context switch. Rearrange thread cleanups to avoid problems with Giant. Clean threads when freed or when recycled. Approved by: re (jhb)
2002-12-10 02:33:45 +00:00
*/
void
thread_wait(struct proc *p)
{
struct thread *td;
Push Giant down a little further: - no longer serialize on Giant for thread_single*() and family in fork, exit and exec - thread_wait() is mpsafe, assert no Giant - reduce scope of Giant in exit to not cover thread_wait and just do vm_waitproc(). - assert that thread_single() family are not called with Giant - remove the DROP/PICKUP_GIANT macros from thread_single() family - assert that thread_suspend_check() s not called with Giant - remove manual drop_giant hack in thread_suspend_check since we know it isn't held. - remove the DROP/PICKUP_GIANT macros from thread_suspend_check() family - mark kse_create() mpsafe
2004-03-13 22:31:39 +00:00
mtx_assert(&Giant, MA_NOTOWNED);
Style. Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-09-03 08:40:16 +00:00
KASSERT(p->p_numthreads == 1, ("multiple threads in thread_wait()"));
KASSERT(p->p_exitthreads == 0, ("p_exitthreads leaking"));
Solve a complex exit race introduced with thread_lock: - Add a count of exiting threads, p_exitthreads, to struct proc. - Increment p_exithreads when we set the deadthread in thread_exit(). - When we thread_stash() a deadthread use an atomic to drop the count. - Spin until the p_exithreads count reaches 0 in thread_wait(). - Lock the last exiting thread momentarily to be certain that it has exited cpu_throw(). - Restructure thread_wait(). It does not need a loop as there will only ever be one thread. Tested by: moose@opera.com Reported by: kris, moose@opera.com
2007-06-12 07:24:46 +00:00
td = FIRST_THREAD_IN_PROC(p);
/* Lock the last thread so we spin until it exits cpu_throw(). */
thread_lock(td);
thread_unlock(td);
- Implement a new mechanism for resetting lock profiling. We now guarantee that all cpus have acknowledged the cleared enable int by scheduling the resetting thread on each cpu in succession. Since all lock profiling happens within a critical section this guarantees that all cpus have left lock profiling before we clear the datastructures. - Assert that the per-thread queue of locks lock profiling is aware of is clear on thread exit. There were several cases where this was not true that slows lock profiling and leaks information. - Remove all objects from all lists before clearing any per-cpu information in reset. Lock profiling objects can migrate between per-cpu caches and previously these migrated objects could be zero'd before they'd been removed Discussed with: attilio Sponsored by: Nokia
2009-03-15 06:41:47 +00:00
lock_profile_thread_exit(td);
Add cpuset, an api for thread to cpu binding and cpu resource grouping and assignment. - Add a reference to a struct cpuset in each thread that is inherited from the thread that created it. - Release the reference when the thread is destroyed. - Add prototypes for syscalls and macros for manipulating cpusets in sys/cpuset.h - Add syscalls to create, get, and set new numbered cpusets: cpuset(), cpuset_{get,set}id() - Add syscalls for getting and setting affinity masks for cpusets or individual threads: cpuid_{get,set}affinity() - Add types for the 'level' and 'which' parameters for the cpuset. This will permit expansion of the api to cover cpu masks for other objects identifiable with an id_t integer. For example, IRQs and Jails may be coming soon. - The root set 0 contains all valid cpus. All thread initially belong to cpuset 1. This permits migrating all threads off of certain cpus to reserve them for special applications. Sponsored by: Nokia Discussed with: arch, rwatson, brooks, davidxu, deischen Reviewed by: antoine
2008-03-02 07:39:22 +00:00
cpuset_rel(td->td_cpuset);
td->td_cpuset = NULL;
Solve a complex exit race introduced with thread_lock: - Add a count of exiting threads, p_exitthreads, to struct proc. - Increment p_exithreads when we set the deadthread in thread_exit(). - When we thread_stash() a deadthread use an atomic to drop the count. - Spin until the p_exithreads count reaches 0 in thread_wait(). - Lock the last exiting thread momentarily to be certain that it has exited cpu_throw(). - Restructure thread_wait(). It does not need a loop as there will only ever be one thread. Tested by: moose@opera.com Reported by: kris, moose@opera.com
2007-06-12 07:24:46 +00:00
cpu_thread_clean(td);
Generalised support for copy-on-write structures shared by threads. Thread credentials are maintained as follows: each thread has a pointer to creds and a reference on them. The pointer is compared with proc's creds on userspace<->kernel boundary and updated if needed. This patch introduces a counter which can be compared instead, so that more structures can use this scheme without adding more comparisons on the boundary.
2015-06-10 10:43:59 +00:00
thread_cow_free(td);
Rewrite subr_sleepqueue.c use of callouts to not depend on the specifics of callout KPI. Esp., do not depend on the exact interface of callout_stop(9) return values. The main change is that instead of requiring precise callouts, code maintains absolute time to wake up. Callouts now should ensure that a wake occurs at the requested moment, but we can tolerate both run-away callout, and callout_stop(9) lying about running callout either way. As consequence, it removes the constant source of the bugs where sleepq_check_timeout() causes uninterruptible thread state where the thread is detached from CPU, see e.g. r234952 and r296320. Patch also removes dual meaning of the TDF_TIMEOUT flag, making code (IMO much) simpler to reason about. Tested by: pho Reviewed by: jhb Sponsored by: The FreeBSD Foundation MFC after: 1 month Differential revision: https://reviews.freebsd.org/D7137
2016-07-28 09:09:55 +00:00
callout_drain(&td->td_slpcallout);
Unbreak the KSE code. Keep track of zobie threads using the Per-CPU storage during the context switch. Rearrange thread cleanups to avoid problems with Giant. Clean threads when freed or when recycled. Approved by: re (jhb)
2002-12-10 02:33:45 +00:00
thread_reap(); /* check for zombie threads etc. */
}
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
* Link a thread to a process.
Use UMA as a complex object allocator. The process allocator now caches and hands out complete process structures *including substructures* . i.e. it get's the process structure with the first thread (and soon KSE) already allocated and attached, all in one hit. For the average non threaded program (non KSE that is) the allocated thread and its stack remain attached to the process, even when the process is unused and in the process cache. This saves having to allocate and attach it later, effectively bringing us (hopefully) close to the efficiency of pre-KSE systems where these were a single structure. Reviewed by: davidxu@freebsd.org, peter@freebsd.org
2002-09-06 07:00:37 +00:00
* set up anything that needs to be initialized for it to
* be used by the process.
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
*/
void
Make KSE a kernel option, turned on by default in all GENERIC kernel configs except sun4v (which doesn't process signals properly with KSE). Reviewed by: davidxu@
2006-10-26 21:42:22 +00:00
thread_link(struct thread *td, struct proc *p)
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
{
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
/*
* XXX This can't be enabled because it's called for proc0 before
- Relax requirements for p_numthreads, p_threads, p_swtick, and p_nice from requiring the per-process spinlock to only requiring the process lock. - Reflect these changes in the proc.h documentation and consumers throughout the kernel. This is a substantial reduction in locking cost for these fields and was made possible by recent changes to threading support.
2008-03-19 06:19:01 +00:00
* its lock has been created.
* PROC_LOCK_ASSERT(p, MA_OWNED);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
*/
- Split the struct kse into struct upcall and struct kse. struct kse will soon be visible only to schedulers. This greatly simplifies much the KSE code. Submitted by: davidxu
2003-02-17 05:14:26 +00:00
td->td_state = TDS_INACTIVE;
td->td_proc = p;
- Move all of the PS_ flags into either p_flag or td_flags. - p_sflag was mostly protected by PROC_LOCK rather than the PROC_SLOCK or previously the sched_lock. These bugs have existed for some time. - Allow swapout to try each thread in a process individually and then swapin the whole process if any of these fail. This allows us to move most scheduler related swap flags into td_flags. - Keep ki_sflag for backwards compat but change all in source tools to use the new and more correct location of P_INMEM. Reported by: pho Reviewed by: attilio, kib Approved by: re (kensmith)
2007-09-17 05:31:39 +00:00
td->td_flags = TDF_INMEM;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
Use UMA as a complex object allocator. The process allocator now caches and hands out complete process structures *including substructures* . i.e. it get's the process structure with the first thread (and soon KSE) already allocated and attached, all in one hit. For the average non threaded program (non KSE that is) the allocated thread and its stack remain attached to the process, even when the process is unused and in the process cache. This saves having to allocate and attach it later, effectively bringing us (hopefully) close to the efficiency of pre-KSE systems where these were a single structure. Reviewed by: davidxu@freebsd.org, peter@freebsd.org
2002-09-06 07:00:37 +00:00
LIST_INIT(&td->td_contested);
- Re-implement lock profiling in such a way that it no longer breaks the ABI when enabled. There is no longer an embedded lock_profile_object in each lock. Instead a list of lock_profile_objects is kept per-thread for each lock it may own. The cnt_hold statistic is now always 0 to facilitate this. - Support shared locking by tracking individual lock instances and statistics in the per-thread per-instance lock_profile_object. - Make the lock profiling hash table a per-cpu singly linked list with a per-cpu static lock_prof allocator. This removes the need for an array of spinlocks and reduces cache contention between cores. - Use a seperate hash for spinlocks and other locks so that only a critical_enter() is required and not a spinlock_enter() to modify the per-cpu tables. - Count time spent spinning in the lock statistics. - Remove the LOCK_PROFILE_SHARED option as it is always supported now. - Specifically drop and release the scheduler locks in both schedulers since we track owners now. In collaboration with: Kip Macy Sponsored by: Nokia
2007-12-15 23:13:31 +00:00
LIST_INIT(&td->td_lprof[0]);
LIST_INIT(&td->td_lprof[1]);
1. Change prototype of trapsignal and sendsig to use ksiginfo_t *, most changes in MD code are trivial, before this change, trapsignal and sendsig use discrete parameters, now they uses member fields of ksiginfo_t structure. For sendsig, this change allows us to pass POSIX realtime signal value to user code. 2. Remove cpu_thread_siginfo, it is no longer needed because we now always generate ksiginfo_t data and feed it to libpthread. 3. Add p_sigqueue to proc structure to hold shared signals which were blocked by all threads in the proc. 4. Add td_sigqueue to thread structure to hold all signals delivered to thread. 5. i386 and amd64 now return POSIX standard si_code, other arches will be fixed. 6. In this sigqueue implementation, pending signal set is kept as before, an extra siginfo list holds additional siginfo_t data for signals. kernel code uses psignal() still behavior as before, it won't be failed even under memory pressure, only exception is when deleting a signal, we should call sigqueue_delete to remove signal from sigqueue but not SIGDELSET. Current there is no kernel code will deliver a signal with additional data, so kernel should be as stable as before, a ksiginfo can carry more information, for example, allow signal to be delivered but throw away siginfo data if memory is not enough. SIGKILL and SIGSTOP have fast path in sigqueue_add, because they can not be caught or masked. The sigqueue() syscall allows user code to queue a signal to target process, if resource is unavailable, EAGAIN will be returned as specification said. Just before thread exits, signal queue memory will be freed by sigqueue_flush. Current, all signals are allowed to be queued, not only realtime signals. Earlier patch reviewed by: jhb, deischen Tested on: i386, amd64
2005-10-14 12:43:47 +00:00
sigqueue_init(&td->td_sigqueue, p);
CALLOUT_MPSAFE has lost its meaning since r141428, i.e., for more than ten years for head. However, it is continuously misused as the mpsafe argument for callout_init(9). Deprecate the flag and clean up callout_init() calls to make them more consistent. Differential Revision: https://reviews.freebsd.org/D2613 Reviewed by: jhb MFC after: 2 weeks
2015-05-22 17:05:21 +00:00
callout_init(&td->td_slpcallout, 1);
Insert new threads at the end of the thread list in the process instead of at the beginning. This allows an intra process signal to be sent to the oldest thread with the signal unmasked - which, if it still exists, is the main thread. This mimics behavior found in Linux and Solaris.
2014-07-25 20:21:02 +00:00
TAILQ_INSERT_TAIL(&p->p_threads, td, td_plist);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
p->p_numthreads++;
}
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
/*
* Called from:
* thread_exit()
*/
Add a thread_unlink() and use it. It could also be used twice in kern_thr.c but that's owned by jeff so I'l let him change it when he's next there.
2003-04-18 00:16:13 +00:00
void
thread_unlink(struct thread *td)
s/Muliple/Multiple Removed whitespace at EOL and EOF.
2004-01-10 18:34:01 +00:00
{
Add a thread_unlink() and use it. It could also be used twice in kern_thr.c but that's owned by jeff so I'l let him change it when he's next there.
2003-04-18 00:16:13 +00:00
struct proc *p = td->td_proc;
- Protect p_numthreads with the sched_lock. - Protect p_singlethread with both the sched_lock and the proc lock. - Protect p_suspcount with the proc lock.
2003-04-23 18:46:51 +00:00
- Relax requirements for p_numthreads, p_threads, p_swtick, and p_nice from requiring the per-process spinlock to only requiring the process lock. - Reflect these changes in the proc.h documentation and consumers throughout the kernel. This is a substantial reduction in locking cost for these fields and was made possible by recent changes to threading support.
2008-03-19 06:19:01 +00:00
PROC_LOCK_ASSERT(p, MA_OWNED);
Add a thread_unlink() and use it. It could also be used twice in kern_thr.c but that's owned by jeff so I'l let him change it when he's next there.
2003-04-18 00:16:13 +00:00
TAILQ_REMOVE(&p->p_threads, td, td_plist);
p->p_numthreads--;
/* could clear a few other things here */
Make KSE a kernel option, turned on by default in all GENERIC kernel configs except sun4v (which doesn't process signals properly with KSE). Reviewed by: davidxu@
2006-10-26 21:42:22 +00:00
/* Must NOT clear links to proc! */
Move thread related code from kern_proc.c to kern_thread.c. Add code to free KSEs and KSEGRPs on exit. Sort KSE prototypes in proc.h. Add the missing kse_exit() syscall. ksetest now does not leak KSEs and KSEGRPS. Submitted by: (parts) davidxu
2002-10-24 08:46:34 +00:00
}
Move the repeated code to calculate the number of the threads in the process that still need to be suspended or exited from thread_single into the new function calc_remaining(). Tested by: pho Reviewed by: jhb Approved by: re (kensmith)
2009-07-14 22:51:31 +00:00
static int
calc_remaining(struct proc *p, int mode)
{
int remaining;
Consistently use process spin lock for protection of the p->p_boundary_count. Race could cause the execve(2) from the threaded process to hung since thread boundary counter was incorrect and single-threading never finished. Reported by: pluknet, pho Tested by: pho MFC after: 1 week
2011-11-18 09:12:26 +00:00
PROC_LOCK_ASSERT(p, MA_OWNED);
PROC_SLOCK_ASSERT(p, MA_OWNED);
Move the repeated code to calculate the number of the threads in the process that still need to be suspended or exited from thread_single into the new function calc_remaining(). Tested by: pho Reviewed by: jhb Approved by: re (kensmith)
2009-07-14 22:51:31 +00:00
if (mode == SINGLE_EXIT)
remaining = p->p_numthreads;
else if (mode == SINGLE_BOUNDARY)
remaining = p->p_numthreads - p->p_boundary_count;
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
else if (mode == SINGLE_NO_EXIT || mode == SINGLE_ALLPROC)
Move the repeated code to calculate the number of the threads in the process that still need to be suspended or exited from thread_single into the new function calc_remaining(). Tested by: pho Reviewed by: jhb Approved by: re (kensmith)
2009-07-14 22:51:31 +00:00
remaining = p->p_numthreads - p->p_suspcount;
else
panic("calc_remaining: wrong mode %d", mode);
return (remaining);
}
Do some refactoring and minor cleanups of the thread_single() code in preparation for the global stop commit. Move the code to weed suspended or sleeping threads into the appropriate state, into the helper weed_inhib(). Current code already has deep nesting and hard to follow [1]. Add currently useless helper remain_for_mode(), which returns the count of threads which are allowed to run, according to the single-threading mode. In thread_single_end(), do not save curthread into local variable, it is unused after, except to find curproc. Remove stray empty line. Requested by: avg [1] Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-12-08 16:27:43 +00:00
static int
remain_for_mode(int mode)
{
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
return (mode == SINGLE_ALLPROC ? 0 : 1);
Do some refactoring and minor cleanups of the thread_single() code in preparation for the global stop commit. Move the code to weed suspended or sleeping threads into the appropriate state, into the helper weed_inhib(). Current code already has deep nesting and hard to follow [1]. Add currently useless helper remain_for_mode(), which returns the count of threads which are allowed to run, according to the single-threading mode. In thread_single_end(), do not save curthread into local variable, it is unused after, except to find curproc. Remove stray empty line. Requested by: avg [1] Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-12-08 16:27:43 +00:00
}
static int
weed_inhib(int mode, struct thread *td2, struct proc *p)
{
int wakeup_swapper;
PROC_LOCK_ASSERT(p, MA_OWNED);
PROC_SLOCK_ASSERT(p, MA_OWNED);
THREAD_LOCK_ASSERT(td2, MA_OWNED);
wakeup_swapper = 0;
switch (mode) {
case SINGLE_EXIT:
if (TD_IS_SUSPENDED(td2))
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
wakeup_swapper |= thread_unsuspend_one(td2, p, true);
Do some refactoring and minor cleanups of the thread_single() code in preparation for the global stop commit. Move the code to weed suspended or sleeping threads into the appropriate state, into the helper weed_inhib(). Current code already has deep nesting and hard to follow [1]. Add currently useless helper remain_for_mode(), which returns the count of threads which are allowed to run, according to the single-threading mode. In thread_single_end(), do not save curthread into local variable, it is unused after, except to find curproc. Remove stray empty line. Requested by: avg [1] Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-12-08 16:27:43 +00:00
if (TD_ON_SLEEPQ(td2) && (td2->td_flags & TDF_SINTR) != 0)
wakeup_swapper |= sleepq_abort(td2, EINTR);
break;
case SINGLE_BOUNDARY:
case SINGLE_NO_EXIT:
if (TD_IS_SUSPENDED(td2) && (td2->td_flags & TDF_BOUNDARY) == 0)
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
wakeup_swapper |= thread_unsuspend_one(td2, p, false);
Do some refactoring and minor cleanups of the thread_single() code in preparation for the global stop commit. Move the code to weed suspended or sleeping threads into the appropriate state, into the helper weed_inhib(). Current code already has deep nesting and hard to follow [1]. Add currently useless helper remain_for_mode(), which returns the count of threads which are allowed to run, according to the single-threading mode. In thread_single_end(), do not save curthread into local variable, it is unused after, except to find curproc. Remove stray empty line. Requested by: avg [1] Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-12-08 16:27:43 +00:00
if (TD_ON_SLEEPQ(td2) && (td2->td_flags & TDF_SINTR) != 0)
wakeup_swapper |= sleepq_abort(td2, ERESTART);
Add missed break. CID: 1258586 Sponsored by: The FreeBSD Foundation MFC after: 4 days
2014-12-16 09:48:23 +00:00
break;
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
case SINGLE_ALLPROC:
/*
* ALLPROC suspend tries to avoid spurious EINTR for
* threads sleeping interruptable, by suspending the
* thread directly, similarly to sig_suspend_threads().
* Since such sleep is not performed at the user
* boundary, TDF_BOUNDARY flag is not set, and TDF_ALLPROCSUSP
* is used to avoid immediate un-suspend.
*/
if (TD_IS_SUSPENDED(td2) && (td2->td_flags & (TDF_BOUNDARY |
TDF_ALLPROCSUSP)) == 0)
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
wakeup_swapper |= thread_unsuspend_one(td2, p, false);
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
if (TD_ON_SLEEPQ(td2) && (td2->td_flags & TDF_SINTR) != 0) {
if ((td2->td_flags & TDF_SBDRY) == 0) {
thread_suspend_one(td2);
td2->td_flags |= TDF_ALLPROCSUSP;
} else {
wakeup_swapper |= sleepq_abort(td2, ERESTART);
}
}
Do some refactoring and minor cleanups of the thread_single() code in preparation for the global stop commit. Move the code to weed suspended or sleeping threads into the appropriate state, into the helper weed_inhib(). Current code already has deep nesting and hard to follow [1]. Add currently useless helper remain_for_mode(), which returns the count of threads which are allowed to run, according to the single-threading mode. In thread_single_end(), do not save curthread into local variable, it is unused after, except to find curproc. Remove stray empty line. Requested by: avg [1] Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-12-08 16:27:43 +00:00
break;
}
return (wakeup_swapper);
}
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
* Enforce single-threading.
*
* Returns 1 if the caller must abort (another thread is waiting to
* exit the process or similar). Process is locked!
* Returns 0 when you are successfully the only thread running.
* A process has successfully single threaded in the suspend mode when
* There are no threads in user mode. Threads in the kernel must be
* allowed to continue until they get to the user boundary. They may even
* copy out their return values and data before suspending. They may however be
o Fix typo in the comment. PR: kern/99632 Submitted by: clsung
2006-06-30 08:10:55 +00:00
* accelerated in reaching the user boundary as we will wake up
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
* any sleeping threads that are interruptable. (PCATCH).
*/
int
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
thread_single(struct proc *p, int mode)
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
{
struct thread *td;
struct thread *td2;
If a thread that is swapped out is made runnable, then the setrunnable() routine wakes up proc0 so that proc0 can swap the thread back in. Historically, this has been done by waking up proc0 directly from setrunnable() itself via a wakeup(). When waking up a sleeping thread that was swapped out (the usual case when waking proc0 since only sleeping threads are eligible to be swapped out), this resulted in a bit of recursion (e.g. wakeup() -> setrunnable() -> wakeup()). With sleep queues having separate locks in 6.x and later, this caused a spin lock LOR (sleepq lock -> sched_lock/thread lock -> sleepq lock). An attempt was made to fix this in 7.0 by making the proc0 wakeup use the ithread mechanism for doing the wakeup. However, this required grabbing proc0's thread lock to perform the wakeup. If proc0 was asleep elsewhere in the kernel (e.g. waiting for disk I/O), then this degenerated into the same LOR since the thread lock would be some other sleepq lock. Fix this by deferring the wakeup of the swapper until after the sleepq lock held by the upper layer has been locked. The setrunnable() routine now returns a boolean value to indicate whether or not proc0 needs to be woken up. The end result is that consumers of the sleepq API such as *sleep/wakeup, condition variables, sx locks, and lockmgr, have to wakeup proc0 if they get a non-zero return value from sleepq_abort(), sleepq_broadcast(), or sleepq_signal(). Discussed with: jeff Glanced at by: sam Tested by: Jurgen Weber jurgen - ish com au MFC after: 2 weeks
2008-08-05 20:02:31 +00:00
int remaining, wakeup_swapper;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
td = curthread;
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
KASSERT(mode == SINGLE_EXIT || mode == SINGLE_BOUNDARY ||
mode == SINGLE_ALLPROC || mode == SINGLE_NO_EXIT,
("invalid mode %d", mode));
/*
* If allowing non-ALLPROC singlethreading for non-curproc
* callers, calc_remaining() and remain_for_mode() should be
* adjusted to also account for td->td_proc != p. For now
* this is not implemented because it is not used.
*/
KASSERT((mode == SINGLE_ALLPROC && td->td_proc != p) ||
(mode != SINGLE_ALLPROC && td->td_proc == p),
("mode %d proc %p curproc %p", mode, p, td->td_proc));
Push Giant down a little further: - no longer serialize on Giant for thread_single*() and family in fork, exit and exec - thread_wait() is mpsafe, assert no Giant - reduce scope of Giant in exit to not cover thread_wait and just do vm_waitproc(). - assert that thread_single() family are not called with Giant - remove the DROP/PICKUP_GIANT macros from thread_single() family - assert that thread_suspend_check() s not called with Giant - remove manual drop_giant hack in thread_suspend_check since we know it isn't held. - remove the DROP/PICKUP_GIANT macros from thread_suspend_check() family - mark kse_create() mpsafe
2004-03-13 22:31:39 +00:00
mtx_assert(&Giant, MA_NOTOWNED);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
PROC_LOCK_ASSERT(p, MA_OWNED);
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
if ((p->p_flag & P_HADTHREADS) == 0 && mode != SINGLE_ALLPROC)
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
return (0);
fix some style problems and remove a mis-merged assert.
2002-07-25 00:27:39 +00:00
/* Is someone already single threading? */
In original kern_execve() code, at the start of the function, it forces all other threads to suicide, problem is execve() could be failed, and a failed execve() would change threaded process to unthreaded, this side effect is unexpected. The new code introduces a new single threading mode SINGLE_BOUNDARY, in the mode, all threads should suspend themself at user boundary except the singler. we can not use SINGLE_NO_EXIT because we want to start from a clean state if execve() is successful, suspending other threads at unknown point and later resuming them from there and forcing them to exit at user boundary may cause the process to start from a dirty state. If execve() is successful, current thread upgrades to SINGLE_EXIT mode and forces other threads to suicide at user boundary, otherwise, other threads will be resumed and their interrupted syscall will be restarted. Reviewed by: julian
2004-10-06 00:40:41 +00:00
if (p->p_singlethread != NULL && p->p_singlethread != td)
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
return (1);
In original kern_execve() code, at the start of the function, it forces all other threads to suicide, problem is execve() could be failed, and a failed execve() would change threaded process to unthreaded, this side effect is unexpected. The new code introduces a new single threading mode SINGLE_BOUNDARY, in the mode, all threads should suspend themself at user boundary except the singler. we can not use SINGLE_NO_EXIT because we want to start from a clean state if execve() is successful, suspending other threads at unknown point and later resuming them from there and forcing them to exit at user boundary may cause the process to start from a dirty state. If execve() is successful, current thread upgrades to SINGLE_EXIT mode and forces other threads to suicide at user boundary, otherwise, other threads will be resumed and their interrupted syscall will be restarted. Reviewed by: julian
2004-10-06 00:40:41 +00:00
if (mode == SINGLE_EXIT) {
p->p_flag |= P_SINGLE_EXIT;
p->p_flag &= ~P_SINGLE_BOUNDARY;
} else {
p->p_flag &= ~P_SINGLE_EXIT;
if (mode == SINGLE_BOUNDARY)
p->p_flag |= P_SINGLE_BOUNDARY;
else
p->p_flag &= ~P_SINGLE_BOUNDARY;
}
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
if (mode == SINGLE_ALLPROC)
p->p_flag |= P_TOTAL_STOP;
s/SGNL/SIG/ s/SNGL/SINGLE/ s/SNGLE/SINGLE/ Fix abbreviation for P_STOPPED_* etc flags, in original code they were inconsistent and difficult to distinguish between them. Approved by: julian (mentor)
2002-09-05 07:30:18 +00:00
p->p_flag |= P_STOPPED_SINGLE;
Revert rev 184216 and 184199, due to the way the thread_lock works, it may cause a lockup. Noticed by: peter, jhb
2008-11-05 03:01:23 +00:00
PROC_SLOCK(p);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
p->p_singlethread = td;
Move the repeated code to calculate the number of the threads in the process that still need to be suspended or exited from thread_single into the new function calc_remaining(). Tested by: pho Reviewed by: jhb Approved by: re (kensmith)
2009-07-14 22:51:31 +00:00
remaining = calc_remaining(p, mode);
Do some refactoring and minor cleanups of the thread_single() code in preparation for the global stop commit. Move the code to weed suspended or sleeping threads into the appropriate state, into the helper weed_inhib(). Current code already has deep nesting and hard to follow [1]. Add currently useless helper remain_for_mode(), which returns the count of threads which are allowed to run, according to the single-threading mode. In thread_single_end(), do not save curthread into local variable, it is unused after, except to find curproc. Remove stray empty line. Requested by: avg [1] Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-12-08 16:27:43 +00:00
while (remaining != remain_for_mode(mode)) {
Rethink it a bit, if there is a STOP flag, don't bother to resume other threads.
2006-03-21 10:05:15 +00:00
if (P_SHOULDSTOP(p) != P_STOPPED_SINGLE)
goto stopme;
If a thread that is swapped out is made runnable, then the setrunnable() routine wakes up proc0 so that proc0 can swap the thread back in. Historically, this has been done by waking up proc0 directly from setrunnable() itself via a wakeup(). When waking up a sleeping thread that was swapped out (the usual case when waking proc0 since only sleeping threads are eligible to be swapped out), this resulted in a bit of recursion (e.g. wakeup() -> setrunnable() -> wakeup()). With sleep queues having separate locks in 6.x and later, this caused a spin lock LOR (sleepq lock -> sched_lock/thread lock -> sleepq lock). An attempt was made to fix this in 7.0 by making the proc0 wakeup use the ithread mechanism for doing the wakeup. However, this required grabbing proc0's thread lock to perform the wakeup. If proc0 was asleep elsewhere in the kernel (e.g. waiting for disk I/O), then this degenerated into the same LOR since the thread lock would be some other sleepq lock. Fix this by deferring the wakeup of the swapper until after the sleepq lock held by the upper layer has been locked. The setrunnable() routine now returns a boolean value to indicate whether or not proc0 needs to be woken up. The end result is that consumers of the sleepq API such as *sleep/wakeup, condition variables, sx locks, and lockmgr, have to wakeup proc0 if they get a non-zero return value from sleepq_abort(), sleepq_broadcast(), or sleepq_signal(). Discussed with: jeff Glanced at by: sam Tested by: Jurgen Weber jurgen - ish com au MFC after: 2 weeks
2008-08-05 20:02:31 +00:00
wakeup_swapper = 0;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
FOREACH_THREAD_IN_PROC(p, td2) {
if (td2 == td)
continue;
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
thread_lock(td2);
- Add a new td flag TDF_NEEDSUSPCHK that is set whenever a thread needs to enter thread_suspend_check(). - Set TDF_ASTPENDING along with TDF_NEEDSUSPCHK so we can move the thread_suspend_check() to ast() rather than userret(). - Check TDF_NEEDSUSPCHK in the sleepq_catch_signals() optimization so that we don't miss a suspend request. If this is set use the expensive signal path. - Set NEEDSUSPCHK when creating a new thread in thr in case the creating thread is due to be suspended as well but has not yet. Reviewed by: davidxu (Authored original patch)
2008-03-21 08:23:25 +00:00
td2->td_flags |= TDF_ASTPENDING | TDF_NEEDSUSPCHK;
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
if (TD_IS_INHIBITED(td2)) {
Do some refactoring and minor cleanups of the thread_single() code in preparation for the global stop commit. Move the code to weed suspended or sleeping threads into the appropriate state, into the helper weed_inhib(). Current code already has deep nesting and hard to follow [1]. Add currently useless helper remain_for_mode(), which returns the count of threads which are allowed to run, according to the single-threading mode. In thread_single_end(), do not save curthread into local variable, it is unused after, except to find curproc. Remove stray empty line. Requested by: avg [1] Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-12-08 16:27:43 +00:00
wakeup_swapper |= weed_inhib(mode, td2, p);
In order to speed up process suspension on MP machine, send IPI to remote CPU. While here, abstract thread suspension code into a function called sig_suspend_threads, the function is called when a process received a STOP signal.
2006-02-13 03:16:55 +00:00
#ifdef SMP
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
} else if (TD_IS_RUNNING(td2) && td != td2) {
In order to speed up process suspension on MP machine, send IPI to remote CPU. While here, abstract thread suspension code into a function called sig_suspend_threads, the function is called when a process received a STOP signal.
2006-02-13 03:16:55 +00:00
forward_signal(td2);
#endif
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
}
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
thread_unlock(td2);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
If a thread that is swapped out is made runnable, then the setrunnable() routine wakes up proc0 so that proc0 can swap the thread back in. Historically, this has been done by waking up proc0 directly from setrunnable() itself via a wakeup(). When waking up a sleeping thread that was swapped out (the usual case when waking proc0 since only sleeping threads are eligible to be swapped out), this resulted in a bit of recursion (e.g. wakeup() -> setrunnable() -> wakeup()). With sleep queues having separate locks in 6.x and later, this caused a spin lock LOR (sleepq lock -> sched_lock/thread lock -> sleepq lock). An attempt was made to fix this in 7.0 by making the proc0 wakeup use the ithread mechanism for doing the wakeup. However, this required grabbing proc0's thread lock to perform the wakeup. If proc0 was asleep elsewhere in the kernel (e.g. waiting for disk I/O), then this degenerated into the same LOR since the thread lock would be some other sleepq lock. Fix this by deferring the wakeup of the swapper until after the sleepq lock held by the upper layer has been locked. The setrunnable() routine now returns a boolean value to indicate whether or not proc0 needs to be woken up. The end result is that consumers of the sleepq API such as *sleep/wakeup, condition variables, sx locks, and lockmgr, have to wakeup proc0 if they get a non-zero return value from sleepq_abort(), sleepq_broadcast(), or sleepq_signal(). Discussed with: jeff Glanced at by: sam Tested by: Jurgen Weber jurgen - ish com au MFC after: 2 weeks
2008-08-05 20:02:31 +00:00
if (wakeup_swapper)
kick_proc0();
Move the repeated code to calculate the number of the threads in the process that still need to be suspended or exited from thread_single into the new function calc_remaining(). Tested by: pho Reviewed by: jhb Approved by: re (kensmith)
2009-07-14 22:51:31 +00:00
remaining = calc_remaining(p, mode);
If thread singler wants to terminate other threads, make sure it includes all threads except itself. Obtained from: julian
2004-06-18 06:15:21 +00:00
s/Muliple/Multiple Removed whitespace at EOL and EOF.
2004-01-10 18:34:01 +00:00
/*
* Maybe we suspended some threads.. was it enough?
More work on the interaction between suspending and sleeping threads. Also clean up some code used with 'single-threading'. Reviewed by: davidxu
2002-10-25 07:11:12 +00:00
*/
Do some refactoring and minor cleanups of the thread_single() code in preparation for the global stop commit. Move the code to weed suspended or sleeping threads into the appropriate state, into the helper weed_inhib(). Current code already has deep nesting and hard to follow [1]. Add currently useless helper remain_for_mode(), which returns the count of threads which are allowed to run, according to the single-threading mode. In thread_single_end(), do not save curthread into local variable, it is unused after, except to find curproc. Remove stray empty line. Requested by: avg [1] Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-12-08 16:27:43 +00:00
if (remaining == remain_for_mode(mode))
More work on the interaction between suspending and sleeping threads. Also clean up some code used with 'single-threading'. Reviewed by: davidxu
2002-10-25 07:11:12 +00:00
break;
Rethink it a bit, if there is a STOP flag, don't bother to resume other threads.
2006-03-21 10:05:15 +00:00
stopme:
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
* Wake us up when everyone else has suspended.
fix some style problems and remove a mis-merged assert.
2002-07-25 00:27:39 +00:00
* In the mean time we suspend as well.
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
*/
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
thread_suspend_switch(td, p);
Move the repeated code to calculate the number of the threads in the process that still need to be suspended or exited from thread_single into the new function calc_remaining(). Tested by: pho Reviewed by: jhb Approved by: re (kensmith)
2009-07-14 22:51:31 +00:00
remaining = calc_remaining(p, mode);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
In original kern_execve() code, at the start of the function, it forces all other threads to suicide, problem is execve() could be failed, and a failed execve() would change threaded process to unthreaded, this side effect is unexpected. The new code introduces a new single threading mode SINGLE_BOUNDARY, in the mode, all threads should suspend themself at user boundary except the singler. we can not use SINGLE_NO_EXIT because we want to start from a clean state if execve() is successful, suspending other threads at unknown point and later resuming them from there and forcing them to exit at user boundary may cause the process to start from a dirty state. If execve() is successful, current thread upgrades to SINGLE_EXIT mode and forces other threads to suicide at user boundary, otherwise, other threads will be resumed and their interrupted syscall will be restarted. Reviewed by: julian
2004-10-06 00:40:41 +00:00
if (mode == SINGLE_EXIT) {
Try harder to get back to being a non threaded process. Submitted by: DavidXu MFC after: 3 days
2004-09-15 18:39:09 +00:00
/*
Retire thread_unthread(), it has only one caller. Update comment in the block of code before the previous call to thread_unthread(). Discussed with: alc Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-09-03 08:35:42 +00:00
* Convert the process to an unthreaded process. The
* SINGLE_EXIT is called by exit1() or execve(), in
* both cases other threads must be retired.
Try harder to get back to being a non threaded process. Submitted by: DavidXu MFC after: 3 days
2004-09-15 18:39:09 +00:00
*/
Retire thread_unthread(), it has only one caller. Update comment in the block of code before the previous call to thread_unthread(). Discussed with: alc Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-09-03 08:35:42 +00:00
KASSERT(p->p_numthreads == 1, ("Unthreading with >1 threads"));
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
p->p_singlethread = NULL;
Retire thread_unthread(), it has only one caller. Update comment in the block of code before the previous call to thread_unthread(). Discussed with: alc Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-09-03 08:35:42 +00:00
p->p_flag &= ~(P_STOPPED_SINGLE | P_SINGLE_EXIT | P_HADTHREADS);
Right now, thread_single(SINGLE_EXIT) returns after the p_numthreads reaches 1. The p_numthreads counter is decremented in thread_exit() by a call to thread_unlink(). This means that the exiting threads may still execute on other CPUs when thread_single(SINGLE_EXIT) returns. As result, vmspace could be destroyed while paging structures are still used on other CPUs by exiting threads. Delay the return from thread_single(SINGLE_EXIT) until all threads are really destroyed by thread_stash() after the last switch out. The p_exitthreads counter already provides the required mechanism, move the wait from the thread_wait() (which is called from wait(2) code) into thread_single(). Reported by: many (as "panic: pmap active <addr>") Reviewed by: alc, jhb Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-09-03 08:18:07 +00:00
/*
* Wait for any remaining threads to exit cpu_throw().
*/
while (p->p_exitthreads != 0) {
PROC_SUNLOCK(p);
PROC_UNLOCK(p);
sched_relinquish(td);
PROC_LOCK(p);
PROC_SLOCK(p);
}
Do not return from thread_single(SINGLE_BOUNDARY) until all stopped thread are guarenteed to be removed from the processors. Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-09 18:32:13 +00:00
} else if (mode == SINGLE_BOUNDARY) {
/*
* Wait until all suspended threads are removed from
* the processors. The thread_suspend_check()
* increments p_boundary_count while it is still
* running, which makes it possible for the execve()
* to destroy vmspace while our other threads are
* still using the address space.
*
* We lock the thread, which is only allowed to
* succeed after context switch code finished using
* the address space.
*/
FOREACH_THREAD_IN_PROC(p, td2) {
if (td2 == td)
continue;
thread_lock(td2);
KASSERT((td2->td_flags & TDF_BOUNDARY) != 0,
("td %p not on boundary", td2));
KASSERT(TD_IS_SUSPENDED(td2),
("td %p is not suspended", td2));
thread_unlock(td2);
}
- Split the struct kse into struct upcall and struct kse. struct kse will soon be visible only to schedulers. This greatly simplifies much the KSE code. Submitted by: davidxu
2003-02-17 05:14:26 +00:00
}
Revert rev 184216 and 184199, due to the way the thread_lock works, it may cause a lockup. Noticed by: peter, jhb
2008-11-05 03:01:23 +00:00
PROC_SUNLOCK(p);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
return (0);
}
Thread waiting for the vfork(2)-ed child to exec or exit, must allow for the suspension. Currently, the loop performs uninterruptible cv_wait(9) call, which prevents suspension until child allows further execution of parent. If child is stopped, suspension or single-threading is delayed indefinitely. Create a helper thread_suspend_check_needed() to identify the need for a call to thread_suspend_check(). It is required since call to the thread_suspend_check() cannot be safely done while owning the child (p2) process lock. Only when suspension is needed, drop p2 lock and call thread_suspend_check(). Perform wait for cv with timeout, in case suspend is requested after wait started; I do not see a better way to interrupt the wait. Reported and tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-12-08 16:18:05 +00:00
bool
thread_suspend_check_needed(void)
{
struct proc *p;
struct thread *td;
td = curthread;
p = td->td_proc;
PROC_LOCK_ASSERT(p, MA_OWNED);
return (P_SHOULDSTOP(p) || ((p->p_flag & P_TRACED) != 0 &&
(td->td_dbgflags & TDB_SUSPEND) != 0));
}
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
* Called in from locations that can safely check to see
* whether we have to suspend or at least throttle for a
* single-thread event (e.g. fork).
*
* Such locations include userret().
* If the "return_instead" argument is non zero, the thread must be able to
* accept 0 (caller may continue), or 1 (caller must abort) as a result.
*
* The 'return_instead' argument tells the function if it may do a
* thread_exit() or suspend, or whether the caller must abort and back
* out instead.
*
* If the thread that set the single_threading request has set the
* P_SINGLE_EXIT bit in the process flags then this call will never return
* if 'return_instead' is false, but will exit.
*
* P_SINGLE_EXIT | return_instead == 0| return_instead != 0
*---------------+--------------------+---------------------
* 0 | returns 0 | returns 0 or 1
Fix a few typos.
2013-02-19 16:35:27 +00:00
* | when ST ends | immediately
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
*---------------+--------------------+---------------------
* 1 | thread exits | returns 1
Fix a few typos.
2013-02-19 16:35:27 +00:00
* | | immediately
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
* 0 = thread_exit() or suspension ok,
* other = return error instead of stopping the thread.
*
* While a full suspension is under effect, even a single threading
* thread would be suspended if it made this call (but it shouldn't).
* This call should only be made from places where
s/Muliple/Multiple Removed whitespace at EOL and EOF.
2004-01-10 18:34:01 +00:00
* thread_exit() would be safe as that may be the outcome unless
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
* return_instead is set.
*/
int
thread_suspend_check(int return_instead)
{
Remove bogus duplicate assignment of local variables.
2002-10-05 04:35:59 +00:00
struct thread *td;
struct proc *p;
Provide helper macros to detect 'non-silent SBDRY' state and to calculate appropriate return value for stops. Simplify the code by using them. Fix typo in sig_suspend_threads(). The thread which sleep must be aborted is td2. (*) In issignal(), when handling stopping signal for thread in TD_SBDRY_INTR state, do not stop, this is wrong and fires assert. This is yet another place where execution should be forced out of SBDRY-protected region. For such case, return -1 from issignal() and translate it to corresponding error code in sleepq_catch_signals(). Assert that other consumers of cursig() are not affected by the new return value. (*) Micro-optimize, mostly VFS and VOP methods, by avoiding calling the functions when SIGDEFERSTOP_NOP non-change is requested. (**) Reported and tested by: pho (*) Requested by: bde (**) Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Approved by: re (gjb)
2016-07-03 18:19:48 +00:00
int wakeup_swapper;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
td = curthread;
p = td->td_proc;
Push Giant down a little further: - no longer serialize on Giant for thread_single*() and family in fork, exit and exec - thread_wait() is mpsafe, assert no Giant - reduce scope of Giant in exit to not cover thread_wait and just do vm_waitproc(). - assert that thread_single() family are not called with Giant - remove the DROP/PICKUP_GIANT macros from thread_single() family - assert that thread_suspend_check() s not called with Giant - remove manual drop_giant hack in thread_suspend_check since we know it isn't held. - remove the DROP/PICKUP_GIANT macros from thread_suspend_check() family - mark kse_create() mpsafe
2004-03-13 22:31:39 +00:00
mtx_assert(&Giant, MA_NOTOWNED);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
PROC_LOCK_ASSERT(p, MA_OWNED);
Thread waiting for the vfork(2)-ed child to exec or exit, must allow for the suspension. Currently, the loop performs uninterruptible cv_wait(9) call, which prevents suspension until child allows further execution of parent. If child is stopped, suspension or single-threading is delayed indefinitely. Create a helper thread_suspend_check_needed() to identify the need for a call to thread_suspend_check(). It is required since call to the thread_suspend_check() cannot be safely done while owning the child (p2) process lock. Only when suspension is needed, drop p2 lock and call thread_suspend_check(). Perform wait for cv with timeout, in case suspend is requested after wait started; I do not see a better way to interrupt the wait. Reported and tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2014-12-08 16:18:05 +00:00
while (thread_suspend_check_needed()) {
s/SGNL/SIG/ s/SNGL/SINGLE/ s/SNGLE/SINGLE/ Fix abbreviation for P_STOPPED_* etc flags, in original code they were inconsistent and difficult to distinguish between them. Approved by: julian (mentor)
2002-09-05 07:30:18 +00:00
if (P_SHOULDSTOP(p) == P_STOPPED_SINGLE) {
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
KASSERT(p->p_singlethread != NULL,
("singlethread not set"));
/*
fix some style problems and remove a mis-merged assert.
2002-07-25 00:27:39 +00:00
* The only suspension in action is a
* single-threading. Single threader need not stop.
Remove XXX comments from kern_thread.c. In one case, there is no reason for it in modern times. In the other case, expand the comment stating instead of doubting. Reviewed by: jhb Sponsored by: The FreeBSD Foundation MFC after: 1 week Approved by: re (hrs) X-Differential revision: https://reviews.freebsd.org/D6731
2016-06-16 12:01:11 +00:00
* It is safe to access p->p_singlethread unlocked
* because it can only be set to our address by us.
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
*/
fix some style problems and remove a mis-merged assert.
2002-07-25 00:27:39 +00:00
if (p->p_singlethread == td)
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
return (0); /* Exempt from stopping. */
s/Muliple/Multiple Removed whitespace at EOL and EOF.
2004-01-10 18:34:01 +00:00
}
Only test return_instead if P_SINGLE_EXIT is set, otherwise a fork() syscall can interrupt other thread's syscall in sleepq_catch_signals(). Current, all callers know thread_suspend_check may suspend thread itself, so we need't to check return_instead for normal suspension flags (no P_SINGLE_EXIT set). Tested by: deischen Reported by: Maarten L. Hekkelman <m.hekkelman@cmbi.kun.nl>
2004-08-29 23:10:02 +00:00
if ((p->p_flag & P_SINGLE_EXIT) && return_instead)
Fix a long standing race between sleep queue and thread suspension code. When a thread A is going to sleep, it calls sleepq_catch_signals() to detect any pending signals or thread suspension request, if nothing happens, it returns without holding process lock or scheduler lock, this opens a race window which allows thread B to come in and do process suspension work, however since A is still at running state, thread B can do nothing to A, thread A continues, and puts itself into actually sleeping state, but B has never seen it, and it sits there forever until B is woken up by other threads sometimes later(this can be very long delay or never happen). Fix this bug by forcing sleepq_catch_signals to return with scheduler lock held. Fix sleepq_abort() by passing it an interrupted code, previously, it worked as wakeup_one(), and the interruption can not be identified correctly by sleep queue code when the sleeping thread is resumed. Let thread_suspend_check() returns EINTR or ERESTART, so sleep queue no longer has to use SIGSTOP as a hack to build a return value. Reviewed by: jhb MFC after: 1 week
2006-02-15 23:52:01 +00:00
return (EINTR);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
In original kern_execve() code, at the start of the function, it forces all other threads to suicide, problem is execve() could be failed, and a failed execve() would change threaded process to unthreaded, this side effect is unexpected. The new code introduces a new single threading mode SINGLE_BOUNDARY, in the mode, all threads should suspend themself at user boundary except the singler. we can not use SINGLE_NO_EXIT because we want to start from a clean state if execve() is successful, suspending other threads at unknown point and later resuming them from there and forcing them to exit at user boundary may cause the process to start from a dirty state. If execve() is successful, current thread upgrades to SINGLE_EXIT mode and forces other threads to suicide at user boundary, otherwise, other threads will be resumed and their interrupted syscall will be restarted. Reviewed by: julian
2004-10-06 00:40:41 +00:00
/* Should we goto user boundary if we didn't come from there? */
if (P_SHOULDSTOP(p) == P_STOPPED_SINGLE &&
(p->p_flag & P_SINGLE_BOUNDARY) && return_instead)
Fix a long standing race between sleep queue and thread suspension code. When a thread A is going to sleep, it calls sleepq_catch_signals() to detect any pending signals or thread suspension request, if nothing happens, it returns without holding process lock or scheduler lock, this opens a race window which allows thread B to come in and do process suspension work, however since A is still at running state, thread B can do nothing to A, thread A continues, and puts itself into actually sleeping state, but B has never seen it, and it sits there forever until B is woken up by other threads sometimes later(this can be very long delay or never happen). Fix this bug by forcing sleepq_catch_signals to return with scheduler lock held. Fix sleepq_abort() by passing it an interrupted code, previously, it worked as wakeup_one(), and the interruption can not be identified correctly by sleep queue code when the sleeping thread is resumed. Let thread_suspend_check() returns EINTR or ERESTART, so sleep queue no longer has to use SIGSTOP as a hack to build a return value. Reviewed by: jhb MFC after: 1 week
2006-02-15 23:52:01 +00:00
return (ERESTART);
In original kern_execve() code, at the start of the function, it forces all other threads to suicide, problem is execve() could be failed, and a failed execve() would change threaded process to unthreaded, this side effect is unexpected. The new code introduces a new single threading mode SINGLE_BOUNDARY, in the mode, all threads should suspend themself at user boundary except the singler. we can not use SINGLE_NO_EXIT because we want to start from a clean state if execve() is successful, suspending other threads at unknown point and later resuming them from there and forcing them to exit at user boundary may cause the process to start from a dirty state. If execve() is successful, current thread upgrades to SINGLE_EXIT mode and forces other threads to suicide at user boundary, otherwise, other threads will be resumed and their interrupted syscall will be restarted. Reviewed by: julian
2004-10-06 00:40:41 +00:00
Another NFS SIGSTOP related fix: Ignore thread suspend requests due to SIGSTOP if stop signals are currently deferred. This can occur if a process is stopped via SIGSTOP while a thread is running or runnable but before it has set TDF_SBDRY. Tested by: pho Reviewed by: kib MFC after: 1 week
2013-03-21 14:06:27 +00:00
/*
If thread requested to not stop on non-boundary, then not only stopping signals should obey, but also all forms of single-threading. Otherwise, thread might sleep interruptible while owning some resources, and single-threading thread could try to access them. An example is owning vnode lock while dumping core. Submitted by: Conrad Meyer Review: https://reviews.freebsd.org/D2612 Tested by: pho MFC after: 1 week
2015-05-23 19:09:04 +00:00
* Ignore suspend requests if they are deferred.
Another NFS SIGSTOP related fix: Ignore thread suspend requests due to SIGSTOP if stop signals are currently deferred. This can occur if a process is stopped via SIGSTOP while a thread is running or runnable but before it has set TDF_SBDRY. Tested by: pho Reviewed by: kib MFC after: 1 week
2013-03-21 14:06:27 +00:00
*/
If thread requested to not stop on non-boundary, then not only stopping signals should obey, but also all forms of single-threading. Otherwise, thread might sleep interruptible while owning some resources, and single-threading thread could try to access them. An example is owning vnode lock while dumping core. Submitted by: Conrad Meyer Review: https://reviews.freebsd.org/D2612 Tested by: pho MFC after: 1 week
2015-05-23 19:09:04 +00:00
if ((td->td_flags & TDF_SBDRY) != 0) {
Another NFS SIGSTOP related fix: Ignore thread suspend requests due to SIGSTOP if stop signals are currently deferred. This can occur if a process is stopped via SIGSTOP while a thread is running or runnable but before it has set TDF_SBDRY. Tested by: pho Reviewed by: kib MFC after: 1 week
2013-03-21 14:06:27 +00:00
KASSERT(return_instead,
("TDF_SBDRY set for unsafe thread_suspend_check"));
Provide helper macros to detect 'non-silent SBDRY' state and to calculate appropriate return value for stops. Simplify the code by using them. Fix typo in sig_suspend_threads(). The thread which sleep must be aborted is td2. (*) In issignal(), when handling stopping signal for thread in TD_SBDRY_INTR state, do not stop, this is wrong and fires assert. This is yet another place where execution should be forced out of SBDRY-protected region. For such case, return -1 from issignal() and translate it to corresponding error code in sleepq_catch_signals(). Assert that other consumers of cursig() are not affected by the new return value. (*) Micro-optimize, mostly VFS and VOP methods, by avoiding calling the functions when SIGDEFERSTOP_NOP non-change is requested. (**) Reported and tested by: pho (*) Requested by: bde (**) Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Approved by: re (gjb)
2016-07-03 18:19:48 +00:00
KASSERT((td->td_flags & (TDF_SEINTR | TDF_SERESTART)) !=
(TDF_SEINTR | TDF_SERESTART),
("both TDF_SEINTR and TDF_SERESTART"));
return (TD_SBDRY_INTR(td) ? TD_SBDRY_ERRNO(td) : 0);
Another NFS SIGSTOP related fix: Ignore thread suspend requests due to SIGSTOP if stop signals are currently deferred. This can occur if a process is stopped via SIGSTOP while a thread is running or runnable but before it has set TDF_SBDRY. Tested by: pho Reviewed by: kib MFC after: 1 week
2013-03-21 14:06:27 +00:00
}
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
* If the process is waiting for us to exit,
* this thread should just suicide.
s/SGNL/SIG/ s/SNGL/SINGLE/ s/SNGLE/SINGLE/ Fix abbreviation for P_STOPPED_* etc flags, in original code they were inconsistent and difficult to distinguish between them. Approved by: julian (mentor)
2002-09-05 07:30:18 +00:00
* Assumes that P_SINGLE_EXIT implies P_STOPPED_SINGLE.
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
*/
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
if ((p->p_flag & P_SINGLE_EXIT) && (p->p_singlethread != td)) {
PROC_UNLOCK(p);
In preparation for switching linuxulator to the use the native 1:1 threads add a hook for cleaning thread resources before the thread die. Differential Revision: https://reviews.freebsd.org/D1038
2015-05-24 14:51:29 +00:00
/*
* Allow Linux emulation layer to do some work
* before thread suicide.
*/
if (__predict_false(p->p_sysent->sv_thread_detach != NULL))
(p->p_sysent->sv_thread_detach)(td);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
umtx_thread_exit(td);
Call kern_thr_exit() instead of duplicating it. This code is missing the racct_subr() call from kern_thr_exit() and would require further code duplication in future changes. Reviewed by: kib MFC after: 1 week
2015-12-29 23:16:20 +00:00
kern_thr_exit(td);
panic("stopped thread did not exit");
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
}
- Insert thread0 into correct thread hash link list. - In thr_exit() and kthread_exit(), only remove thread from hash if it can directly exit, otherwise let exit1() do it. - In thread_suspend_check(), fix cleanup code when thread needs to exit. This change seems fixed the "Bad link elm " panic found by Peter Holm. Stress testing: pho
2010-10-17 11:01:52 +00:00
PROC_SLOCK(p);
thread_stopped(p);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
if (P_SHOULDSTOP(p) == P_STOPPED_SINGLE) {
if (p->p_numthreads == p->p_suspcount + 1) {
thread_lock(p->p_singlethread);
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
wakeup_swapper = thread_unsuspend_one(
p->p_singlethread, p, false);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
thread_unlock(p->p_singlethread);
A suspended thread can, in fact, be swapped out. Thus, thread_unsuspend_one() needs to optionally wakeup the swapper. Since we hold the thread lock for that entire function, however, we have to push that requirement up into the caller. Found by: rwatson
2008-08-22 16:15:58 +00:00
if (wakeup_swapper)
kick_proc0();
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
}
}
Actually, for signal and thread suspension, extra process spin lock is unnecessary, the normal process lock and thread lock are enough. The spin lock is still needed for process and thread exiting to mimic single sched_lock.
2008-10-23 07:55:38 +00:00
PROC_UNLOCK(p);
Revert rev 184216 and 184199, due to the way the thread_lock works, it may cause a lockup. Noticed by: peter, jhb
2008-11-05 03:01:23 +00:00
thread_lock(td);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
* When a thread suspends, it just
Threading cleanup.. part 2 of several. Make part of John Birrell's KSE patch permanent.. Specifically, remove: Any reference of the ksegrp structure. This feature was never fully utilised and made things overly complicated. All code in the scheduler that tried to make threaded programs fair to unthreaded programs. Libpthread processes will already do this to some extent and libthr processes already disable it. Also: Since this makes such a big change to the scheduler(s), take the opportunity to rename some structures and elements that had to be moved anyhow. This makes the code a lot more readable. The ULE scheduler compiles again but I have no idea if it works. The 4bsd scheduler still reqires a little cleaning and some functions that now do ALMOST nothing will go away, but I thought I'd do that as a separate commit. Tested by David Xu, and Dan Eischen using libthr and libpthread.
2006-12-06 06:34:57 +00:00
* gets taken off all queues.
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
*/
Completely redo thread states. Reviewed by: davidxu@freebsd.org
2002-09-11 08:13:56 +00:00
thread_suspend_one(td);
In original kern_execve() code, at the start of the function, it forces all other threads to suicide, problem is execve() could be failed, and a failed execve() would change threaded process to unthreaded, this side effect is unexpected. The new code introduces a new single threading mode SINGLE_BOUNDARY, in the mode, all threads should suspend themself at user boundary except the singler. we can not use SINGLE_NO_EXIT because we want to start from a clean state if execve() is successful, suspending other threads at unknown point and later resuming them from there and forcing them to exit at user boundary may cause the process to start from a dirty state. If execve() is successful, current thread upgrades to SINGLE_EXIT mode and forces other threads to suicide at user boundary, otherwise, other threads will be resumed and their interrupted syscall will be restarted. Reviewed by: julian
2004-10-06 00:40:41 +00:00
if (return_instead == 0) {
p->p_boundary_count++;
td->td_flags |= TDF_BOUNDARY;
}
Revert rev 184216 and 184199, due to the way the thread_lock works, it may cause a lockup. Noticed by: peter, jhb
2008-11-05 03:01:23 +00:00
PROC_SUNLOCK(p);
- Make SCHED_STATS more generic by adding a wrapper to create the variables and sysctl nodes. - In reset walk the children of kern_sched_stats and reset the counters via the oid_arg1 pointer. This allows us to add arbitrary counters to the tree and still reset them properly. - Define a set of switch types to be passed with flags to mi_switch(). These types are named SWT_*. These types correspond to SCHED_STATS counters and are automatically handled in this way. - Make the new SWT_ types more specific than the older switch stats. There are now stats for idle switches, remote idle wakeups, remote preemption ithreads idling, etc. - Add switch statistics for ULE's pickcpu algorithm. These stats include how much migration there is, how often affinity was successful, how often threads were migrated to the local cpu on wakeup, etc. Sponsored by: Nokia
2008-04-17 04:20:10 +00:00
mi_switch(SW_INVOL | SWT_SUSPEND, NULL);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
thread_unlock(td);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
PROC_LOCK(p);
}
return (0);
}
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
void
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
thread_suspend_switch(struct thread *td, struct proc *p)
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
{
KASSERT(!TD_IS_SUSPENDED(td), ("already suspended"));
PROC_LOCK_ASSERT(p, MA_OWNED);
Revert rev 184216 and 184199, due to the way the thread_lock works, it may cause a lockup. Noticed by: peter, jhb
2008-11-05 03:01:23 +00:00
PROC_SLOCK_ASSERT(p, MA_OWNED);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
/*
* We implement thread_suspend_one in stages here to avoid
* dropping the proc lock while the thread lock is owned.
*/
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
if (p == td->td_proc) {
thread_stopped(p);
p->p_suspcount++;
}
Actually, for signal and thread suspension, extra process spin lock is unnecessary, the normal process lock and thread lock are enough. The spin lock is still needed for process and thread exiting to mimic single sched_lock.
2008-10-23 07:55:38 +00:00
PROC_UNLOCK(p);
Revert rev 184216 and 184199, due to the way the thread_lock works, it may cause a lockup. Noticed by: peter, jhb
2008-11-05 03:01:23 +00:00
thread_lock(td);
- Add a new td flag TDF_NEEDSUSPCHK that is set whenever a thread needs to enter thread_suspend_check(). - Set TDF_ASTPENDING along with TDF_NEEDSUSPCHK so we can move the thread_suspend_check() to ast() rather than userret(). - Check TDF_NEEDSUSPCHK in the sleepq_catch_signals() optimization so that we don't miss a suspend request. If this is set use the expensive signal path. - Set NEEDSUSPCHK when creating a new thread in thr in case the creating thread is due to be suspended as well but has not yet. Reviewed by: davidxu (Authored original patch)
2008-03-21 08:23:25 +00:00
td->td_flags &= ~TDF_NEEDSUSPCHK;
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
TD_SET_SUSPENDED(td);
- Pass the priority argument from *sleep() into sleepq and down into sched_sleep(). This removes extra thread_lock() acquisition and allows the scheduler to decide what to do with the static boost. - Change the priority arguments to cv_* to match sleepq/msleep/etc. where 0 means no priority change. Catch -1 in cv_broadcastpri() and convert it to 0 for now. - Set a flag when sleeping in a way that is compatible with swapping since direct priority comparisons are meaningless now. - Add a sysctl to ule, kern.sched.static_boost, that defaults to on which controls the boost behavior. Turning it off gives better performance in some workloads but needs more investigation. - While we're modifying sleepq, change signal and broadcast to both return with the lock held as the lock was held on enter. Reviewed by: jhb, peter
2008-03-12 06:31:06 +00:00
sched_sleep(td, 0);
Revert rev 184216 and 184199, due to the way the thread_lock works, it may cause a lockup. Noticed by: peter, jhb
2008-11-05 03:01:23 +00:00
PROC_SUNLOCK(p);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
DROP_GIANT();
- Make SCHED_STATS more generic by adding a wrapper to create the variables and sysctl nodes. - In reset walk the children of kern_sched_stats and reset the counters via the oid_arg1 pointer. This allows us to add arbitrary counters to the tree and still reset them properly. - Define a set of switch types to be passed with flags to mi_switch(). These types are named SWT_*. These types correspond to SCHED_STATS counters and are automatically handled in this way. - Make the new SWT_ types more specific than the older switch stats. There are now stats for idle switches, remote idle wakeups, remote preemption ithreads idling, etc. - Add switch statistics for ULE's pickcpu algorithm. These stats include how much migration there is, how often affinity was successful, how often threads were migrated to the local cpu on wakeup, etc. Sponsored by: Nokia
2008-04-17 04:20:10 +00:00
mi_switch(SW_VOL | SWT_SUSPEND, NULL);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
thread_unlock(td);
PICKUP_GIANT();
PROC_LOCK(p);
Revert rev 184216 and 184199, due to the way the thread_lock works, it may cause a lockup. Noticed by: peter, jhb
2008-11-05 03:01:23 +00:00
PROC_SLOCK(p);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
}
In the kernel code, we have the tsleep() call with the PCATCH argument. PCATCH means 'if we get a signal, interrupt me!" and tsleep returns either EINTR or ERESTART depending on the circumstances. ERESTART is "special" because it causes the system call to fail, but right as it returns back to userland it tells the trap handler to move %eip back a bit so that userland will immediately re-run the syscall. This is a syscall restart. It only works for things like read() etc where nothing has changed yet. Note that *userland* is tricked into restarting the syscall by the kernel. The kernel doesn't actually do the restart. It is deadly for things like select, poll, nanosleep etc where it might cause the elapsed time to be reset and start again from scratch. So those syscalls do this to prevent userland rerunning the syscall: if (error == ERESTART) error = EINTR; Fake "signals" like SIGTSTP from ^Z etc do not normally invoke userland signal handlers. But, in -current, the PCATCH *is* being triggered and tsleep is returning ERESTART, and the syscall is aborted even though no userland signal handler was run. That is the fault here. We're triggering the PCATCH in cases that we shouldn't. ie: it is being triggered on *any* signal processing, rather than the case where the signal is posted to userland. --- Peter The work of psignal() is a patchwork of special case required by the process debugging and job-control facilities... --- Kirk McKusick "The design and impelementation of the 4.4BSD Operating system" Page 105 in STABLE source, when psignal is posting a STOP signal to sleeping process and the signal action of the process is SIG_DFL, system will directly change the process state from SSLEEP to SSTOP, and when SIGCONT is posted to the stopped process, if it finds that the process is still on sleep queue, the process state will be restored to SSLEEP, and won't wakeup the process. this commit mimics the behaviour in STABLE source tree. Reviewed by: Jon Mini, Tim Robbins, Peter Wemm Approved by: julian@freebsd.org (mentor)
2002-09-03 12:56:01 +00:00
void
thread_suspend_one(struct thread *td)
{
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
struct proc *p;
In the kernel code, we have the tsleep() call with the PCATCH argument. PCATCH means 'if we get a signal, interrupt me!" and tsleep returns either EINTR or ERESTART depending on the circumstances. ERESTART is "special" because it causes the system call to fail, but right as it returns back to userland it tells the trap handler to move %eip back a bit so that userland will immediately re-run the syscall. This is a syscall restart. It only works for things like read() etc where nothing has changed yet. Note that *userland* is tricked into restarting the syscall by the kernel. The kernel doesn't actually do the restart. It is deadly for things like select, poll, nanosleep etc where it might cause the elapsed time to be reset and start again from scratch. So those syscalls do this to prevent userland rerunning the syscall: if (error == ERESTART) error = EINTR; Fake "signals" like SIGTSTP from ^Z etc do not normally invoke userland signal handlers. But, in -current, the PCATCH *is* being triggered and tsleep is returning ERESTART, and the syscall is aborted even though no userland signal handler was run. That is the fault here. We're triggering the PCATCH in cases that we shouldn't. ie: it is being triggered on *any* signal processing, rather than the case where the signal is posted to userland. --- Peter The work of psignal() is a patchwork of special case required by the process debugging and job-control facilities... --- Kirk McKusick "The design and impelementation of the 4.4BSD Operating system" Page 105 in STABLE source, when psignal is posting a STOP signal to sleeping process and the signal action of the process is SIG_DFL, system will directly change the process state from SSLEEP to SSTOP, and when SIGCONT is posted to the stopped process, if it finds that the process is still on sleep queue, the process state will be restored to SSLEEP, and won't wakeup the process. this commit mimics the behaviour in STABLE source tree. Reviewed by: Jon Mini, Tim Robbins, Peter Wemm Approved by: julian@freebsd.org (mentor)
2002-09-03 12:56:01 +00:00
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
p = td->td_proc;
Revert rev 184216 and 184199, due to the way the thread_lock works, it may cause a lockup. Noticed by: peter, jhb
2008-11-05 03:01:23 +00:00
PROC_SLOCK_ASSERT(p, MA_OWNED);
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
THREAD_LOCK_ASSERT(td, MA_OWNED);
Fix threaded process job control bug. SMP tested. Reviewed by: julian
2003-03-11 00:07:53 +00:00
KASSERT(!TD_IS_SUSPENDED(td), ("already suspended"));
In the kernel code, we have the tsleep() call with the PCATCH argument. PCATCH means 'if we get a signal, interrupt me!" and tsleep returns either EINTR or ERESTART depending on the circumstances. ERESTART is "special" because it causes the system call to fail, but right as it returns back to userland it tells the trap handler to move %eip back a bit so that userland will immediately re-run the syscall. This is a syscall restart. It only works for things like read() etc where nothing has changed yet. Note that *userland* is tricked into restarting the syscall by the kernel. The kernel doesn't actually do the restart. It is deadly for things like select, poll, nanosleep etc where it might cause the elapsed time to be reset and start again from scratch. So those syscalls do this to prevent userland rerunning the syscall: if (error == ERESTART) error = EINTR; Fake "signals" like SIGTSTP from ^Z etc do not normally invoke userland signal handlers. But, in -current, the PCATCH *is* being triggered and tsleep is returning ERESTART, and the syscall is aborted even though no userland signal handler was run. That is the fault here. We're triggering the PCATCH in cases that we shouldn't. ie: it is being triggered on *any* signal processing, rather than the case where the signal is posted to userland. --- Peter The work of psignal() is a patchwork of special case required by the process debugging and job-control facilities... --- Kirk McKusick "The design and impelementation of the 4.4BSD Operating system" Page 105 in STABLE source, when psignal is posting a STOP signal to sleeping process and the signal action of the process is SIG_DFL, system will directly change the process state from SSLEEP to SSTOP, and when SIGCONT is posted to the stopped process, if it finds that the process is still on sleep queue, the process state will be restored to SSLEEP, and won't wakeup the process. this commit mimics the behaviour in STABLE source tree. Reviewed by: Jon Mini, Tim Robbins, Peter Wemm Approved by: julian@freebsd.org (mentor)
2002-09-03 12:56:01 +00:00
p->p_suspcount++;
- Add a new td flag TDF_NEEDSUSPCHK that is set whenever a thread needs to enter thread_suspend_check(). - Set TDF_ASTPENDING along with TDF_NEEDSUSPCHK so we can move the thread_suspend_check() to ast() rather than userret(). - Check TDF_NEEDSUSPCHK in the sleepq_catch_signals() optimization so that we don't miss a suspend request. If this is set use the expensive signal path. - Set NEEDSUSPCHK when creating a new thread in thr in case the creating thread is due to be suspended as well but has not yet. Reviewed by: davidxu (Authored original patch)
2008-03-21 08:23:25 +00:00
td->td_flags &= ~TDF_NEEDSUSPCHK;
Completely redo thread states. Reviewed by: davidxu@freebsd.org
2002-09-11 08:13:56 +00:00
TD_SET_SUSPENDED(td);
- Pass the priority argument from *sleep() into sleepq and down into sched_sleep(). This removes extra thread_lock() acquisition and allows the scheduler to decide what to do with the static boost. - Change the priority arguments to cv_* to match sleepq/msleep/etc. where 0 means no priority change. Catch -1 in cv_broadcastpri() and convert it to 0 for now. - Set a flag when sleeping in a way that is compatible with swapping since direct priority comparisons are meaningless now. - Add a sysctl to ule, kern.sched.static_boost, that defaults to on which controls the boost behavior. Turning it off gives better performance in some workloads but needs more investigation. - While we're modifying sleepq, change signal and broadcast to both return with the lock held as the lock was held on enter. Reviewed by: jhb, peter
2008-03-12 06:31:06 +00:00
sched_sleep(td, 0);
In the kernel code, we have the tsleep() call with the PCATCH argument. PCATCH means 'if we get a signal, interrupt me!" and tsleep returns either EINTR or ERESTART depending on the circumstances. ERESTART is "special" because it causes the system call to fail, but right as it returns back to userland it tells the trap handler to move %eip back a bit so that userland will immediately re-run the syscall. This is a syscall restart. It only works for things like read() etc where nothing has changed yet. Note that *userland* is tricked into restarting the syscall by the kernel. The kernel doesn't actually do the restart. It is deadly for things like select, poll, nanosleep etc where it might cause the elapsed time to be reset and start again from scratch. So those syscalls do this to prevent userland rerunning the syscall: if (error == ERESTART) error = EINTR; Fake "signals" like SIGTSTP from ^Z etc do not normally invoke userland signal handlers. But, in -current, the PCATCH *is* being triggered and tsleep is returning ERESTART, and the syscall is aborted even though no userland signal handler was run. That is the fault here. We're triggering the PCATCH in cases that we shouldn't. ie: it is being triggered on *any* signal processing, rather than the case where the signal is posted to userland. --- Peter The work of psignal() is a patchwork of special case required by the process debugging and job-control facilities... --- Kirk McKusick "The design and impelementation of the 4.4BSD Operating system" Page 105 in STABLE source, when psignal is posting a STOP signal to sleeping process and the signal action of the process is SIG_DFL, system will directly change the process state from SSLEEP to SSTOP, and when SIGCONT is posted to the stopped process, if it finds that the process is still on sleep queue, the process state will be restored to SSLEEP, and won't wakeup the process. this commit mimics the behaviour in STABLE source tree. Reviewed by: Jon Mini, Tim Robbins, Peter Wemm Approved by: julian@freebsd.org (mentor)
2002-09-03 12:56:01 +00:00
}
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
static int
thread_unsuspend_one(struct thread *td, struct proc *p, bool boundary)
In the kernel code, we have the tsleep() call with the PCATCH argument. PCATCH means 'if we get a signal, interrupt me!" and tsleep returns either EINTR or ERESTART depending on the circumstances. ERESTART is "special" because it causes the system call to fail, but right as it returns back to userland it tells the trap handler to move %eip back a bit so that userland will immediately re-run the syscall. This is a syscall restart. It only works for things like read() etc where nothing has changed yet. Note that *userland* is tricked into restarting the syscall by the kernel. The kernel doesn't actually do the restart. It is deadly for things like select, poll, nanosleep etc where it might cause the elapsed time to be reset and start again from scratch. So those syscalls do this to prevent userland rerunning the syscall: if (error == ERESTART) error = EINTR; Fake "signals" like SIGTSTP from ^Z etc do not normally invoke userland signal handlers. But, in -current, the PCATCH *is* being triggered and tsleep is returning ERESTART, and the syscall is aborted even though no userland signal handler was run. That is the fault here. We're triggering the PCATCH in cases that we shouldn't. ie: it is being triggered on *any* signal processing, rather than the case where the signal is posted to userland. --- Peter The work of psignal() is a patchwork of special case required by the process debugging and job-control facilities... --- Kirk McKusick "The design and impelementation of the 4.4BSD Operating system" Page 105 in STABLE source, when psignal is posting a STOP signal to sleeping process and the signal action of the process is SIG_DFL, system will directly change the process state from SSLEEP to SSTOP, and when SIGCONT is posted to the stopped process, if it finds that the process is still on sleep queue, the process state will be restored to SSLEEP, and won't wakeup the process. this commit mimics the behaviour in STABLE source tree. Reviewed by: Jon Mini, Tim Robbins, Peter Wemm Approved by: julian@freebsd.org (mentor)
2002-09-03 12:56:01 +00:00
{
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
THREAD_LOCK_ASSERT(td, MA_OWNED);
Threading cleanup.. part 2 of several. Make part of John Birrell's KSE patch permanent.. Specifically, remove: Any reference of the ksegrp structure. This feature was never fully utilised and made things overly complicated. All code in the scheduler that tried to make threaded programs fair to unthreaded programs. Libpthread processes will already do this to some extent and libthr processes already disable it. Also: Since this makes such a big change to the scheduler(s), take the opportunity to rename some structures and elements that had to be moved anyhow. This makes the code a lot more readable. The ULE scheduler compiles again but I have no idea if it works. The 4bsd scheduler still reqires a little cleaning and some functions that now do ALMOST nothing will go away, but I thought I'd do that as a separate commit. Tested by David Xu, and Dan Eischen using libthr and libpthread.
2006-12-06 06:34:57 +00:00
KASSERT(TD_IS_SUSPENDED(td), ("Thread not suspended"));
Completely redo thread states. Reviewed by: davidxu@freebsd.org
2002-09-11 08:13:56 +00:00
TD_CLR_SUSPENDED(td);
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
td->td_flags &= ~TDF_ALLPROCSUSP;
if (td->td_proc == p) {
PROC_SLOCK_ASSERT(p, MA_OWNED);
p->p_suspcount--;
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
if (boundary && (td->td_flags & TDF_BOUNDARY) != 0) {
td->td_flags &= ~TDF_BOUNDARY;
p->p_boundary_count--;
}
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
}
A suspended thread can, in fact, be swapped out. Thus, thread_unsuspend_one() needs to optionally wakeup the swapper. Since we hold the thread lock for that entire function, however, we have to push that requirement up into the caller. Found by: rwatson
2008-08-22 16:15:58 +00:00
return (setrunnable(td));
In the kernel code, we have the tsleep() call with the PCATCH argument. PCATCH means 'if we get a signal, interrupt me!" and tsleep returns either EINTR or ERESTART depending on the circumstances. ERESTART is "special" because it causes the system call to fail, but right as it returns back to userland it tells the trap handler to move %eip back a bit so that userland will immediately re-run the syscall. This is a syscall restart. It only works for things like read() etc where nothing has changed yet. Note that *userland* is tricked into restarting the syscall by the kernel. The kernel doesn't actually do the restart. It is deadly for things like select, poll, nanosleep etc where it might cause the elapsed time to be reset and start again from scratch. So those syscalls do this to prevent userland rerunning the syscall: if (error == ERESTART) error = EINTR; Fake "signals" like SIGTSTP from ^Z etc do not normally invoke userland signal handlers. But, in -current, the PCATCH *is* being triggered and tsleep is returning ERESTART, and the syscall is aborted even though no userland signal handler was run. That is the fault here. We're triggering the PCATCH in cases that we shouldn't. ie: it is being triggered on *any* signal processing, rather than the case where the signal is posted to userland. --- Peter The work of psignal() is a patchwork of special case required by the process debugging and job-control facilities... --- Kirk McKusick "The design and impelementation of the 4.4BSD Operating system" Page 105 in STABLE source, when psignal is posting a STOP signal to sleeping process and the signal action of the process is SIG_DFL, system will directly change the process state from SSLEEP to SSTOP, and when SIGCONT is posted to the stopped process, if it finds that the process is still on sleep queue, the process state will be restored to SSLEEP, and won't wakeup the process. this commit mimics the behaviour in STABLE source tree. Reviewed by: Jon Mini, Tim Robbins, Peter Wemm Approved by: julian@freebsd.org (mentor)
2002-09-03 12:56:01 +00:00
}
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
* Allow all threads blocked by single threading to continue running.
*/
void
thread_unsuspend(struct proc *p)
{
struct thread *td;
A suspended thread can, in fact, be swapped out. Thus, thread_unsuspend_one() needs to optionally wakeup the swapper. Since we hold the thread lock for that entire function, however, we have to push that requirement up into the caller. Found by: rwatson
2008-08-22 16:15:58 +00:00
int wakeup_swapper;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
PROC_LOCK_ASSERT(p, MA_OWNED);
Revert rev 184216 and 184199, due to the way the thread_lock works, it may cause a lockup. Noticed by: peter, jhb
2008-11-05 03:01:23 +00:00
PROC_SLOCK_ASSERT(p, MA_OWNED);
A suspended thread can, in fact, be swapped out. Thus, thread_unsuspend_one() needs to optionally wakeup the swapper. Since we hold the thread lock for that entire function, however, we have to push that requirement up into the caller. Found by: rwatson
2008-08-22 16:15:58 +00:00
wakeup_swapper = 0;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
if (!P_SHOULDSTOP(p)) {
Threading cleanup.. part 2 of several. Make part of John Birrell's KSE patch permanent.. Specifically, remove: Any reference of the ksegrp structure. This feature was never fully utilised and made things overly complicated. All code in the scheduler that tried to make threaded programs fair to unthreaded programs. Libpthread processes will already do this to some extent and libthr processes already disable it. Also: Since this makes such a big change to the scheduler(s), take the opportunity to rename some structures and elements that had to be moved anyhow. This makes the code a lot more readable. The ULE scheduler compiles again but I have no idea if it works. The 4bsd scheduler still reqires a little cleaning and some functions that now do ALMOST nothing will go away, but I thought I'd do that as a separate commit. Tested by David Xu, and Dan Eischen using libthr and libpthread.
2006-12-06 06:34:57 +00:00
FOREACH_THREAD_IN_PROC(p, td) {
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
thread_lock(td);
Threading cleanup.. part 2 of several. Make part of John Birrell's KSE patch permanent.. Specifically, remove: Any reference of the ksegrp structure. This feature was never fully utilised and made things overly complicated. All code in the scheduler that tried to make threaded programs fair to unthreaded programs. Libpthread processes will already do this to some extent and libthr processes already disable it. Also: Since this makes such a big change to the scheduler(s), take the opportunity to rename some structures and elements that had to be moved anyhow. This makes the code a lot more readable. The ULE scheduler compiles again but I have no idea if it works. The 4bsd scheduler still reqires a little cleaning and some functions that now do ALMOST nothing will go away, but I thought I'd do that as a separate commit. Tested by David Xu, and Dan Eischen using libthr and libpthread.
2006-12-06 06:34:57 +00:00
if (TD_IS_SUSPENDED(td)) {
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
wakeup_swapper |= thread_unsuspend_one(td, p,
true);
Threading cleanup.. part 2 of several. Make part of John Birrell's KSE patch permanent.. Specifically, remove: Any reference of the ksegrp structure. This feature was never fully utilised and made things overly complicated. All code in the scheduler that tried to make threaded programs fair to unthreaded programs. Libpthread processes will already do this to some extent and libthr processes already disable it. Also: Since this makes such a big change to the scheduler(s), take the opportunity to rename some structures and elements that had to be moved anyhow. This makes the code a lot more readable. The ULE scheduler compiles again but I have no idea if it works. The 4bsd scheduler still reqires a little cleaning and some functions that now do ALMOST nothing will go away, but I thought I'd do that as a separate commit. Tested by David Xu, and Dan Eischen using libthr and libpthread.
2006-12-06 06:34:57 +00:00
}
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
thread_unlock(td);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
} else if (P_SHOULDSTOP(p) == P_STOPPED_SINGLE &&
p->p_numthreads == p->p_suspcount) {
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
/*
* Stopping everything also did the job for the single
* threading request. Now we've downgraded to single-threaded,
* let it continue.
*/
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
if (p->p_singlethread->td_proc == p) {
thread_lock(p->p_singlethread);
wakeup_swapper = thread_unsuspend_one(
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
p->p_singlethread, p, false);
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
thread_unlock(p->p_singlethread);
}
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
A suspended thread can, in fact, be swapped out. Thus, thread_unsuspend_one() needs to optionally wakeup the swapper. Since we hold the thread lock for that entire function, however, we have to push that requirement up into the caller. Found by: rwatson
2008-08-22 16:15:58 +00:00
if (wakeup_swapper)
kick_proc0();
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
Refactor a bunch of scheduler code to give basically the same behaviour but with slightly cleaned up interfaces. The KSE structure has become the same as the "per thread scheduler private data" structure. In order to not make the diffs too great one is #defined as the other at this time. The KSE (or td_sched) structure is now allocated per thread and has no allocation code of its own. Concurrency for a KSEGRP is now kept track of via a simple pair of counters rather than using KSE structures as tokens. Since the KSE structure is different in each scheduler, kern_switch.c is now included at the end of each scheduler. Nothing outside the scheduler knows the contents of the KSE (aka td_sched) structure. The fields in the ksegrp structure that are to do with the scheduler's queueing mechanisms are now moved to the kg_sched structure. (per ksegrp scheduler private data structure). In other words how the scheduler queues and keeps track of threads is no-one's business except the scheduler's. This should allow people to write experimental schedulers with completely different internal structuring. A scheduler call sched_set_concurrency(kg, N) has been added that notifies teh scheduler that no more than N threads from that ksegrp should be allowed to be on concurrently scheduled. This is also used to enforce 'fainess' at this time so that a ksegrp with 10000 threads can not swamp a the run queue and force out a process with 1 thread, since the current code will not set the concurrency above NCPU, and both schedulers will not allow more than that many onto the system run queue at a time. Each scheduler should eventualy develop their own methods to do this now that they are effectively separated. Rejig libthr's kernel interface to follow the same code paths as linkse for scope system threads. This has slightly hurt libthr's performance but I will work to recover as much of it as I can. Thread exit code has been cleaned up greatly. exit and exec code now transitions a process back to 'standard non-threaded mode' before taking the next step. Reviewed by: scottl, peter MFC after: 1 week
2004-09-05 02:09:54 +00:00
/*
* End the single threading mode..
*/
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
void
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
thread_single_end(struct proc *p, int mode)
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
{
struct thread *td;
A suspended thread can, in fact, be swapped out. Thus, thread_unsuspend_one() needs to optionally wakeup the swapper. Since we hold the thread lock for that entire function, however, we have to push that requirement up into the caller. Found by: rwatson
2008-08-22 16:15:58 +00:00
int wakeup_swapper;
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
KASSERT(mode == SINGLE_EXIT || mode == SINGLE_BOUNDARY ||
mode == SINGLE_ALLPROC || mode == SINGLE_NO_EXIT,
("invalid mode %d", mode));
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
PROC_LOCK_ASSERT(p, MA_OWNED);
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
KASSERT((mode == SINGLE_ALLPROC && (p->p_flag & P_TOTAL_STOP) != 0) ||
(mode != SINGLE_ALLPROC && (p->p_flag & P_TOTAL_STOP) == 0),
("mode %d does not match P_TOTAL_STOP", mode));
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
KASSERT(mode == SINGLE_ALLPROC || p->p_singlethread == curthread,
("thread_single_end from other thread %p %p",
curthread, p->p_singlethread));
KASSERT(mode != SINGLE_BOUNDARY ||
(p->p_flag & P_SINGLE_BOUNDARY) != 0,
("mis-matched SINGLE_BOUNDARY flags %x", p->p_flag));
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
p->p_flag &= ~(P_STOPPED_SINGLE | P_SINGLE_EXIT | P_SINGLE_BOUNDARY |
P_TOTAL_STOP);
Revert rev 184216 and 184199, due to the way the thread_lock works, it may cause a lockup. Noticed by: peter, jhb
2008-11-05 03:01:23 +00:00
PROC_SLOCK(p);
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
p->p_singlethread = NULL;
A suspended thread can, in fact, be swapped out. Thus, thread_unsuspend_one() needs to optionally wakeup the swapper. Since we hold the thread lock for that entire function, however, we have to push that requirement up into the caller. Found by: rwatson
2008-08-22 16:15:58 +00:00
wakeup_swapper = 0;
slight cleanup of single-threading code for KSE processes
2002-08-22 21:45:58 +00:00
/*
A suspended thread can, in fact, be swapped out. Thus, thread_unsuspend_one() needs to optionally wakeup the swapper. Since we hold the thread lock for that entire function, however, we have to push that requirement up into the caller. Found by: rwatson
2008-08-22 16:15:58 +00:00
* If there are other threads they may now run,
slight cleanup of single-threading code for KSE processes
2002-08-22 21:45:58 +00:00
* unless of course there is a blanket 'stop order'
* on the process. The single threader must be allowed
* to continue however as this is a bad place to stop.
*/
Add facility to stop all userspace processes. The supposed use of the feature is to quisce the system before suspend. Stop is implemented by reusing the thread_single(9) with the special mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing single-threading modes by allowing (requiring) caller to operate on other process. Interruptible sleeps for !TDF_SBDRY threads are suspended like SIGSTOP does it, instead of aborting the sleep, like SINGLE_NO_EXIT, to avoid spurious EINTRs on resume. Provide debugging sysctl debug.stop_all_proc, which causes total stop and suspends syncer, while waiting for variable reset for resume. It is used for debugging; should be removed after the real use of the interface is added. In collaboration with: pho Discussed with: avg Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2014-12-13 16:18:29 +00:00
if (p->p_numthreads != remain_for_mode(mode) && !P_SHOULDSTOP(p)) {
Threading cleanup.. part 2 of several. Make part of John Birrell's KSE patch permanent.. Specifically, remove: Any reference of the ksegrp structure. This feature was never fully utilised and made things overly complicated. All code in the scheduler that tried to make threaded programs fair to unthreaded programs. Libpthread processes will already do this to some extent and libthr processes already disable it. Also: Since this makes such a big change to the scheduler(s), take the opportunity to rename some structures and elements that had to be moved anyhow. This makes the code a lot more readable. The ULE scheduler compiles again but I have no idea if it works. The 4bsd scheduler still reqires a little cleaning and some functions that now do ALMOST nothing will go away, but I thought I'd do that as a separate commit. Tested by David Xu, and Dan Eischen using libthr and libpthread.
2006-12-06 06:34:57 +00:00
FOREACH_THREAD_IN_PROC(p, td) {
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
thread_lock(td);
Threading cleanup.. part 2 of several. Make part of John Birrell's KSE patch permanent.. Specifically, remove: Any reference of the ksegrp structure. This feature was never fully utilised and made things overly complicated. All code in the scheduler that tried to make threaded programs fair to unthreaded programs. Libpthread processes will already do this to some extent and libthr processes already disable it. Also: Since this makes such a big change to the scheduler(s), take the opportunity to rename some structures and elements that had to be moved anyhow. This makes the code a lot more readable. The ULE scheduler compiles again but I have no idea if it works. The 4bsd scheduler still reqires a little cleaning and some functions that now do ALMOST nothing will go away, but I thought I'd do that as a separate commit. Tested by David Xu, and Dan Eischen using libthr and libpthread.
2006-12-06 06:34:57 +00:00
if (TD_IS_SUSPENDED(td)) {
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
wakeup_swapper |= thread_unsuspend_one(td, p,
mode == SINGLE_BOUNDARY);
Threading cleanup.. part 2 of several. Make part of John Birrell's KSE patch permanent.. Specifically, remove: Any reference of the ksegrp structure. This feature was never fully utilised and made things overly complicated. All code in the scheduler that tried to make threaded programs fair to unthreaded programs. Libpthread processes will already do this to some extent and libthr processes already disable it. Also: Since this makes such a big change to the scheduler(s), take the opportunity to rename some structures and elements that had to be moved anyhow. This makes the code a lot more readable. The ULE scheduler compiles again but I have no idea if it works. The 4bsd scheduler still reqires a little cleaning and some functions that now do ALMOST nothing will go away, but I thought I'd do that as a separate commit. Tested by David Xu, and Dan Eischen using libthr and libpthread.
2006-12-06 06:34:57 +00:00
}
Commit 4/14 of sched_lock decomposition. - Use thread_lock() rather than sched_lock for per-thread scheduling sychronization. - Use the per-process spinlock rather than the sched_lock for per-process scheduling synchronization. - Move some common code into thread_suspend_switch() to handle the mechanics of suspending a thread. The locking here is incredibly convoluted and should be simplified. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
2007-06-04 23:52:24 +00:00
thread_unlock(td);
slight cleanup of single-threading code for KSE processes
2002-08-22 21:45:58 +00:00
}
}
Right now, the process' p_boundary_count counter is decremented by the suspended thread itself, on the return path from thread_suspend_check(). A consequence is that return from thread_single_end(SINGLE_BOUNDARY) may leave p_boundary_count non-zero, it might be even equal to the threads count. Now, assume that we have two threads in the process, both calling execve(2). Suppose that the first thread won the race to be the suspension thread, and that afterward its exec failed for any reason. After the first thread did thread_single_end(SINGLE_BOUNDARY), second thread becomes the process suspension thread and checks p_boundary_count. The non-zero value of the count allows the suspension loop to finish without actually suspending some threads. In other words, we enter exec code with some threads not suspended. Fix this by decrementing p_boundary_count in the thread_single_end()->thread_unsuspend_one() during marking the thread as runnable. This way, a return from thread_single_end() guarantees that the counter is cleared. We do not care whether the unsuspended thread has a chance to run. Add some asserts to ensure the state of the process when single boundary suspension is lifted. Also make thread_unuspend_one() static. In collaboration with: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week
2015-05-15 07:54:31 +00:00
KASSERT(mode != SINGLE_BOUNDARY || p->p_boundary_count == 0,
("inconsistent boundary count %d", p->p_boundary_count));
Revert rev 184216 and 184199, due to the way the thread_lock works, it may cause a lockup. Noticed by: peter, jhb
2008-11-05 03:01:23 +00:00
PROC_SUNLOCK(p);
A suspended thread can, in fact, be swapped out. Thus, thread_unsuspend_one() needs to optionally wakeup the swapper. Since we hold the thread lock for that entire function, however, we have to push that requirement up into the caller. Found by: rwatson
2008-08-22 16:15:58 +00:00
if (wakeup_swapper)
kick_proc0();
Add files that are new for KSE.
2002-06-29 07:04:59 +00:00
}
Keep track of threads waiting in kse_release() to avoid a race condition where kse_wakeup() doesn't yet see them in (interruptible) sleep queues. Also add an upcall check to sleepqueue_catch_signals() suggested by jhb. This commit should fix recent mysql hangs. Reviewed by: jhb, davidxu Mysql'd by: Robin P. Blanchard <robin.blanchard at gactr uga edu>
2004-04-28 20:36:53 +00:00
Add thread_find() function to search a thread by lwpid.
2005-11-03 01:34:08 +00:00
struct thread *
thread_find(struct proc *p, lwpid_t tid)
{
struct thread *td;
PROC_LOCK_ASSERT(p, MA_OWNED);
FOREACH_THREAD_IN_PROC(p, td) {
if (td->td_tid == tid)
break;
}
return (td);
}
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
/* Locate a thread by number; return with proc lock held. */
struct thread *
tdfind(lwpid_t tid, pid_t pid)
{
#define RUN_THRESH 16
struct thread *td;
int run = 0;
rw_rlock(&tidhash_lock);
LIST_FOREACH(td, TIDHASH(tid), td_hash) {
if (td->td_tid == tid) {
if (pid != -1 && td->td_proc->p_pid != pid) {
td = NULL;
break;
}
Fix some locking nits with the p_state field of struct proc: - Hold the proc lock while changing the state from PRS_NEW to PRS_NORMAL in fork to honor the locking requirements. While here, expand the scope of the PROC_LOCK() on the new process (p2) to avoid some LORs. Previously the code was locking the new child process (p2) after it had locked the parent process (p1). However, when locking two processes, the safe order is to lock the child first, then the parent. - Fix various places that were checking p_state against PRS_NEW without having the process locked to use PROC_LOCK(). Every place was already locking the process, just after the PRS_NEW check. - Remove or reduce the use of PROC_SLOCK() for places that were checking p_state against PRS_NEW. The PROC_LOCK() alone is sufficient for reading the current state. - Reorder fill_kinfo_proc() slightly so it only acquires PROC_SLOCK() once. MFC after: 1 week
2011-03-24 18:40:11 +00:00
PROC_LOCK(td->td_proc);
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
if (td->td_proc->p_state == PRS_NEW) {
Fix some locking nits with the p_state field of struct proc: - Hold the proc lock while changing the state from PRS_NEW to PRS_NORMAL in fork to honor the locking requirements. While here, expand the scope of the PROC_LOCK() on the new process (p2) to avoid some LORs. Previously the code was locking the new child process (p2) after it had locked the parent process (p1). However, when locking two processes, the safe order is to lock the child first, then the parent. - Fix various places that were checking p_state against PRS_NEW without having the process locked to use PROC_LOCK(). Every place was already locking the process, just after the PRS_NEW check. - Remove or reduce the use of PROC_SLOCK() for places that were checking p_state against PRS_NEW. The PROC_LOCK() alone is sufficient for reading the current state. - Reorder fill_kinfo_proc() slightly so it only acquires PROC_SLOCK() once. MFC after: 1 week
2011-03-24 18:40:11 +00:00
PROC_UNLOCK(td->td_proc);
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
td = NULL;
break;
}
if (run > RUN_THRESH) {
if (rw_try_upgrade(&tidhash_lock)) {
LIST_REMOVE(td, td_hash);
LIST_INSERT_HEAD(TIDHASH(td->td_tid),
td, td_hash);
rw_wunlock(&tidhash_lock);
return (td);
}
}
break;
}
run++;
}
rw_runlock(&tidhash_lock);
return (td);
}
void
tidhash_add(struct thread *td)
{
rw_wlock(&tidhash_lock);
- Insert thread0 into correct thread hash link list. - In thr_exit() and kthread_exit(), only remove thread from hash if it can directly exit, otherwise let exit1() do it. - In thread_suspend_check(), fix cleanup code when thread needs to exit. This change seems fixed the "Bad link elm " panic found by Peter Holm. Stress testing: pho
2010-10-17 11:01:52 +00:00
LIST_INSERT_HEAD(TIDHASH(td->td_tid), td, td_hash);
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
rw_wunlock(&tidhash_lock);
}
void
tidhash_remove(struct thread *td)
{
rw_wlock(&tidhash_lock);
- Insert thread0 into correct thread hash link list. - In thr_exit() and kthread_exit(), only remove thread from hash if it can directly exit, otherwise let exit1() do it. - In thread_suspend_check(), fix cleanup code when thread needs to exit. This change seems fixed the "Bad link elm " panic found by Peter Holm. Stress testing: pho
2010-10-17 11:01:52 +00:00
LIST_REMOVE(td, td_hash);
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
rw_wunlock(&tidhash_lock);
}
Reference in New Issue Copy Permalink