d/freebsd-nq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
a9355dbe82
freebsd-nq/sys/kern/vfs_aio.c

3070 lines
76 KiB
C
Raw Normal View History

/* -> /*- for copyright notices, minor format tweaks as necessary
2005-01-06 23:35:40 +00:00
/*-
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
* Copyright (c) 1997 John S. Dyson. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. John S. Dyson's name may not be used to endorse or promote products
* derived from this software without specific prior written permission.
*
* DISCLAIMER: This code isn't warranted to do anything useful. Anything
* bad that happens because of using this software isn't the responsibility
* of the author. This software is distributed AS-IS.
*/
/*
Finish _POSIX_PRIORITY_SCHEDULING. Needs P1003_1B and _KPOSIX_PRIORITY_SCHEDULING options to work. Changes: Change all "posix4" to "p1003_1b". Misnamed files are left as "posix4" until I'm told if I can simply delete them and add new ones; Add _POSIX_PRIORITY_SCHEDULING system calls for FreeBSD and Linux; Add man pages for _POSIX_PRIORITY_SCHEDULING system calls; Add options to LINT; Minor fixes to P1003_1B code during testing.
1998-03-28 11:51:01 +00:00
* This file contains support for the POSIX 1003.1B AIO/LIO facility.
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
*/
Use __FBSDID().
2003-06-11 00:56:59 +00:00
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
#include "opt_compat.h"
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
#include <sys/param.h>
#include <sys/systm.h>
Lock struct pgrp, session and sigio. New locks are: - pgrpsess_lock which locks the whole pgrps and sessions, - pg_mtx which protects the pgrp members, and - s_mtx which protects the session members. Please refer to sys/proc.h for the coverage of these locks. Changes on the pgrp/session interface: - pgfind() needs the pgrpsess_lock held. - The caller of enterpgrp() is responsible to allocate a new pgrp and session. - Call enterthispgrp() in order to enter an existing pgrp. - pgsignal() requires a pgrp lock held. Reviewed by: jhb, alfred Tested on: cvsup.jp.FreeBSD.org (which is a quad-CPU machine running -current)
2002-02-23 11:12:57 +00:00
#include <sys/malloc.h>
Separate the struct bio related stuff out of <sys/buf.h> into <sys/bio.h>. <sys/bio.h> is now a prerequisite for <sys/buf.h> but it shall not be made a nested include according to bdes teachings on the subject of nested includes. Diskdrivers and similar stuff below specfs::strategy() should no longer need to include <sys/buf.> unless they need caching of data. Still a few bogus uses of struct buf to track down. Repocopy by: peter
2000-05-05 09:59:14 +00:00
#include <sys/bio.h>
Added a used #include (don't depend on "vnode_if.h" including <sys/buf.h>).
1999-02-25 15:54:06 +00:00
#include <sys/buf.h>
Second-to-last commit implementing Capsicum capabilities in the FreeBSD kernel for FreeBSD 9.0: Add a new capability mask argument to fget(9) and friends, allowing system call code to declare what capabilities are required when an integer file descriptor is converted into an in-kernel struct file *. With options CAPABILITIES compiled into the kernel, this enforces capability protection; without, this change is effectively a no-op. Some cases require special handling, such as mmap(2), which must preserve information about the maximum rights at the time of mapping in the memory map so that they can later be enforced in mprotect(2) -- this is done by narrowing the rights in the existing max_protection field used for similar purposes with file permissions. In namei(9), we assert that the code is not reached from within capability mode, as we're not yet ready to enforce namespace capabilities there. This will follow in a later commit. Update two capability names: CAP_EVENT and CAP_KEVENT become CAP_POST_KEVENT and CAP_POLL_KEVENT to more accurately indicate what they represent. Approved by: re (bz) Submitted by: jonathan Sponsored by: Google Inc
2011-08-11 12:30:23 +00:00
#include <sys/capability.h>
Replace the at_fork, at_exec, and at_exit functions with the slightly more flexible process_fork, process_exec, and process_exit eventhandlers. This reduces code duplication and also means that I don't have to go duplicate the eventhandler locking three more times for each of at_fork, at_exec, and at_exit. Reviewed by: phk, jake, almost complete silence on arch@
2003-03-24 21:15:35 +00:00
#include <sys/eventhandler.h>
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
#include <sys/sysproto.h>
#include <sys/filedesc.h>
#include <sys/kernel.h>
Add missing #include <sys/module.h>
2004-05-30 20:34:58 +00:00
#include <sys/module.h>
Use the kthread API to create and destroy AIO daemons. Submitted by: jhb
2001-03-09 06:27:01 +00:00
#include <sys/kthread.h>
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
#include <sys/fcntl.h>
#include <sys/file.h>
Deprecate machine/limits.h in favor of new sys/limits.h. Change all in-tree consumers to include <sys/limits.h> Discussed on: standards@ Partially submitted by: Craig Rodrigues <rodrigc@attbi.com>
2003-04-29 13:36:06 +00:00
#include <sys/limits.h>
Get locking stuff by #including <sys/lock.h> instead of <sys/vnode.h>.
1997-11-18 10:02:40 +00:00
#include <sys/lock.h>
Catch up to moving headers: - machine/ipl.h -> sys/ipl.h - machine/mutex.h -> sys/mutex.h
2000-10-20 07:58:15 +00:00
#include <sys/mutex.h>
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
#include <sys/unistd.h>
Merge posix4/* into normal kernel hierarchy. Reviewed by: glanced at by jhb Approved by: silence on -arch@ and -standards@
2006-11-11 16:26:58 +00:00
#include <sys/posix4.h>
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
#include <sys/proc.h>
Fixed nonsense overflow checking (checking that a long variable is less than INT_MAX after it has possibly overflowed). Removed an unused variable and its associated 2 style bugs. Removed unused includes.
1998-08-17 17:28:10 +00:00
#include <sys/resourcevar.h>
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
#include <sys/signalvar.h>
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
#include <sys/protosw.h>
Switch the vm_object mutex to be a rwlock. This will enable in the future further optimizations where the vm_object lock will be held in read mode most of the time the page cache resident pool of pages are accessed for reading purposes. The change is mostly mechanical but few notes are reported: * The KPI changes as follow: - VM_OBJECT_LOCK() -> VM_OBJECT_WLOCK() - VM_OBJECT_TRYLOCK() -> VM_OBJECT_TRYWLOCK() - VM_OBJECT_UNLOCK() -> VM_OBJECT_WUNLOCK() - VM_OBJECT_LOCK_ASSERT(MA_OWNED) -> VM_OBJECT_ASSERT_WLOCKED() (in order to avoid visibility of implementation details) - The read-mode operations are added: VM_OBJECT_RLOCK(), VM_OBJECT_TRYRLOCK(), VM_OBJECT_RUNLOCK(), VM_OBJECT_ASSERT_RLOCKED(), VM_OBJECT_ASSERT_LOCKED() * The vm/vm_pager.h namespace pollution avoidance (forcing requiring sys/mutex.h in consumers directly to cater its inlining functions using VM_OBJECT_LOCK()) imposes that all the vm/vm_pager.h consumers now must include also sys/rwlock.h. * zfs requires a quite convoluted fix to include FreeBSD rwlocks into the compat layer because the name clash between FreeBSD and solaris versions must be avoided. At this purpose zfs redefines the vm_object locking functions directly, isolating the FreeBSD components in specific compat stubs. The KPI results heavilly broken by this commit. Thirdy part ports must be updated accordingly (I can think off-hand of VirtualBox, for example). Sponsored by: EMC / Isilon storage division Reviewed by: jeff Reviewed by: pjd (ZFS specific review) Discussed with: alc Tested by: pho
2013-03-09 02:32:23 +00:00
#include <sys/rwlock.h>
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
#include <sys/sema.h>
#include <sys/socket.h>
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
#include <sys/socketvar.h>
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
#include <sys/syscall.h>
#include <sys/sysent.h>
Major cleanup and debugging of the new AIO/LIO code. The LIO code is now corrected. New tunables/instrumentation added. The code is now likely "good enough to use." I will add the userland support soon. The "high performance" mode for raw devices is still missing, and will be added next. POSIX system calls that now appear to work: aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, lio_listio. Missing, but to be added soon: aio_fsync.
1997-10-09 04:14:41 +00:00
#include <sys/sysctl.h>
Added used include of <sys/sx.h>. Don't depend on namespace pollution in <sys/file.h> or <sys/socketvar.h>.
2002-03-25 21:52:04 +00:00
#include <sys/sx.h>
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
#include <sys/taskqueue.h>
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
#include <sys/vnode.h>
#include <sys/conf.h>
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
#include <sys/event.h>
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
#include <sys/mount.h>
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
#include <machine/atomic.h>
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
#include <vm/vm.h>
#include <vm/vm_extern.h>
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
#include <vm/pmap.h>
#include <vm/vm_map.h>
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
#include <vm/vm_object.h>
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
#include <vm/uma.h>
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
#include <sys/aio.h>
Clean up some lint associated with the AIO code.
1997-07-17 04:49:43 +00:00
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
#include "opt_vfs_aio.h"
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/*
* Counter for allocating reference ids to new jobs. Wrapped to 1 on
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
* overflow. (XXX will be removed soon.)
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
*/
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
static u_long jobrefid;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
/*
* Counter for aio_fsync.
*/
static uint64_t jobseqno;
#define JOBST_NULL 0
#define JOBST_JOBQSOCK 1
#define JOBST_JOBQGLOBAL 2
#define JOBST_JOBRUNNING 3
#define JOBST_JOBFINISHED 4
#define JOBST_JOBQBUF 5
#define JOBST_JOBQSYNC 6
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
#ifndef MAX_AIO_PER_PROC
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
#define MAX_AIO_PER_PROC 32
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
#endif
#ifndef MAX_AIO_QUEUE_PER_PROC
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
#define MAX_AIO_QUEUE_PER_PROC 256 /* Bigger than AIO_LISTIO_MAX */
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
#endif
#ifndef MAX_AIO_PROCS
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
#define MAX_AIO_PROCS 32
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
#endif
#ifndef MAX_AIO_QUEUE
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
#define MAX_AIO_QUEUE 1024 /* Bigger than AIO_LISTIO_MAX */
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
#endif
#ifndef TARGET_AIO_PROCS
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
#define TARGET_AIO_PROCS 4
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
#endif
#ifndef MAX_BUF_AIO
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
#define MAX_BUF_AIO 16
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
#endif
#ifndef AIOD_TIMEOUT_DEFAULT
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
#define AIOD_TIMEOUT_DEFAULT (10 * hz)
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
#endif
#ifndef AIOD_LIFETIME_DEFAULT
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
#define AIOD_LIFETIME_DEFAULT (30 * hz)
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
#endif
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Use FEATURE() macro to advertise aio availability.
2008-02-01 11:59:14 +00:00
FEATURE(aio, "Asynchronous I/O");
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
static MALLOC_DEFINE(M_LIO, "lio", "listio aio control block list");
Make a SYSCTL_NODE static
2005-02-10 12:23:29 +00:00
static SYSCTL_NODE(_vfs, OID_AUTO, aio, CTLFLAG_RW, 0, "Async IO management");
Major cleanup and debugging of the new AIO/LIO code. The LIO code is now corrected. New tunables/instrumentation added. The code is now likely "good enough to use." I will add the userland support soon. The "high performance" mode for raw devices is still missing, and will be added next. POSIX system calls that now appear to work: aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, lio_listio. Missing, but to be added soon: aio_fsync.
1997-10-09 04:14:41 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
static int max_aio_procs = MAX_AIO_PROCS;
Major cleanup and debugging of the new AIO/LIO code. The LIO code is now corrected. New tunables/instrumentation added. The code is now likely "good enough to use." I will add the userland support soon. The "high performance" mode for raw devices is still missing, and will be added next. POSIX system calls that now appear to work: aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, lio_listio. Missing, but to be added soon: aio_fsync.
1997-10-09 04:14:41 +00:00
SYSCTL_INT(_vfs_aio, OID_AUTO, max_aio_procs,
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
CTLFLAG_RW, &max_aio_procs, 0,
"Maximum number of kernel threads to use for handling async IO ");
Major cleanup and debugging of the new AIO/LIO code. The LIO code is now corrected. New tunables/instrumentation added. The code is now likely "good enough to use." I will add the userland support soon. The "high performance" mode for raw devices is still missing, and will be added next. POSIX system calls that now appear to work: aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, lio_listio. Missing, but to be added soon: aio_fsync.
1997-10-09 04:14:41 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
static int num_aio_procs = 0;
Major cleanup and debugging of the new AIO/LIO code. The LIO code is now corrected. New tunables/instrumentation added. The code is now likely "good enough to use." I will add the userland support soon. The "high performance" mode for raw devices is still missing, and will be added next. POSIX system calls that now appear to work: aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, lio_listio. Missing, but to be added soon: aio_fsync.
1997-10-09 04:14:41 +00:00
SYSCTL_INT(_vfs_aio, OID_AUTO, num_aio_procs,
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
CTLFLAG_RD, &num_aio_procs, 0,
"Number of presently active kernel threads for async IO");
Major cleanup and debugging of the new AIO/LIO code. The LIO code is now corrected. New tunables/instrumentation added. The code is now likely "good enough to use." I will add the userland support soon. The "high performance" mode for raw devices is still missing, and will be added next. POSIX system calls that now appear to work: aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, lio_listio. Missing, but to be added soon: aio_fsync.
1997-10-09 04:14:41 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/*
* The code will adjust the actual number of AIO processes towards this
* number when it gets a chance.
*/
static int target_aio_procs = TARGET_AIO_PROCS;
SYSCTL_INT(_vfs_aio, OID_AUTO, target_aio_procs, CTLFLAG_RW, &target_aio_procs,
0, "Preferred number of ready kernel threads for async IO");
Major cleanup and debugging of the new AIO/LIO code. The LIO code is now corrected. New tunables/instrumentation added. The code is now likely "good enough to use." I will add the userland support soon. The "high performance" mode for raw devices is still missing, and will be added next. POSIX system calls that now appear to work: aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, lio_listio. Missing, but to be added soon: aio_fsync.
1997-10-09 04:14:41 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
static int max_queue_count = MAX_AIO_QUEUE;
SYSCTL_INT(_vfs_aio, OID_AUTO, max_aio_queue, CTLFLAG_RW, &max_queue_count, 0,
"Maximum number of aio requests to queue, globally");
Major cleanup and debugging of the new AIO/LIO code. The LIO code is now corrected. New tunables/instrumentation added. The code is now likely "good enough to use." I will add the userland support soon. The "high performance" mode for raw devices is still missing, and will be added next. POSIX system calls that now appear to work: aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, lio_listio. Missing, but to be added soon: aio_fsync.
1997-10-09 04:14:41 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
static int num_queue_count = 0;
SYSCTL_INT(_vfs_aio, OID_AUTO, num_queue_count, CTLFLAG_RD, &num_queue_count, 0,
"Number of queued aio requests");
Major cleanup and debugging of the new AIO/LIO code. The LIO code is now corrected. New tunables/instrumentation added. The code is now likely "good enough to use." I will add the userland support soon. The "high performance" mode for raw devices is still missing, and will be added next. POSIX system calls that now appear to work: aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, lio_listio. Missing, but to be added soon: aio_fsync.
1997-10-09 04:14:41 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
static int num_buf_aio = 0;
SYSCTL_INT(_vfs_aio, OID_AUTO, num_buf_aio, CTLFLAG_RD, &num_buf_aio, 0,
"Number of aio requests presently handled by the buf subsystem");
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/* Number of async I/O thread in the process of being started */
1) Merge _aio_aqueue and aio_aqueue, check quota in aio_aqueue, so that lio_listio won't exceed the quota. 2) Remove lio_ref_count, it is no longer used.
2006-01-23 02:49:34 +00:00
/* XXX This should be local to aio_aqueue() */
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
static int num_aio_resv_start = 0;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
static int aiod_timeout;
SYSCTL_INT(_vfs_aio, OID_AUTO, aiod_timeout, CTLFLAG_RW, &aiod_timeout, 0,
"Timeout value for synchronous aio operations");
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
static int aiod_lifetime;
SYSCTL_INT(_vfs_aio, OID_AUTO, aiod_lifetime, CTLFLAG_RW, &aiod_lifetime, 0,
"Maximum lifetime for idle aiod");
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
static int unloadable = 0;
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
SYSCTL_INT(_vfs_aio, OID_AUTO, unloadable, CTLFLAG_RW, &unloadable, 0,
"Allow unload of aio (not recommended)");
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
static int max_aio_per_proc = MAX_AIO_PER_PROC;
SYSCTL_INT(_vfs_aio, OID_AUTO, max_aio_per_proc, CTLFLAG_RW, &max_aio_per_proc,
0, "Maximum active aio requests per process (stored in the process)");
static int max_aio_queue_per_proc = MAX_AIO_QUEUE_PER_PROC;
SYSCTL_INT(_vfs_aio, OID_AUTO, max_aio_queue_per_proc, CTLFLAG_RW,
&max_aio_queue_per_proc, 0,
"Maximum queued aio requests per process (stored in the process)");
static int max_buf_aio = MAX_BUF_AIO;
SYSCTL_INT(_vfs_aio, OID_AUTO, max_buf_aio, CTLFLAG_RW, &max_buf_aio, 0,
"Maximum buf aio requests per process (stored in the process)");
Fix sigevent's POSIX incompatible problem by adding member fields sigev_notify_function and sigev_notify_attributes. AIO syscalls use sigevent, so they have to be adjusted. Reviewed by: alc
2005-10-30 02:12:49 +00:00
typedef struct oaiocb {
int aio_fildes; /* File descriptor */
off_t aio_offset; /* File offset for I/O */
volatile void *aio_buf; /* I/O buffer in process space */
size_t aio_nbytes; /* Number of bytes for I/O */
struct osigevent aio_sigevent; /* Signal to deliver */
int aio_lio_opcode; /* LIO opcode */
int aio_reqprio; /* Request priority -- ignored */
struct __aiocb_private _aiocb_private;
} oaiocb_t;
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
/*
* Below is a key of locks used to protect each member of struct aiocblist
* aioliojob and kaioinfo and any backends.
*
* * - need not protected
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
* a - locked by kaioinfo lock
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
* b - locked by backend lock, the backend lock can be null in some cases,
* for example, BIO belongs to this type, in this case, proc lock is
* reused.
* c - locked by aio_job_mtx, the lock for the generic file I/O backend.
*/
/*
* Current, there is only two backends: BIO and generic file I/O.
* socket I/O is served by generic file I/O, this is not a good idea, since
* disk file I/O and any other types without O_NONBLOCK flag can block daemon
* threads, if there is no thread to serve socket I/O, the socket I/O will be
* delayed too long or starved, we should create some threads dedicated to
* sockets to do non-blocking I/O, same for pipe and fifo, for these I/O
* systems we really need non-blocking interface, fiddling O_NONBLOCK in file
* structure is not safe because there is race between userland and aio
* daemons.
*/
o Add missing synchronization (splnet()/splx()) in aio_free_entry(). o Move the definition of struct aiocblist from sys/aio.h to kern/vfs_aio.c. o Make aio_swake_cb() static.
2002-01-06 21:03:39 +00:00
struct aiocblist {
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
TAILQ_ENTRY(aiocblist) list; /* (b) internal list of for backend */
TAILQ_ENTRY(aiocblist) plist; /* (a) list of jobs for each backend */
TAILQ_ENTRY(aiocblist) allist; /* (a) list of all jobs in proc */
int jobflags; /* (a) job flags */
int jobstate; /* (b) job state */
int inputcharge; /* (*) input blockes */
int outputcharge; /* (*) output blockes */
struct buf *bp; /* (*) private to BIO backend,
* buffer pointer
*/
struct proc *userproc; /* (*) user process */
struct ucred *cred; /* (*) active credential when created */
struct file *fd_file; /* (*) pointer to file structure */
struct aioliojob *lio; /* (*) optional lio job */
struct aiocb *uuaiocb; /* (*) pointer in userspace of aiocb */
struct knlist klist; /* (a) list of knotes */
struct aiocb uaiocb; /* (*) kernel I/O control block */
ksiginfo_t ksi; /* (a) realtime signal info */
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
struct task biotask; /* (*) private to BIO backend */
uint64_t seqno; /* (*) job number */
int pending; /* (a) number of pending I/O, aio_fsync only */
o Add missing synchronization (splnet()/splx()) in aio_free_entry(). o Move the definition of struct aiocblist from sys/aio.h to kern/vfs_aio.c. o Make aio_swake_cb() static.
2002-01-06 21:03:39 +00:00
};
/* jobflags */
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
#define AIOCBLIST_DONE 0x01
#define AIOCBLIST_BUFDONE 0x02
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
#define AIOCBLIST_RUNDOWN 0x04
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
#define AIOCBLIST_CHECKSYNC 0x08
o Add missing synchronization (splnet()/splx()) in aio_free_entry(). o Move the definition of struct aiocblist from sys/aio.h to kern/vfs_aio.c. o Make aio_swake_cb() static.
2002-01-06 21:03:39 +00:00
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
/*
* AIO process info
*/
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
#define AIOP_FREE 0x1 /* proc on free queue */
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct aiothreadlist {
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
int aiothreadflags; /* (c) AIO proc flags */
TAILQ_ENTRY(aiothreadlist) list; /* (c) list of processes */
struct thread *aiothread; /* (*) the AIO thread */
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
};
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
/*
* data-structure for lio signal management
*/
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
struct aioliojob {
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
int lioj_flags; /* (a) listio flags */
int lioj_count; /* (a) listio flags */
int lioj_finished_count; /* (a) listio flags */
struct sigevent lioj_signal; /* (a) signal on all I/O done */
TAILQ_ENTRY(aioliojob) lioj_list; /* (a) lio list */
struct knlist klist; /* (a) list of knotes */
ksiginfo_t lioj_ksi; /* (a) Realtime signal info */
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
};
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
#define LIOJ_SIGNAL 0x1 /* signal on all done (lio) */
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
#define LIOJ_SIGNAL_POSTED 0x2 /* signal has been posted */
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
#define LIOJ_KEVENT_POSTED 0x4 /* kevent triggered */
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
/*
* per process aio data structure
*/
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct kaioinfo {
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
struct mtx kaio_mtx; /* the lock to protect this struct */
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
int kaio_flags; /* (a) per process kaio flags */
int kaio_maxactive_count; /* (*) maximum number of AIOs */
int kaio_active_count; /* (c) number of currently used AIOs */
int kaio_qallowed_count; /* (*) maxiumu size of AIO queue */
int kaio_count; /* (a) size of AIO queue */
int kaio_ballowed_count; /* (*) maximum number of buffers */
int kaio_buffer_count; /* (a) number of physio buffers */
TAILQ_HEAD(,aiocblist) kaio_all; /* (a) all AIOs in the process */
TAILQ_HEAD(,aiocblist) kaio_done; /* (a) done queue for process */
TAILQ_HEAD(,aioliojob) kaio_liojoblist; /* (a) list of lio jobs */
TAILQ_HEAD(,aiocblist) kaio_jobqueue; /* (a) job queue for process */
TAILQ_HEAD(,aiocblist) kaio_bufqueue; /* (a) buffer job queue for process */
TAILQ_HEAD(,aiocblist) kaio_sockqueue; /* (a) queue for aios waiting on sockets,
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
* NOT USED YET.
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
*/
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
TAILQ_HEAD(,aiocblist) kaio_syncqueue; /* (a) queue for aio_fsync */
struct task kaio_task; /* (*) task to kick aio threads */
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
};
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
#define AIO_LOCK(ki) mtx_lock(&(ki)->kaio_mtx)
#define AIO_UNLOCK(ki) mtx_unlock(&(ki)->kaio_mtx)
#define AIO_LOCK_ASSERT(ki, f) mtx_assert(&(ki)->kaio_mtx, (f))
#define AIO_MTX(ki) (&(ki)->kaio_mtx)
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
#define KAIO_RUNDOWN 0x1 /* process is being run down */
#define KAIO_WAKEUP 0x2 /* wakeup process when there is a significant event */
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
/*
* Operations used to interact with userland aio control blocks.
* Different ABIs provide their own operations.
*/
struct aiocb_ops {
int (*copyin)(struct aiocb *ujob, struct aiocb *kjob);
long (*fetch_status)(struct aiocb *ujob);
long (*fetch_error)(struct aiocb *ujob);
int (*store_status)(struct aiocb *ujob, long status);
int (*store_error)(struct aiocb *ujob, long error);
int (*store_kernelinfo)(struct aiocb *ujob, long jobref);
int (*store_aiocb)(struct aiocb **ujobp, struct aiocb *ujob);
};
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
static TAILQ_HEAD(,aiothreadlist) aio_freeproc; /* (c) Idle daemons */
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
static struct sema aio_newproc_sem;
static struct mtx aio_job_mtx;
static struct mtx aio_sock_mtx;
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
static TAILQ_HEAD(,aiocblist) aio_jobs; /* (c) Async job list */
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
static struct unrhdr *aiod_unr;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
MFP4 (with some minor changes): Implement the linux_io_* syscalls (AIO). They are only enabled if the native AIO code is available (either compiled in to the kernel or as a module) at the time the functions are used. If the AIO stuff is not available there will be a ENOSYS. From the submitter: ---snip--- DESIGN NOTES: 1. Linux permits a process to own multiple AIO queues (distinguished by "context"), but FreeBSD creates only one single AIO queue per process. My code maintains a request queue (STAILQ of queue(3)) per "context", and throws all AIO requests of all contexts owned by a process into the single FreeBSD per-process AIO queue. When the process calls io_destroy(2), io_getevents(2), io_submit(2) and io_cancel(2), my code can pick out requests owned by the specified context from the single FreeBSD per-process AIO queue according to the per-context request queues maintained by my code. 2. The request queue maintained by my code stores contrast information between Linux IO control blocks (struct linux_iocb) and FreeBSD IO control blocks (struct aiocb). FreeBSD IO control block actually exists in userland memory space, required by FreeBSD native aio_XXXXXX(2). 3. It is quite troubling that the function io_getevents() of libaio-0.3.105 needs to use Linux-specific "struct aio_ring", which is a partial mirror of context in user space. I would rather take the address of context in kernel as the context ID, but the io_getevents() of libaio forces me to take the address of the "ring" in user space as the context ID. To my surprise, one comment line in the file "io_getevents.c" of libaio-0.3.105 reads: Ben will hate me for this REFERENCE: 1. Linux kernel source code: http://www.kernel.org/pub/linux/kernel/v2.6/ (include/linux/aio_abi.h, fs/aio.c) 2. Linux manual pages: http://www.kernel.org/pub/linux/docs/manpages/ (io_setup(2), io_destroy(2), io_getevents(2), io_submit(2), io_cancel(2)) 3. Linux Scalability Effort: http://lse.sourceforge.net/io/aio.html The design notes: http://lse.sourceforge.net/io/aionotes.txt 4. The package libaio, both source and binary: http://rpmfind.net/linux/rpm2html/search.php?query=libaio Simple transparent interface to Linux AIO system calls. 5. Libaio-oracle: http://oss.oracle.com/projects/libaio-oracle/ POSIX AIO implementation based on Linux AIO system calls (depending on libaio). ---snip--- Submitted by: Li, Xiao <intron@intron.ac>
2006-10-15 14:22:14 +00:00
void aio_init_aioinfo(struct proc *p);
Convert aio syscall registration to SYSCALL_INIT_HELPER. Reviewed by: jhb MFC after: 2 weeks
2010-03-19 11:11:34 +00:00
static int aio_onceonly(void);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
static int aio_free_entry(struct aiocblist *aiocbe);
Separate LIO_SYNC processing into a separate function aio_process_sync(), and rename aio_process() into aio_process_rw(). Reviewed by: kib Sponsored by: Nginx, Inc.
2013-06-08 13:02:43 +00:00
static void aio_process_rw(struct aiocblist *aiocbe);
static void aio_process_sync(struct aiocblist *aiocbe);
Add new system call - aio_mlock(). The name speaks for itself. It allows to perform the mlock(2) operation, which can consume a lot of time, under control of aio(4). Reviewed by: kib, jilles Sponsored by: Nginx, Inc.
2013-06-08 13:27:57 +00:00
static void aio_process_mlock(struct aiocblist *aiocbe);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
static int aio_newproc(int *);
MFP4 (with some minor changes): Implement the linux_io_* syscalls (AIO). They are only enabled if the native AIO code is available (either compiled in to the kernel or as a module) at the time the functions are used. If the AIO stuff is not available there will be a ENOSYS. From the submitter: ---snip--- DESIGN NOTES: 1. Linux permits a process to own multiple AIO queues (distinguished by "context"), but FreeBSD creates only one single AIO queue per process. My code maintains a request queue (STAILQ of queue(3)) per "context", and throws all AIO requests of all contexts owned by a process into the single FreeBSD per-process AIO queue. When the process calls io_destroy(2), io_getevents(2), io_submit(2) and io_cancel(2), my code can pick out requests owned by the specified context from the single FreeBSD per-process AIO queue according to the per-context request queues maintained by my code. 2. The request queue maintained by my code stores contrast information between Linux IO control blocks (struct linux_iocb) and FreeBSD IO control blocks (struct aiocb). FreeBSD IO control block actually exists in userland memory space, required by FreeBSD native aio_XXXXXX(2). 3. It is quite troubling that the function io_getevents() of libaio-0.3.105 needs to use Linux-specific "struct aio_ring", which is a partial mirror of context in user space. I would rather take the address of context in kernel as the context ID, but the io_getevents() of libaio forces me to take the address of the "ring" in user space as the context ID. To my surprise, one comment line in the file "io_getevents.c" of libaio-0.3.105 reads: Ben will hate me for this REFERENCE: 1. Linux kernel source code: http://www.kernel.org/pub/linux/kernel/v2.6/ (include/linux/aio_abi.h, fs/aio.c) 2. Linux manual pages: http://www.kernel.org/pub/linux/docs/manpages/ (io_setup(2), io_destroy(2), io_getevents(2), io_submit(2), io_cancel(2)) 3. Linux Scalability Effort: http://lse.sourceforge.net/io/aio.html The design notes: http://lse.sourceforge.net/io/aionotes.txt 4. The package libaio, both source and binary: http://rpmfind.net/linux/rpm2html/search.php?query=libaio Simple transparent interface to Linux AIO system calls. 5. Libaio-oracle: http://oss.oracle.com/projects/libaio-oracle/ POSIX AIO implementation based on Linux AIO system calls (depending on libaio). ---snip--- Submitted by: Li, Xiao <intron@intron.ac>
2006-10-15 14:22:14 +00:00
int aio_aqueue(struct thread *td, struct aiocb *job,
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
struct aioliojob *lio, int type, struct aiocb_ops *ops);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
static void aio_physwakeup(struct buf *bp);
Replace the at_fork, at_exec, and at_exit functions with the slightly more flexible process_fork, process_exec, and process_exit eventhandlers. This reduces code duplication and also means that I don't have to go duplicate the eventhandler locking three more times for each of at_fork, at_exec, and at_exit. Reviewed by: phk, jake, almost complete silence on arch@
2003-03-24 21:15:35 +00:00
static void aio_proc_rundown(void *arg, struct proc *p);
- Change process_exec function handlers prototype to include struct image_params arg. - Change struct image_params to include struct sysentvec pointer and initialize it. - Change all consumers of process_exit/process_exec eventhandlers to new prototypes (includes splitting up into distinct exec/exit functions). - Add eventhandler to userret. Sponsored by: Google SoC 2006 Submitted by: rdivacky Parts suggested by: jhb (on hackers@)
2006-08-15 12:10:57 +00:00
static void aio_proc_rundown_exec(void *arg, struct proc *p, struct image_params *imgp);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
static int aio_qphysio(struct proc *p, struct aiocblist *iocb);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
static void biohelper(void *, int);
static void aio_daemon(void *param);
o Add missing synchronization (splnet()/splx()) in aio_free_entry(). o Move the definition of struct aiocblist from sys/aio.h to kern/vfs_aio.c. o Make aio_swake_cb() static.
2002-01-06 21:03:39 +00:00
static void aio_swake_cb(struct socket *, struct sockbuf *);
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
static int aio_unload(void);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
static void aio_bio_done_notify(struct proc *userp, struct aiocblist *aiocbe, int type);
#define DONE_BUF 1
#define DONE_QUEUE 2
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
static int aio_kick(struct proc *userp);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
static void aio_kick_nowait(struct proc *userp);
static void aio_kick_helper(void *context, int pending);
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
static int filt_aioattach(struct knote *kn);
static void filt_aiodetach(struct knote *kn);
static int filt_aio(struct knote *kn, long hint);
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
static int filt_lioattach(struct knote *kn);
static void filt_liodetach(struct knote *kn);
static int filt_lio(struct knote *kn, long hint);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/*
* Zones for:
* kaio Per process async io info
* aiop async io thread data
* aiocb async io jobs
* aiol list io job pointer - internal to aio_suspend XXX
* aiolio list io jobs
*/
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
static uma_zone_t kaio_zone, aiop_zone, aiocb_zone, aiol_zone, aiolio_zone;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/* kqueue filters for aio */
Use C99 initialization for struct filterops. Obtained from: Mac OS X Sponsored by: Apple Inc. MFC after: 3 weeks
2009-09-12 20:03:45 +00:00
static struct filterops aio_filtops = {
.f_isfd = 0,
.f_attach = filt_aioattach,
.f_detach = filt_aiodetach,
.f_event = filt_aio,
};
static struct filterops lio_filtops = {
.f_isfd = 0,
.f_attach = filt_lioattach,
.f_detach = filt_liodetach,
.f_event = filt_lio
};
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
Replace the at_fork, at_exec, and at_exit functions with the slightly more flexible process_fork, process_exec, and process_exit eventhandlers. This reduces code duplication and also means that I don't have to go duplicate the eventhandler locking three more times for each of at_fork, at_exec, and at_exit. Reviewed by: phk, jake, almost complete silence on arch@
2003-03-24 21:15:35 +00:00
static eventhandler_tag exit_tag, exec_tag;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TASKQUEUE_DEFINE_THREAD(aiod_bio);
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/*
* Main operations function for use as a kernel module.
*/
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
static int
aio_modload(struct module *module, int cmd, void *arg)
{
int error = 0;
switch (cmd) {
case MOD_LOAD:
aio_onceonly();
break;
case MOD_UNLOAD:
error = aio_unload();
break;
case MOD_SHUTDOWN:
break;
default:
error = EINVAL;
break;
}
return (error);
}
static moduledata_t aio_mod = {
"aio",
&aio_modload,
NULL
};
Convert aio syscall registration to SYSCALL_INIT_HELPER. Reviewed by: jhb MFC after: 2 weeks
2010-03-19 11:11:34 +00:00
static struct syscall_helper_data aio_syscalls[] = {
SYSCALL_INIT_HELPER(aio_cancel),
SYSCALL_INIT_HELPER(aio_error),
SYSCALL_INIT_HELPER(aio_fsync),
Add new system call - aio_mlock(). The name speaks for itself. It allows to perform the mlock(2) operation, which can consume a lot of time, under control of aio(4). Reviewed by: kib, jilles Sponsored by: Nginx, Inc.
2013-06-08 13:27:57 +00:00
SYSCALL_INIT_HELPER(aio_mlock),
Convert aio syscall registration to SYSCALL_INIT_HELPER. Reviewed by: jhb MFC after: 2 weeks
2010-03-19 11:11:34 +00:00
SYSCALL_INIT_HELPER(aio_read),
SYSCALL_INIT_HELPER(aio_return),
SYSCALL_INIT_HELPER(aio_suspend),
SYSCALL_INIT_HELPER(aio_waitcomplete),
SYSCALL_INIT_HELPER(aio_write),
SYSCALL_INIT_HELPER(lio_listio),
SYSCALL_INIT_HELPER(oaio_read),
SYSCALL_INIT_HELPER(oaio_write),
SYSCALL_INIT_HELPER(olio_listio),
SYSCALL_INIT_LAST
};
#ifdef COMPAT_FREEBSD32
#include <sys/mount.h>
#include <sys/socket.h>
#include <compat/freebsd32/freebsd32.h>
#include <compat/freebsd32/freebsd32_proto.h>
#include <compat/freebsd32/freebsd32_signal.h>
#include <compat/freebsd32/freebsd32_syscall.h>
#include <compat/freebsd32/freebsd32_util.h>
static struct syscall_helper_data aio32_syscalls[] = {
SYSCALL32_INIT_HELPER(freebsd32_aio_return),
SYSCALL32_INIT_HELPER(freebsd32_aio_suspend),
SYSCALL32_INIT_HELPER(freebsd32_aio_cancel),
SYSCALL32_INIT_HELPER(freebsd32_aio_error),
SYSCALL32_INIT_HELPER(freebsd32_aio_fsync),
Add new system call - aio_mlock(). The name speaks for itself. It allows to perform the mlock(2) operation, which can consume a lot of time, under control of aio(4). Reviewed by: kib, jilles Sponsored by: Nginx, Inc.
2013-06-08 13:27:57 +00:00
SYSCALL32_INIT_HELPER(freebsd32_aio_mlock),
Convert aio syscall registration to SYSCALL_INIT_HELPER. Reviewed by: jhb MFC after: 2 weeks
2010-03-19 11:11:34 +00:00
SYSCALL32_INIT_HELPER(freebsd32_aio_read),
SYSCALL32_INIT_HELPER(freebsd32_aio_write),
SYSCALL32_INIT_HELPER(freebsd32_aio_waitcomplete),
SYSCALL32_INIT_HELPER(freebsd32_lio_listio),
SYSCALL32_INIT_HELPER(freebsd32_oaio_read),
SYSCALL32_INIT_HELPER(freebsd32_oaio_write),
SYSCALL32_INIT_HELPER(freebsd32_olio_listio),
SYSCALL_INIT_LAST
};
#endif
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
DECLARE_MODULE(aio, aio_mod,
SI_SUB_VFS, SI_ORDER_ANY);
MODULE_VERSION(aio, 1);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
/*
* Startup initialization
*/
Convert aio syscall registration to SYSCALL_INIT_HELPER. Reviewed by: jhb MFC after: 2 weeks
2010-03-19 11:11:34 +00:00
static int
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
aio_onceonly(void)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
Convert aio syscall registration to SYSCALL_INIT_HELPER. Reviewed by: jhb MFC after: 2 weeks
2010-03-19 11:11:34 +00:00
int error;
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
/* XXX: should probably just use so->callback */
aio_swake = &aio_swake_cb;
Replace the at_fork, at_exec, and at_exit functions with the slightly more flexible process_fork, process_exec, and process_exit eventhandlers. This reduces code duplication and also means that I don't have to go duplicate the eventhandler locking three more times for each of at_fork, at_exec, and at_exit. Reviewed by: phk, jake, almost complete silence on arch@
2003-03-24 21:15:35 +00:00
exit_tag = EVENTHANDLER_REGISTER(process_exit, aio_proc_rundown, NULL,
EVENTHANDLER_PRI_ANY);
- Change process_exec function handlers prototype to include struct image_params arg. - Change struct image_params to include struct sysentvec pointer and initialize it. - Change all consumers of process_exit/process_exec eventhandlers to new prototypes (includes splitting up into distinct exec/exit functions). - Add eventhandler to userret. Sponsored by: Google SoC 2006 Submitted by: rdivacky Parts suggested by: jhb (on hackers@)
2006-08-15 12:10:57 +00:00
exec_tag = EVENTHANDLER_REGISTER(process_exec, aio_proc_rundown_exec, NULL,
Replace the at_fork, at_exec, and at_exit functions with the slightly more flexible process_fork, process_exec, and process_exit eventhandlers. This reduces code duplication and also means that I don't have to go duplicate the eventhandler locking three more times for each of at_fork, at_exec, and at_exit. Reviewed by: phk, jake, almost complete silence on arch@
2003-03-24 21:15:35 +00:00
EVENTHANDLER_PRI_ANY);
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
kqueue_add_filteropts(EVFILT_AIO, &aio_filtops);
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
kqueue_add_filteropts(EVFILT_LIO, &lio_filtops);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
TAILQ_INIT(&aio_freeproc);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
sema_init(&aio_newproc_sem, 0, "aio_new_proc");
mtx_init(&aio_job_mtx, "aio_job", NULL, MTX_DEF);
mtx_init(&aio_sock_mtx, "aio_sock", NULL, MTX_DEF);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
TAILQ_INIT(&aio_jobs);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
aiod_unr = new_unrhdr(1, INT_MAX, NULL);
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
kaio_zone = uma_zcreate("AIO", sizeof(struct kaioinfo), NULL, NULL,
NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_NOFREE);
aiop_zone = uma_zcreate("AIOP", sizeof(struct aiothreadlist), NULL,
NULL, NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_NOFREE);
aiocb_zone = uma_zcreate("AIOCB", sizeof(struct aiocblist), NULL, NULL,
NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_NOFREE);
aiol_zone = uma_zcreate("AIOL", AIO_LISTIO_MAX*sizeof(intptr_t) , NULL,
NULL, NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_NOFREE);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
aiolio_zone = uma_zcreate("AIOLIO", sizeof(struct aioliojob), NULL,
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
NULL, NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_NOFREE);
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
aiod_timeout = AIOD_TIMEOUT_DEFAULT;
aiod_lifetime = AIOD_LIFETIME_DEFAULT;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
jobrefid = 1;
Change the way support for asynchronous I/O is indicated to applications to conform to 1003.1-2001. Make it possible for applications to actually tell whether or not asynchronous I/O is supported. Since FreeBSD's aio implementation works on all descriptor types, don't call down into file or vnode ops when [f]pathconf() is asked about _PC_ASYNC_IO; this avoids the need for every file and vnode op to know about it.
2002-10-27 18:07:41 +00:00
async_io_version = _POSIX_VERSION;
Call 'p31b_setcfg(CTL_P1003_1B_AIO_LISTIO_MAX, AIO_LISTIO_MAX)' when AIO is initialized so that sysconf() gives correct results. Reported by: Craig Rodrigues <rodrigc@attbi.com>
2002-11-16 04:22:55 +00:00
p31b_setcfg(CTL_P1003_1B_AIO_LISTIO_MAX, AIO_LISTIO_MAX);
Export the values for _SC_AIO_MAX and _SC_AIO_PRIO_DELTA_MAX via the p1003b sysctl interface.
2002-11-16 06:38:07 +00:00
p31b_setcfg(CTL_P1003_1B_AIO_MAX, MAX_AIO_QUEUE);
p31b_setcfg(CTL_P1003_1B_AIO_PRIO_DELTA_MAX, 0);
Convert aio syscall registration to SYSCALL_INIT_HELPER. Reviewed by: jhb MFC after: 2 weeks
2010-03-19 11:11:34 +00:00
error = syscall_helper_register(aio_syscalls);
if (error)
return (error);
#ifdef COMPAT_FREEBSD32
error = syscall32_helper_register(aio32_syscalls);
if (error)
return (error);
#endif
return (0);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/*
* Callback for unload of AIO when used as a module.
*/
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
static int
aio_unload(void)
{
Add locking to the kqueue subsystem. This also makes the kqueue subsystem a more complete subsystem, and removes the knowlege of how things are implemented from the drivers. Include locking around filter ops, so a module like aio will know when not to be unloaded if there are outstanding knotes using it's filter ops. Currently, it uses the MTX_DUPOK even though it is not always safe to aquire duplicate locks. Witness currently doesn't support the ability to discover if a dup lock is ok (in some cases). Reviewed by: green, rwatson (both earlier versions)
2004-08-15 06:24:42 +00:00
int error;
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
/*
* XXX: no unloads by default, it's too dangerous.
* perhaps we could do it if locked out callers and then
* did an aio_proc_rundown() on each process.
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
*
* jhb: aio_proc_rundown() needs to run on curproc though,
* so I don't think that would fly.
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
*/
if (!unloadable)
return (EOPNOTSUPP);
Convert aio syscall registration to SYSCALL_INIT_HELPER. Reviewed by: jhb MFC after: 2 weeks
2010-03-19 11:11:34 +00:00
#ifdef COMPAT_FREEBSD32
syscall32_helper_unregister(aio32_syscalls);
#endif
syscall_helper_unregister(aio_syscalls);
Add locking to the kqueue subsystem. This also makes the kqueue subsystem a more complete subsystem, and removes the knowlege of how things are implemented from the drivers. Include locking around filter ops, so a module like aio will know when not to be unloaded if there are outstanding knotes using it's filter ops. Currently, it uses the MTX_DUPOK even though it is not always safe to aquire duplicate locks. Witness currently doesn't support the ability to discover if a dup lock is ok (in some cases). Reviewed by: green, rwatson (both earlier versions)
2004-08-15 06:24:42 +00:00
error = kqueue_del_filteropts(EVFILT_AIO);
More cleanup for aio code: 1) unregsiter kqueue filter for EVFILT_LIO. 2) free uma_zones. 3) call setsid directly to enter another session rather than implementing by itself. Submitted by: jhb
2006-01-24 02:46:15 +00:00
if (error)
return error;
error = kqueue_del_filteropts(EVFILT_LIO);
Add locking to the kqueue subsystem. This also makes the kqueue subsystem a more complete subsystem, and removes the knowlege of how things are implemented from the drivers. Include locking around filter ops, so a module like aio will know when not to be unloaded if there are outstanding knotes using it's filter ops. Currently, it uses the MTX_DUPOK even though it is not always safe to aquire duplicate locks. Witness currently doesn't support the ability to discover if a dup lock is ok (in some cases). Reviewed by: green, rwatson (both earlier versions)
2004-08-15 06:24:42 +00:00
if (error)
return error;
Change the way support for asynchronous I/O is indicated to applications to conform to 1003.1-2001. Make it possible for applications to actually tell whether or not asynchronous I/O is supported. Since FreeBSD's aio implementation works on all descriptor types, don't call down into file or vnode ops when [f]pathconf() is asked about _PC_ASYNC_IO; this avoids the need for every file and vnode op to know about it.
2002-10-27 18:07:41 +00:00
async_io_version = 0;
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
aio_swake = NULL;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
taskqueue_free(taskqueue_aiod_bio);
delete_unrhdr(aiod_unr);
More cleanup for aio code: 1) unregsiter kqueue filter for EVFILT_LIO. 2) free uma_zones. 3) call setsid directly to enter another session rather than implementing by itself. Submitted by: jhb
2006-01-24 02:46:15 +00:00
uma_zdestroy(kaio_zone);
uma_zdestroy(aiop_zone);
uma_zdestroy(aiocb_zone);
uma_zdestroy(aiol_zone);
uma_zdestroy(aiolio_zone);
Replace the at_fork, at_exec, and at_exit functions with the slightly more flexible process_fork, process_exec, and process_exit eventhandlers. This reduces code duplication and also means that I don't have to go duplicate the eventhandler locking three more times for each of at_fork, at_exec, and at_exit. Reviewed by: phk, jake, almost complete silence on arch@
2003-03-24 21:15:35 +00:00
EVENTHANDLER_DEREGISTER(process_exit, exit_tag);
EVENTHANDLER_DEREGISTER(process_exec, exec_tag);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_destroy(&aio_job_mtx);
mtx_destroy(&aio_sock_mtx);
sema_destroy(&aio_newproc_sem);
Rework the sysconf(3) interaction with aio: sysconf.c: Use 'break' rather than 'goto yesno' in sysconf.c so that we report a '0' return value from the kernel sysctl. vfs_aio.c: Make aio reset its configuration parameters to -1 after unloading instead of 0. posix4_mib.c: Initialize the aio configuration parameters to -1 to indicate that it is not loaded. Add a facility (p31b_iscfg()) to determine if a posix4 facility has been initialized to avoid having to re-order the SYSINITs. Use p31b_iscfg() to determine if aio has had a chance to run yet which is likely if it is compiled into the kernel and avoid spamming its values. Introduce a macro P31B_VALID() instead of doing the same comparison over and over. posix4.h: Prototype p31b_iscfg().
2002-11-17 04:15:34 +00:00
p31b_setcfg(CTL_P1003_1B_AIO_LISTIO_MAX, -1);
p31b_setcfg(CTL_P1003_1B_AIO_MAX, -1);
p31b_setcfg(CTL_P1003_1B_AIO_PRIO_DELTA_MAX, -1);
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
return (0);
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Init the per-process aioinfo structure. The aioinfo limits are set
* per-process for user limit (resource) management.
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
*/
MFP4 (with some minor changes): Implement the linux_io_* syscalls (AIO). They are only enabled if the native AIO code is available (either compiled in to the kernel or as a module) at the time the functions are used. If the AIO stuff is not available there will be a ENOSYS. From the submitter: ---snip--- DESIGN NOTES: 1. Linux permits a process to own multiple AIO queues (distinguished by "context"), but FreeBSD creates only one single AIO queue per process. My code maintains a request queue (STAILQ of queue(3)) per "context", and throws all AIO requests of all contexts owned by a process into the single FreeBSD per-process AIO queue. When the process calls io_destroy(2), io_getevents(2), io_submit(2) and io_cancel(2), my code can pick out requests owned by the specified context from the single FreeBSD per-process AIO queue according to the per-context request queues maintained by my code. 2. The request queue maintained by my code stores contrast information between Linux IO control blocks (struct linux_iocb) and FreeBSD IO control blocks (struct aiocb). FreeBSD IO control block actually exists in userland memory space, required by FreeBSD native aio_XXXXXX(2). 3. It is quite troubling that the function io_getevents() of libaio-0.3.105 needs to use Linux-specific "struct aio_ring", which is a partial mirror of context in user space. I would rather take the address of context in kernel as the context ID, but the io_getevents() of libaio forces me to take the address of the "ring" in user space as the context ID. To my surprise, one comment line in the file "io_getevents.c" of libaio-0.3.105 reads: Ben will hate me for this REFERENCE: 1. Linux kernel source code: http://www.kernel.org/pub/linux/kernel/v2.6/ (include/linux/aio_abi.h, fs/aio.c) 2. Linux manual pages: http://www.kernel.org/pub/linux/docs/manpages/ (io_setup(2), io_destroy(2), io_getevents(2), io_submit(2), io_cancel(2)) 3. Linux Scalability Effort: http://lse.sourceforge.net/io/aio.html The design notes: http://lse.sourceforge.net/io/aionotes.txt 4. The package libaio, both source and binary: http://rpmfind.net/linux/rpm2html/search.php?query=libaio Simple transparent interface to Linux AIO system calls. 5. Libaio-oracle: http://oss.oracle.com/projects/libaio-oracle/ POSIX AIO implementation based on Linux AIO system calls (depending on libaio). ---snip--- Submitted by: Li, Xiao <intron@intron.ac>
2006-10-15 14:22:14 +00:00
void
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
aio_init_aioinfo(struct proc *p)
{
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct kaioinfo *ki;
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
Use the proc mtx to prevent simultaneous changes to p_aioinfo.
2005-05-30 19:33:33 +00:00
ki = uma_zalloc(kaio_zone, M_WAITOK);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
mtx_init(&ki->kaio_mtx, "aiomtx", NULL, MTX_DEF);
Use the proc mtx to prevent simultaneous changes to p_aioinfo.
2005-05-30 19:33:33 +00:00
ki->kaio_flags = 0;
ki->kaio_maxactive_count = max_aio_per_proc;
ki->kaio_active_count = 0;
ki->kaio_qallowed_count = max_aio_queue_per_proc;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
ki->kaio_count = 0;
Use the proc mtx to prevent simultaneous changes to p_aioinfo.
2005-05-30 19:33:33 +00:00
ki->kaio_ballowed_count = max_buf_aio;
ki->kaio_buffer_count = 0;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TAILQ_INIT(&ki->kaio_all);
TAILQ_INIT(&ki->kaio_done);
Use the proc mtx to prevent simultaneous changes to p_aioinfo.
2005-05-30 19:33:33 +00:00
TAILQ_INIT(&ki->kaio_jobqueue);
TAILQ_INIT(&ki->kaio_bufqueue);
TAILQ_INIT(&ki->kaio_liojoblist);
TAILQ_INIT(&ki->kaio_sockqueue);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
TAILQ_INIT(&ki->kaio_syncqueue);
TASK_INIT(&ki->kaio_task, 0, aio_kick_helper, p);
Use the proc mtx to prevent simultaneous changes to p_aioinfo.
2005-05-30 19:33:33 +00:00
PROC_LOCK(p);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
if (p->p_aioinfo == NULL) {
p->p_aioinfo = ki;
Use the proc mtx to prevent simultaneous changes to p_aioinfo.
2005-05-30 19:33:33 +00:00
PROC_UNLOCK(p);
} else {
PROC_UNLOCK(p);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
mtx_destroy(&ki->kaio_mtx);
Use the proc mtx to prevent simultaneous changes to p_aioinfo.
2005-05-30 19:33:33 +00:00
uma_zfree(kaio_zone, ki);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
clean up whitespace...
2004-08-13 17:43:53 +00:00
Use minimum of max_aio_procs and target_aio_procs when spawning new aiod since there should be no more then max_aio_procs processes.
2008-06-21 11:34:34 +00:00
while (num_aio_procs < MIN(target_aio_procs, max_aio_procs))
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
aio_newproc(NULL);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Support sending realtime signal information via signal queue, realtime signal memory is pre-allocated, so kernel can always notify user code.
2005-11-03 05:25:26 +00:00
static int
aio_sendsig(struct proc *p, struct sigevent *sigev, ksiginfo_t *ksi)
{
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
struct thread *td;
int error;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
error = sigev_findtd(p, sigev, &td);
if (error)
return (error);
Support sending realtime signal information via signal queue, realtime signal memory is pre-allocated, so kernel can always notify user code.
2005-11-03 05:25:26 +00:00
if (!KSI_ONQ(ksi)) {
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
ksiginfo_set_sigev(ksi, sigev);
Support sending realtime signal information via signal queue, realtime signal memory is pre-allocated, so kernel can always notify user code.
2005-11-03 05:25:26 +00:00
ksi->ksi_code = SI_ASYNCIO;
ksi->ksi_flags |= KSI_EXT | KSI_INS;
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
tdsendsignal(p, td, ksi->ksi_signo, ksi);
Support sending realtime signal information via signal queue, realtime signal memory is pre-allocated, so kernel can always notify user code.
2005-11-03 05:25:26 +00:00
}
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
PROC_UNLOCK(p);
Create a global thread hash table to speed up thread lookup, use rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
2010-10-09 02:50:23 +00:00
return (error);
Support sending realtime signal information via signal queue, realtime signal memory is pre-allocated, so kernel can always notify user code.
2005-11-03 05:25:26 +00:00
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Free a job entry. Wait for completion if it is currently active, but don't
* delay forever. If we delay, we return a flag that says that we have to
* restart the queue scan.
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
*/
Eliminate the aio_freejobs list. Its purpose was to store free aiocb's allocated by zalloc(). In other words, zfree() was never called. Now, we call zfree(). Why eliminate this micro- optimization? At some later point, when we multithread the AIO system, we would need a mutex to synchronize access to aio_freejobs, making its use nearly indistinguishable in cost from zalloc() and zfree(). Remove unnecessary fhold() and fdrop() calls from aio_qphysio(), undo'ing a part of revision 1.86. The reference count on the file structure is already incremented by _aio_aqueue() before it calls aio_qphysio(). (Update the comments to document this fact.) Remove unnecessary casts from _aio_aqueue(), aio_read(), aio_write() and aio_waitcomplete(). Remove an unnecessary "return;" from aio_process(). Add "static" in various places.
2001-03-05 01:30:23 +00:00
static int
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
aio_free_entry(struct aiocblist *aiocbe)
{
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct kaioinfo *ki;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
struct aioliojob *lj;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct proc *p;
p = aiocbe->userproc;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
MPASS(curproc == p);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
ki = p->p_aioinfo;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
MPASS(ki != NULL);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK_ASSERT(ki, MA_OWNED);
MPASS(aiocbe->jobstate == JOBST_JOBFINISHED);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
atomic_subtract_int(&num_queue_count, 1);
ki->kaio_count--;
MPASS(ki->kaio_count >= 0);
1. Remove aio entry from lists earlier in aio_free_entry, so other threads can not see it if we unlock the proc lock (this can happen in knlist_delete). Don't do wakeup, it is not necessary. 2. Decrease kaio_buffer_count in biohelper rather than doing it in aio_bio_done_notify. 3. In aio_bio_done_notify, don't send notification if KAIO_RUNDOWN was set, because the process is already in single thread mode. 4. Use assignment to initialize aiothreadflags. 5. AIOCBLIST_RUNDOWN is not useful, axe the code using it. 6. use LIO_NOP instead of zero.
2006-02-26 12:56:23 +00:00
TAILQ_REMOVE(&ki->kaio_done, aiocbe, plist);
TAILQ_REMOVE(&ki->kaio_all, aiocbe, allist);
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
lj = aiocbe->lio;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (lj) {
lj->lioj_count--;
lj->lioj_finished_count--;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
1) Merge _aio_aqueue and aio_aqueue, check quota in aio_aqueue, so that lio_listio won't exceed the quota. 2) Remove lio_ref_count, it is no longer used.
2006-01-23 02:49:34 +00:00
if (lj->lioj_count == 0) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TAILQ_REMOVE(&ki->kaio_liojoblist, lj, lioj_list);
/* lio is going away, we need to destroy any knotes */
knlist_delete(&lj->klist, curthread, 1);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
PROC_LOCK(p);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
sigqueue_take(&lj->lioj_ksi);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
PROC_UNLOCK(p);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
uma_zfree(aiolio_zone, lj);
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
}
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
}
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
/* aiocbe is going away, we need to destroy any knotes */
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
knlist_delete(&aiocbe->klist, curthread, 1);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
PROC_LOCK(p);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
sigqueue_take(&aiocbe->ksi);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
PROC_UNLOCK(p);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
MPASS(aiocbe->bp == NULL);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
aiocbe->jobstate = JOBST_NULL;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
/*
* The thread argument here is used to find the owning process
* and is also passed to fo_close() which may pass it to various
* places such as devsw close() routines. Because of that, we
* need a thread pointer from the process owning the job that is
* persistent and won't disappear out from under us or move to
* another process.
*
* Currently, all the callers of this function call it to remove
* an aiocblist from the current process' job list either via a
* syscall or due to the current process calling exit() or
* execve(). Thus, we know that p == curproc. We also know that
* curthread can't exit since we are curthread.
*
* Therefore, we use curthread as the thread to pass to
* knlist_delete(). This does mean that it is possible for the
* thread pointer at close time to differ from the thread pointer
* at open time, but this is already true of file descriptors in
* a multithreaded process.
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
*/
Add new system call - aio_mlock(). The name speaks for itself. It allows to perform the mlock(2) operation, which can consume a lot of time, under control of aio(4). Reviewed by: kib, jilles Sponsored by: Nginx, Inc.
2013-06-08 13:27:57 +00:00
if (aiocbe->fd_file)
fdrop(aiocbe->fd_file, curthread);
Do a bit more work in the aio code to simulate the credential environment of the original AIO request: save and restore the active thread credential as well as using the file credential, since MAC (and some other bits of the system) rely on the thread credential instead of/as well as the file credential. In brief: cache td->td_ucred when the AIO operation is queued, temporarily set and restore the kernel thread credential, and release the credential when done. Similar to ktrace credential management. Reviewed by: alc Approved by: re Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
2002-11-07 20:46:37 +00:00
crfree(aiocbe->cred);
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
uma_zfree(aiocb_zone, aiocbe);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (0);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
- Change process_exec function handlers prototype to include struct image_params arg. - Change struct image_params to include struct sysentvec pointer and initialize it. - Change all consumers of process_exit/process_exec eventhandlers to new prototypes (includes splitting up into distinct exec/exit functions). - Add eventhandler to userret. Sponsored by: Google SoC 2006 Submitted by: rdivacky Parts suggested by: jhb (on hackers@)
2006-08-15 12:10:57 +00:00
static void
aio_proc_rundown_exec(void *arg, struct proc *p, struct image_params *imgp __unused)
{
aio_proc_rundown(arg, p);
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
/*
clean up whitespace...
2004-08-13 17:43:53 +00:00
* Rundown the jobs for a given process.
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
*/
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
static void
Replace the at_fork, at_exec, and at_exit functions with the slightly more flexible process_fork, process_exec, and process_exit eventhandlers. This reduces code duplication and also means that I don't have to go duplicate the eventhandler locking three more times for each of at_fork, at_exec, and at_exit. Reviewed by: phk, jake, almost complete silence on arch@
2003-03-24 21:15:35 +00:00
aio_proc_rundown(void *arg, struct proc *p)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct kaioinfo *ki;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
struct aioliojob *lj;
struct aiocblist *cbe, *cbn;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
struct file *fp;
struct socket *so;
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
int remove;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
KASSERT(curthread->td_proc == p,
("%s: called on non-curproc", __func__));
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
ki = p->p_aioinfo;
if (ki == NULL)
return;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
1. Remove aio entry from lists earlier in aio_free_entry, so other threads can not see it if we unlock the proc lock (this can happen in knlist_delete). Don't do wakeup, it is not necessary. 2. Decrease kaio_buffer_count in biohelper rather than doing it in aio_bio_done_notify. 3. In aio_bio_done_notify, don't send notification if KAIO_RUNDOWN was set, because the process is already in single thread mode. 4. Use assignment to initialize aiothreadflags. 5. AIOCBLIST_RUNDOWN is not useful, axe the code using it. 6. use LIO_NOP instead of zero.
2006-02-26 12:56:23 +00:00
ki->kaio_flags |= KAIO_RUNDOWN;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
restart:
Major cleanup and debugging of the new AIO/LIO code. The LIO code is now corrected. New tunables/instrumentation added. The code is now likely "good enough to use." I will add the userland support soon. The "high performance" mode for raw devices is still missing, and will be added next. POSIX system calls that now appear to work: aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, lio_listio. Missing, but to be added soon: aio_fsync.
1997-10-09 04:14:41 +00:00
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/*
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
* Try to cancel all pending requests. This code simulates
* aio_cancel on all pending I/O requests.
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
*/
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TAILQ_FOREACH_SAFE(cbe, &ki->kaio_jobqueue, plist, cbn) {
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
remove = 0;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_lock(&aio_job_mtx);
if (cbe->jobstate == JOBST_JOBQGLOBAL) {
TAILQ_REMOVE(&aio_jobs, cbe, list);
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
remove = 1;
} else if (cbe->jobstate == JOBST_JOBQSOCK) {
fp = cbe->fd_file;
MPASS(fp->f_type == DTYPE_SOCKET);
so = fp->f_data;
TAILQ_REMOVE(&so->so_aiojobq, cbe, list);
remove = 1;
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
} else if (cbe->jobstate == JOBST_JOBQSYNC) {
TAILQ_REMOVE(&ki->kaio_syncqueue, cbe, list);
remove = 1;
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
}
mtx_unlock(&aio_job_mtx);
if (remove) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
cbe->jobstate = JOBST_JOBFINISHED;
cbe->uaiocb._aiocb_private.status = -1;
cbe->uaiocb._aiocb_private.error = ECANCELED;
TAILQ_REMOVE(&ki->kaio_jobqueue, cbe, plist);
aio_bio_done_notify(p, cbe, DONE_QUEUE);
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
/* Wait for all running I/O to be finished */
if (TAILQ_FIRST(&ki->kaio_bufqueue) ||
TAILQ_FIRST(&ki->kaio_jobqueue)) {
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
ki->kaio_flags |= KAIO_WAKEUP;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
msleep(&p->p_aioinfo, AIO_MTX(ki), PRIBIO, "aioprn", hz);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
goto restart;
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
/* Free all completed I/O requests. */
while ((cbe = TAILQ_FIRST(&ki->kaio_done)) != NULL)
aio_free_entry(cbe);
while ((lj = TAILQ_FIRST(&ki->kaio_liojoblist)) != NULL) {
1) Merge _aio_aqueue and aio_aqueue, check quota in aio_aqueue, so that lio_listio won't exceed the quota. 2) Remove lio_ref_count, it is no longer used.
2006-01-23 02:49:34 +00:00
if (lj->lioj_count == 0) {
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
TAILQ_REMOVE(&ki->kaio_liojoblist, lj, lioj_list);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
knlist_delete(&lj->klist, curthread, 1);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
PROC_LOCK(p);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
sigqueue_take(&lj->lioj_ksi);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
PROC_UNLOCK(p);
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
uma_zfree(aiolio_zone, lj);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
} else {
1) Merge _aio_aqueue and aio_aqueue, check quota in aio_aqueue, so that lio_listio won't exceed the quota. 2) Remove lio_ref_count, it is no longer used.
2006-01-23 02:49:34 +00:00
panic("LIO job not cleaned up: C:%d, FC:%d\n",
lj->lioj_count, lj->lioj_finished_count);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
}
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
}
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
taskqueue_drain(taskqueue_aiod_bio, &ki->kaio_task);
Destroy the kaio_mtx on the freeing the struct kaioinfo in the aio_proc_rundown. Do not allow for zero-length read to be passed to the fo_read file method by aio. Reported and tested by: Peter Holm Approved by: re (kensmith)
2007-08-20 11:53:26 +00:00
mtx_destroy(&ki->kaio_mtx);
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
uma_zfree(kaio_zone, ki);
Major cleanup and debugging of the new AIO/LIO code. The LIO code is now corrected. New tunables/instrumentation added. The code is now likely "good enough to use." I will add the userland support soon. The "high performance" mode for raw devices is still missing, and will be added next. POSIX system calls that now appear to work: aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, lio_listio. Missing, but to be added soon: aio_fsync.
1997-10-09 04:14:41 +00:00
p->p_aioinfo = NULL;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Select a job to run (called by an AIO daemon).
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
*/
static struct aiocblist *
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
aio_selectjob(struct aiothreadlist *aiop)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct aiocblist *aiocbe;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
struct kaioinfo *ki;
struct proc *userp;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_assert(&aio_job_mtx, MA_OWNED);
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
TAILQ_FOREACH(aiocbe, &aio_jobs, list) {
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
userp = aiocbe->userproc;
ki = userp->p_aioinfo;
if (ki->kaio_active_count < ki->kaio_maxactive_count) {
TAILQ_REMOVE(&aio_jobs, aiocbe, list);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
/* Account for currently active jobs. */
ki->kaio_active_count++;
aiocbe->jobstate = JOBST_JOBRUNNING;
break;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
return (aiocbe);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
/*
* Move all data to a permanent storage device, this code
* simulates fsync syscall.
*/
static int
aio_fsync_vnode(struct thread *td, struct vnode *vp)
{
struct mount *mp;
int error;
if ((error = vn_start_write(vp, &mp, V_WAIT | PCATCH)) != 0)
goto drop;
vn_lock() is currently only used with the 'curthread' passed as argument. Remove this argument and pass curthread directly to underlying VOP_LOCK1() VFS method. This modify makes the code cleaner and in particular remove an annoying dependence helping next lockmgr() cleanup. KPI results, obviously, changed. Manpage and FreeBSD_version will be updated through further commits. As a side note, would be valuable to say that next commits will address a similar cleanup about VFS methods, in particular vop_lock1 and vop_unlock. Tested by: Diego Sardina <siarodx at gmail dot com>, Andrea Di Pasquale <whyx dot it at gmail dot com>
2008-01-10 01:10:58 +00:00
vn_lock(vp, LK_EXCLUSIVE | LK_RETRY);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
if (vp->v_object != NULL) {
Switch the vm_object mutex to be a rwlock. This will enable in the future further optimizations where the vm_object lock will be held in read mode most of the time the page cache resident pool of pages are accessed for reading purposes. The change is mostly mechanical but few notes are reported: * The KPI changes as follow: - VM_OBJECT_LOCK() -> VM_OBJECT_WLOCK() - VM_OBJECT_TRYLOCK() -> VM_OBJECT_TRYWLOCK() - VM_OBJECT_UNLOCK() -> VM_OBJECT_WUNLOCK() - VM_OBJECT_LOCK_ASSERT(MA_OWNED) -> VM_OBJECT_ASSERT_WLOCKED() (in order to avoid visibility of implementation details) - The read-mode operations are added: VM_OBJECT_RLOCK(), VM_OBJECT_TRYRLOCK(), VM_OBJECT_RUNLOCK(), VM_OBJECT_ASSERT_RLOCKED(), VM_OBJECT_ASSERT_LOCKED() * The vm/vm_pager.h namespace pollution avoidance (forcing requiring sys/mutex.h in consumers directly to cater its inlining functions using VM_OBJECT_LOCK()) imposes that all the vm/vm_pager.h consumers now must include also sys/rwlock.h. * zfs requires a quite convoluted fix to include FreeBSD rwlocks into the compat layer because the name clash between FreeBSD and solaris versions must be avoided. At this purpose zfs redefines the vm_object locking functions directly, isolating the FreeBSD components in specific compat stubs. The KPI results heavilly broken by this commit. Thirdy part ports must be updated accordingly (I can think off-hand of VirtualBox, for example). Sponsored by: EMC / Isilon storage division Reviewed by: jeff Reviewed by: pjd (ZFS specific review) Discussed with: alc Tested by: pho
2013-03-09 02:32:23 +00:00
VM_OBJECT_WLOCK(vp->v_object);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
vm_object_page_clean(vp->v_object, 0, 0, 0);
Switch the vm_object mutex to be a rwlock. This will enable in the future further optimizations where the vm_object lock will be held in read mode most of the time the page cache resident pool of pages are accessed for reading purposes. The change is mostly mechanical but few notes are reported: * The KPI changes as follow: - VM_OBJECT_LOCK() -> VM_OBJECT_WLOCK() - VM_OBJECT_TRYLOCK() -> VM_OBJECT_TRYWLOCK() - VM_OBJECT_UNLOCK() -> VM_OBJECT_WUNLOCK() - VM_OBJECT_LOCK_ASSERT(MA_OWNED) -> VM_OBJECT_ASSERT_WLOCKED() (in order to avoid visibility of implementation details) - The read-mode operations are added: VM_OBJECT_RLOCK(), VM_OBJECT_TRYRLOCK(), VM_OBJECT_RUNLOCK(), VM_OBJECT_ASSERT_RLOCKED(), VM_OBJECT_ASSERT_LOCKED() * The vm/vm_pager.h namespace pollution avoidance (forcing requiring sys/mutex.h in consumers directly to cater its inlining functions using VM_OBJECT_LOCK()) imposes that all the vm/vm_pager.h consumers now must include also sys/rwlock.h. * zfs requires a quite convoluted fix to include FreeBSD rwlocks into the compat layer because the name clash between FreeBSD and solaris versions must be avoided. At this purpose zfs redefines the vm_object locking functions directly, isolating the FreeBSD components in specific compat stubs. The KPI results heavilly broken by this commit. Thirdy part ports must be updated accordingly (I can think off-hand of VirtualBox, for example). Sponsored by: EMC / Isilon storage division Reviewed by: jeff Reviewed by: pjd (ZFS specific review) Discussed with: alc Tested by: pho
2013-03-09 02:32:23 +00:00
VM_OBJECT_WUNLOCK(vp->v_object);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
}
error = VOP_FSYNC(vp, MNT_WAIT, td);
VOP_LOCK1() (and so VOP_LOCK()) and VOP_UNLOCK() are only used in conjuction with 'thread' argument passing which is always curthread. Remove the unuseful extra-argument and pass explicitly curthread to lower layer functions, when necessary. KPI results broken by this change, which should affect several ports, so version bumping and manpage update will be further committed. Tested by: kris, pho, Diego Sardina <siarodx at gmail dot com>
2008-01-13 14:44:15 +00:00
VOP_UNLOCK(vp, 0);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
vn_finished_write(mp);
drop:
return (error);
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
/*
Separate LIO_SYNC processing into a separate function aio_process_sync(), and rename aio_process() into aio_process_rw(). Reviewed by: kib Sponsored by: Nginx, Inc.
2013-06-08 13:02:43 +00:00
* The AIO processing activity for LIO_READ/LIO_WRITE. This is the code that
* does the I/O request for the non-physio version of the operations. The
* normal vn operations are used, and this code should work in all instances
* for every type of file, including pipes, sockets, fifos, and regular files.
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
*
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
* XXX I don't think it works well for socket, pipe, and fifo.
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
*/
Eliminate the aio_freejobs list. Its purpose was to store free aiocb's allocated by zalloc(). In other words, zfree() was never called. Now, we call zfree(). Why eliminate this micro- optimization? At some later point, when we multithread the AIO system, we would need a mutex to synchronize access to aio_freejobs, making its use nearly indistinguishable in cost from zalloc() and zfree(). Remove unnecessary fhold() and fdrop() calls from aio_qphysio(), undo'ing a part of revision 1.86. The reference count on the file structure is already incremented by _aio_aqueue() before it calls aio_qphysio(). (Update the comments to document this fact.) Remove unnecessary casts from _aio_aqueue(), aio_read(), aio_write() and aio_waitcomplete(). Remove an unnecessary "return;" from aio_process(). Add "static" in various places.
2001-03-05 01:30:23 +00:00
static void
Separate LIO_SYNC processing into a separate function aio_process_sync(), and rename aio_process() into aio_process_rw(). Reviewed by: kib Sponsored by: Nginx, Inc.
2013-06-08 13:02:43 +00:00
aio_process_rw(struct aiocblist *aiocbe)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
Do a bit more work in the aio code to simulate the credential environment of the original AIO request: save and restore the active thread credential as well as using the file credential, since MAC (and some other bits of the system) rely on the thread credential instead of/as well as the file credential. In brief: cache td->td_ucred when the AIO operation is queued, temporarily set and restore the kernel thread credential, and release the credential when done. Similar to ktrace credential management. Reviewed by: alc Approved by: re Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
2002-11-07 20:46:37 +00:00
struct ucred *td_savedcred;
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct thread *td;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct aiocb *cb;
struct file *fp;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
struct socket *so;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct uio auio;
struct iovec aiov;
int cnt;
int error;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
int oublock_st, oublock_end;
int inblock_st, inblock_end;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Separate LIO_SYNC processing into a separate function aio_process_sync(), and rename aio_process() into aio_process_rw(). Reviewed by: kib Sponsored by: Nginx, Inc.
2013-06-08 13:02:43 +00:00
KASSERT(aiocbe->uaiocb.aio_lio_opcode == LIO_READ ||
aiocbe->uaiocb.aio_lio_opcode == LIO_WRITE,
("%s: opcode %d", __func__, aiocbe->uaiocb.aio_lio_opcode));
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
td = curthread;
Do a bit more work in the aio code to simulate the credential environment of the original AIO request: save and restore the active thread credential as well as using the file credential, since MAC (and some other bits of the system) rely on the thread credential instead of/as well as the file credential. In brief: cache td->td_ucred when the AIO operation is queued, temporarily set and restore the kernel thread credential, and release the credential when done. Similar to ktrace credential management. Reviewed by: alc Approved by: re Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
2002-11-07 20:46:37 +00:00
td_savedcred = td->td_ucred;
td->td_ucred = aiocbe->cred;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
cb = &aiocbe->uaiocb;
o Use aiocblist::fd_file in the AIO threads rather than recomputing the file * from the calling process's descriptor table. o Eliminate sharing of the calling process's descriptor table with the AIO threads.
2002-04-14 03:04:19 +00:00
fp = aiocbe->fd_file;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
o Eliminate unnecessary synchronization from filt_aiodetach(). o The manual page for kevent says that EVFILT_AIO returns under the same conditions as aio_error(). With that in mind, set the data field of the returned struct kevent to the value that would be returned by aio_error(). o Fix two compilation warnings.
2001-12-09 08:16:36 +00:00
aiov.iov_base = (void *)(uintptr_t)cb->aio_buf;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
aiov.iov_len = cb->aio_nbytes;
auio.uio_iov = &aiov;
auio.uio_iovcnt = 1;
o aio_process needn't fhold()/fdrop() the fp now that _aio_aqueue() and aio_free_entry() do this. o Remove two unnecessary/unused variables from aio_process() and one field from aiocblist.
2002-04-04 02:13:20 +00:00
auio.uio_offset = cb->aio_offset;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
auio.uio_resid = cb->aio_nbytes;
cnt = cb->aio_nbytes;
auio.uio_segflg = UIO_USERSPACE;
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
auio.uio_td = td;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
- Move rusage from being per-process in struct pstats to per-thread in td_ru. This removes the requirement for per-process synchronization in statclock() and mi_switch(). This was previously supported by sched_lock which is going away. All modifications to rusage are now done in the context of the owning thread. reads proceed without locks. - Aggregate exiting threads rusage in thread_exit() such that the exiting thread's rusage is not lost. - Provide a new routine, rufetch() to fetch an aggregate of all rusage structures from all threads in a process. This routine must be used in any place requiring a rusage from a process prior to it's exit. The exited process's rusage is still available via p_ru. - Aggregate tick statistics only on demand via rufetch() or when a thread exits. Tick statistics are kept in the thread and protected by sched_lock until it exits. Initial patch by: attilio Reviewed by: attilio, bde (some objections), arch (mostly silent)
2007-06-01 01:12:45 +00:00
inblock_st = td->td_ru.ru_inblock;
oublock_st = td->td_ru.ru_oublock;
This patchset fixes a large number of file descriptor race conditions. Pre-rfork code assumed inherent locking of a process's file descriptor array. However, with the advent of rfork() the file descriptor table could be shared between processes. This patch closes over a dozen serious race conditions related to one thread manipulating the table (e.g. closing or dup()ing a descriptor) while another is blocked in an open(), close(), fcntl(), read(), write(), etc... PR: kern/11629 Discussed with: Alexander Viro <viro@math.psu.edu>
2000-11-18 21:01:04 +00:00
/*
1) Merge _aio_aqueue and aio_aqueue, check quota in aio_aqueue, so that lio_listio won't exceed the quota. 2) Remove lio_ref_count, it is no longer used.
2006-01-23 02:49:34 +00:00
* aio_aqueue() acquires a reference to the file that is
o aio_process needn't fhold()/fdrop() the fp now that _aio_aqueue() and aio_free_entry() do this. o Remove two unnecessary/unused variables from aio_process() and one field from aiocblist.
2002-04-04 02:13:20 +00:00
* released in aio_free_entry().
This patchset fixes a large number of file descriptor race conditions. Pre-rfork code assumed inherent locking of a process's file descriptor array. However, with the advent of rfork() the file descriptor table could be shared between processes. This patch closes over a dozen serious race conditions related to one thread manipulating the table (e.g. closing or dup()ing a descriptor) while another is blocked in an open(), close(), fcntl(), read(), write(), etc... PR: kern/11629 Discussed with: Alexander Viro <viro@math.psu.edu>
2000-11-18 21:01:04 +00:00
*/
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
if (cb->aio_lio_opcode == LIO_READ) {
auio.uio_rw = UIO_READ;
Destroy the kaio_mtx on the freeing the struct kaioinfo in the aio_proc_rundown. Do not allow for zero-length read to be passed to the fo_read file method by aio. Reported and tested by: Peter Holm Approved by: re (kensmith)
2007-08-20 11:53:26 +00:00
if (auio.uio_resid == 0)
error = 0;
else
error = fo_read(fp, &auio, fp->f_cred, FOF_OFFSET, td);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
} else {
Just like dofilewrite(), call bwillwrite before fo_write.
2006-01-27 08:02:25 +00:00
if (fp->f_type == DTYPE_VNODE)
bwillwrite();
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
auio.uio_rw = UIO_WRITE;
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
error = fo_write(fp, &auio, fp->f_cred, FOF_OFFSET, td);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
- Move rusage from being per-process in struct pstats to per-thread in td_ru. This removes the requirement for per-process synchronization in statclock() and mi_switch(). This was previously supported by sched_lock which is going away. All modifications to rusage are now done in the context of the owning thread. reads proceed without locks. - Aggregate exiting threads rusage in thread_exit() such that the exiting thread's rusage is not lost. - Provide a new routine, rufetch() to fetch an aggregate of all rusage structures from all threads in a process. This routine must be used in any place requiring a rusage from a process prior to it's exit. The exited process's rusage is still available via p_ru. - Aggregate tick statistics only on demand via rufetch() or when a thread exits. Tick statistics are kept in the thread and protected by sched_lock until it exits. Initial patch by: attilio Reviewed by: attilio, bde (some objections), arch (mostly silent)
2007-06-01 01:12:45 +00:00
inblock_end = td->td_ru.ru_inblock;
oublock_end = td->td_ru.ru_oublock;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
aiocbe->inputcharge = inblock_end - inblock_st;
aiocbe->outputcharge = oublock_end - oublock_st;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
if ((error) && (auio.uio_resid != cnt)) {
if (error == ERESTART || error == EINTR || error == EWOULDBLOCK)
error = 0;
Grab the process lock while calling psignal and before calling psignal.
2001-03-07 03:37:06 +00:00
if ((error == EPIPE) && (cb->aio_lio_opcode == LIO_WRITE)) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
int sigpipe = 1;
if (fp->f_type == DTYPE_SOCKET) {
so = fp->f_data;
if (so->so_options & SO_NOSIGPIPE)
sigpipe = 0;
}
if (sigpipe) {
PROC_LOCK(aiocbe->userproc);
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
kern_psignal(aiocbe->userproc, SIGPIPE);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
PROC_UNLOCK(aiocbe->userproc);
}
Grab the process lock while calling psignal and before calling psignal.
2001-03-07 03:37:06 +00:00
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
cnt -= auio.uio_resid;
cb->_aiocb_private.error = error;
cb->_aiocb_private.status = cnt;
Do a bit more work in the aio code to simulate the credential environment of the original AIO request: save and restore the active thread credential as well as using the file credential, since MAC (and some other bits of the system) rely on the thread credential instead of/as well as the file credential. In brief: cache td->td_ucred when the AIO operation is queued, temporarily set and restore the kernel thread credential, and release the credential when done. Similar to ktrace credential management. Reviewed by: alc Approved by: re Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
2002-11-07 20:46:37 +00:00
td->td_ucred = td_savedcred;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Separate LIO_SYNC processing into a separate function aio_process_sync(), and rename aio_process() into aio_process_rw(). Reviewed by: kib Sponsored by: Nginx, Inc.
2013-06-08 13:02:43 +00:00
static void
aio_process_sync(struct aiocblist *aiocbe)
{
struct thread *td = curthread;
struct ucred *td_savedcred = td->td_ucred;
struct aiocb *cb = &aiocbe->uaiocb;
struct file *fp = aiocbe->fd_file;
int error = 0;
KASSERT(aiocbe->uaiocb.aio_lio_opcode == LIO_SYNC,
("%s: opcode %d", __func__, aiocbe->uaiocb.aio_lio_opcode));
td->td_ucred = aiocbe->cred;
if (fp->f_vnode != NULL)
error = aio_fsync_vnode(td, fp->f_vnode);
cb->_aiocb_private.error = error;
cb->_aiocb_private.status = 0;
td->td_ucred = td_savedcred;
}
Add new system call - aio_mlock(). The name speaks for itself. It allows to perform the mlock(2) operation, which can consume a lot of time, under control of aio(4). Reviewed by: kib, jilles Sponsored by: Nginx, Inc.
2013-06-08 13:27:57 +00:00
static void
aio_process_mlock(struct aiocblist *aiocbe)
{
struct aiocb *cb = &aiocbe->uaiocb;
int error;
KASSERT(aiocbe->uaiocb.aio_lio_opcode == LIO_MLOCK,
("%s: opcode %d", __func__, aiocbe->uaiocb.aio_lio_opcode));
error = vm_mlock(aiocbe->userproc, aiocbe->cred,
__DEVOLATILE(void *, cb->aio_buf), cb->aio_nbytes);
cb->_aiocb_private.error = error;
cb->_aiocb_private.status = 0;
}
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
static void
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
aio_bio_done_notify(struct proc *userp, struct aiocblist *aiocbe, int type)
{
struct aioliojob *lj;
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
struct kaioinfo *ki;
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
struct aiocblist *scb, *scbn;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
int lj_done;
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
ki = userp->p_aioinfo;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK_ASSERT(ki, MA_OWNED);
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
lj = aiocbe->lio;
lj_done = 0;
if (lj) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
lj->lioj_finished_count++;
if (lj->lioj_count == lj->lioj_finished_count)
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
lj_done = 1;
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (type == DONE_QUEUE) {
aiocbe->jobflags |= AIOCBLIST_DONE;
} else {
aiocbe->jobflags |= AIOCBLIST_BUFDONE;
}
TAILQ_INSERT_TAIL(&ki->kaio_done, aiocbe, plist);
aiocbe->jobstate = JOBST_JOBFINISHED;
1. Remove aio entry from lists earlier in aio_free_entry, so other threads can not see it if we unlock the proc lock (this can happen in knlist_delete). Don't do wakeup, it is not necessary. 2. Decrease kaio_buffer_count in biohelper rather than doing it in aio_bio_done_notify. 3. In aio_bio_done_notify, don't send notification if KAIO_RUNDOWN was set, because the process is already in single thread mode. 4. Use assignment to initialize aiothreadflags. 5. AIOCBLIST_RUNDOWN is not useful, axe the code using it. 6. use LIO_NOP instead of zero.
2006-02-26 12:56:23 +00:00
if (ki->kaio_flags & KAIO_RUNDOWN)
goto notification_done;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (aiocbe->uaiocb.aio_sigevent.sigev_notify == SIGEV_SIGNAL ||
aiocbe->uaiocb.aio_sigevent.sigev_notify == SIGEV_THREAD_ID)
aio_sendsig(userp, &aiocbe->uaiocb.aio_sigevent, &aiocbe->ksi);
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
KNOTE_LOCKED(&aiocbe->klist, 1);
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (lj_done) {
if (lj->lioj_signal.sigev_notify == SIGEV_KEVENT) {
lj->lioj_flags |= LIOJ_KEVENT_POSTED;
KNOTE_LOCKED(&lj->klist, 1);
}
if ((lj->lioj_flags & (LIOJ_SIGNAL|LIOJ_SIGNAL_POSTED))
== LIOJ_SIGNAL
&& (lj->lioj_signal.sigev_notify == SIGEV_SIGNAL ||
lj->lioj_signal.sigev_notify == SIGEV_THREAD_ID)) {
aio_sendsig(userp, &lj->lioj_signal, &lj->lioj_ksi);
lj->lioj_flags |= LIOJ_SIGNAL_POSTED;
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
}
}
1. Remove aio entry from lists earlier in aio_free_entry, so other threads can not see it if we unlock the proc lock (this can happen in knlist_delete). Don't do wakeup, it is not necessary. 2. Decrease kaio_buffer_count in biohelper rather than doing it in aio_bio_done_notify. 3. In aio_bio_done_notify, don't send notification if KAIO_RUNDOWN was set, because the process is already in single thread mode. 4. Use assignment to initialize aiothreadflags. 5. AIOCBLIST_RUNDOWN is not useful, axe the code using it. 6. use LIO_NOP instead of zero.
2006-02-26 12:56:23 +00:00
notification_done:
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
if (aiocbe->jobflags & AIOCBLIST_CHECKSYNC) {
TAILQ_FOREACH_SAFE(scb, &ki->kaio_syncqueue, list, scbn) {
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
if (aiocbe->fd_file == scb->fd_file &&
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
aiocbe->seqno < scb->seqno) {
if (--scb->pending == 0) {
mtx_lock(&aio_job_mtx);
scb->jobstate = JOBST_JOBQGLOBAL;
TAILQ_REMOVE(&ki->kaio_syncqueue, scb, list);
TAILQ_INSERT_TAIL(&aio_jobs, scb, list);
aio_kick_nowait(userp);
mtx_unlock(&aio_job_mtx);
}
}
}
}
1. Remove aio entry from lists earlier in aio_free_entry, so other threads can not see it if we unlock the proc lock (this can happen in knlist_delete). Don't do wakeup, it is not necessary. 2. Decrease kaio_buffer_count in biohelper rather than doing it in aio_bio_done_notify. 3. In aio_bio_done_notify, don't send notification if KAIO_RUNDOWN was set, because the process is already in single thread mode. 4. Use assignment to initialize aiothreadflags. 5. AIOCBLIST_RUNDOWN is not useful, axe the code using it. 6. use LIO_NOP instead of zero.
2006-02-26 12:56:23 +00:00
if (ki->kaio_flags & KAIO_WAKEUP) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
ki->kaio_flags &= ~KAIO_WAKEUP;
wakeup(&userp->p_aioinfo);
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
}
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
/*
Separate LIO_SYNC processing into a separate function aio_process_sync(), and rename aio_process() into aio_process_rw(). Reviewed by: kib Sponsored by: Nginx, Inc.
2013-06-08 13:02:43 +00:00
* The AIO daemon, most of the actual work is done in aio_process_*,
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
* but the setup (and address space mgmt) is done in this routine.
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
*/
static void
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
aio_daemon(void *_id)
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
{
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
struct aiocblist *aiocbe;
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct aiothreadlist *aiop;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
struct kaioinfo *ki;
struct proc *curcp, *mycp, *userp;
struct vmspace *myvm, *tmpvm;
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct thread *td = curthread;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
int id = (intptr_t)_id;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
/*
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
* Local copies of curproc (cp) and vmspace (myvm)
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
*/
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
mycp = td->td_proc;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
myvm = mycp->p_vmspace;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
- kthread's don't have p_textvp set to anything, so replace code that dealt with that possibility with a KASSERT(). - No need to set P_SYSTEM, kthread_create() does that for us.
2003-04-17 22:37:48 +00:00
KASSERT(mycp->p_textvp == NULL, ("kthread has a textvp"));
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Allocate and ready the aio control info. There is one aiop structure
* per daemon.
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
*/
Back out M_* changes, per decision of the TRB. Approved by: trb
2003-02-19 05:47:46 +00:00
aiop = uma_zalloc(aiop_zone, M_WAITOK);
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
aiop->aiothread = td;
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
aiop->aiothreadflags = 0;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
Er, rescure a deleted comment line.
2006-01-24 02:50:42 +00:00
/* The daemon resides in its own pgrp. */
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_setsid(td, NULL);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
/*
* Wakeup parent process. (Parent sleeps to keep from blasting away
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
* and creating too many daemons.)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
*/
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
sema_post(&aio_newproc_sem);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_lock(&aio_job_mtx);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
for (;;) {
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
/*
* curcp is the current daemon process context.
* userp is the current user process context.
*/
curcp = mycp;
Make the target for the number of AIO daemons work.
1997-10-11 01:07:03 +00:00
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
/*
* Take daemon off of free queue
*/
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
if (aiop->aiothreadflags & AIOP_FREE) {
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
TAILQ_REMOVE(&aio_freeproc, aiop, list);
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
aiop->aiothreadflags &= ~AIOP_FREE;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Check for jobs.
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
*/
Fix warnings in preparation for adding -Wall -Wcast-qual to the kernel compile
1999-01-27 21:50:00 +00:00
while ((aiocbe = aio_selectjob(aiop)) != NULL) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_unlock(&aio_job_mtx);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
userp = aiocbe->userproc;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Connect to process address space for user program.
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
*/
if (userp != curcp) {
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Save the current address space that we are
* connected to.
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
*/
tmpvm = mycp->p_vmspace;
clean up whitespace...
2004-08-13 17:43:53 +00:00
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Point to the new user address space, and
* refer to it.
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
*/
mycp->p_vmspace = userp->p_vmspace;
- Use atomic ops for updating the vmspace's refcnt and exitingcnt. - Push down Giant into shmexit(). (Giant is acquired only if the vmspace contains shm segments.) - Eliminate the acquisition of Giant from proc_rwmem(). - Reduce the scope of Giant in exit1(), uncovering the destruction of the address space.
2004-07-27 03:53:41 +00:00
atomic_add_int(&mycp->p_vmspace->vm_refcnt, 1);
clean up whitespace...
2004-08-13 17:43:53 +00:00
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/* Activate the new mapping. */
Pre-KSE/M3 commit. this is a low-functionality change that changes the kernel to access the main thread of a process via the linked list of threads rather than assuming that it is embedded in the process. It IS still embeded there but remove all teh code that assumes that in preparation for the next commit which will actually move it out. Reviewed by: peter@freebsd.org, gallatin@cs.duke.edu, benno rice,
2002-02-07 20:58:47 +00:00
pmap_activate(FIRST_THREAD_IN_PROC(mycp));
clean up whitespace...
2004-08-13 17:43:53 +00:00
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* If the old address space wasn't the daemons
* own address space, then we need to remove the
* daemon's reference from the other process
* that it was acting on behalf of.
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
*/
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
if (tmpvm != myvm) {
vmspace_free(tmpvm);
}
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
curcp = userp;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
ki = userp->p_aioinfo;
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/* Do the I/O function. */
Separate LIO_SYNC processing into a separate function aio_process_sync(), and rename aio_process() into aio_process_rw(). Reviewed by: kib Sponsored by: Nginx, Inc.
2013-06-08 13:02:43 +00:00
switch(aiocbe->uaiocb.aio_lio_opcode) {
case LIO_READ:
case LIO_WRITE:
aio_process_rw(aiocbe);
break;
case LIO_SYNC:
aio_process_sync(aiocbe);
break;
Add new system call - aio_mlock(). The name speaks for itself. It allows to perform the mlock(2) operation, which can consume a lot of time, under control of aio(4). Reviewed by: kib, jilles Sponsored by: Nginx, Inc.
2013-06-08 13:27:57 +00:00
case LIO_MLOCK:
aio_process_mlock(aiocbe);
break;
Separate LIO_SYNC processing into a separate function aio_process_sync(), and rename aio_process() into aio_process_rw(). Reviewed by: kib Sponsored by: Nginx, Inc.
2013-06-08 13:02:43 +00:00
}
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
Decrease kaio_active_count first, because user process may go away after we notified it.
2006-01-22 09:25:52 +00:00
mtx_lock(&aio_job_mtx);
/* Decrement the active job count. */
ki->kaio_active_count--;
mtx_unlock(&aio_job_mtx);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TAILQ_REMOVE(&ki->kaio_jobqueue, aiocbe, plist);
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
aio_bio_done_notify(userp, aiocbe, DONE_QUEUE);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_lock(&aio_job_mtx);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Disconnect from user address space.
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
*/
if (curcp != mycp) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_unlock(&aio_job_mtx);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/* Get the user address space to disconnect from. */
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
tmpvm = mycp->p_vmspace;
clean up whitespace...
2004-08-13 17:43:53 +00:00
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/* Get original address space for daemon. */
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
mycp->p_vmspace = myvm;
clean up whitespace...
2004-08-13 17:43:53 +00:00
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/* Activate the daemon's address space. */
Pre-KSE/M3 commit. this is a low-functionality change that changes the kernel to access the main thread of a process via the linked list of threads rather than assuming that it is embedded in the process. It IS still embeded there but remove all teh code that assumes that in preparation for the next commit which will actually move it out. Reviewed by: peter@freebsd.org, gallatin@cs.duke.edu, benno rice,
2002-02-07 20:58:47 +00:00
pmap_activate(FIRST_THREAD_IN_PROC(mycp));
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
#ifdef DIAGNOSTIC
if (tmpvm == myvm) {
printf("AIOD: vmspace problem -- %d\n",
mycp->p_pid);
}
Fix error handling for VCHR type I/O. Also, fix another spl problem, and remove alot of overly verbose debugging statements. ioproclist { int aioprocflags; /* AIO proc flags */ TAILQ_ENTRY(aioproclist) list; /* List of processes */ struct proc *aioproc; /* The AIO thread */ TAILQ_HEAD (,aiocblist) jobtorun; /* suggested job to run */ }; /* * data-structure for lio signal management */ struct aio_liojob { int lioj_flags; int lioj_buffer_count; int lioj_buffer_finished_count; int lioj_queue_count; int lioj_queue_finished_count; struct sigevent lioj_signal; /* signal on all I/O done */ TAILQ_ENTRY (aio_liojob) lioj_list; struct kaioinfo *lioj_ki; }; #define LIOJ_SIGNAL 0x1 /* signal on all done (lio) */ #define LIOJ_SIGNAL_POSTED 0x2 /* signal has been posted */ /* * per process aio data structure */ struct kaioinfo { int kaio_flags; /* per process kaio flags */ int kaio_maxactive_count; /* maximum number of AIOs */ int kaio_active_count; /* number of currently used AIOs */ int kaio_qallowed_count; /* maxiumu size of AIO queue */ int kaio_queue_count; /* size of AIO queue */ int kaio_ballowed_count; /* maximum number of buffers */ int kaio_queue_finished_count; /* number of daemon jobs finished */ int kaio_buffer_count; /* number of physio buffers */ int kaio_buffer_finished_count; /* count of I/O done */ struct proc *kaio_p; /* process that uses this kaio block */ TAILQ_HEAD (,aio_liojob) kaio_liojoblist; /* list of lio jobs */ TAILQ_HEAD (,aiocblist) kaio_jobqueue; /* job queue for process */ TAILQ_HEAD (,aiocblist) kaio_jobdone; /* done queue for process */ TAILQ_HEAD (,aiocblist) kaio_bufqueue; /* buffer job queue for process */ TAILQ_HEAD (,aiocblist) kaio_bufdone; /* buffer done queue for process */ }; #define KAIO_RUNDOWN 0x1 /* process is being run down */ #define KAIO_WAKEUP 0x2 /* wakeup process when there is a significant event */ TAILQ_HEAD (,aioproclist) aio_freeproc, aio_activeproc; TAILQ_HEAD(,aiocblist) aio_jobs; /* Async job list */ TAILQ_HEAD(,aiocblist) aio_bufjobs; /* Phys I/O job list */ TAILQ_HEAD(,aiocblist) aio_freejobs; /* Pool of free jobs */ static void aio_init_aioinfo(struct proc *p) ; static void aio_onceonly(void *) ; static int aio_free_entry(struct aiocblist *aiocbe); static void aio_process(struct aiocblist *aiocbe); static int aio_newproc(void) ; static int aio_aqueue(struct proc *p, struct aiocb *job, int type) ; static void aio_physwakeup(struct buf *bp); static int aio_fphysio(struct proc *p, struct aiocblist *aiocbe, int type); static int aio_qphysio(struct proc *p, struct aiocblist *iocb); static void aio_daemon(void *uproc); SYSINIT(aio, SI_SUB_VFS, SI_ORDER_ANY, aio_onceonly, NULL); static vm_zone_t kaio_zone=0, aiop_zone=0, aiocb_zone=0, aiol_zone=0, aiolio_zone=0; /* * Single AIOD vmspace shared amongst all of them */ static struct vmspace *aiovmspace = NULL; /* * Startup initialization */ void aio_onceonly(void *na) { TAILQ_INIT(&aio_freeproc); TAILQ_INIT(&aio_activeproc); TAILQ_INIT(&aio_jobs); TAILQ_INIT(&aio_bufjobs); TAILQ_INIT(&aio_freejobs); kaio_zone = zinit("AIO", sizeof (struct kaioinfo), 0, 0, 1); aiop_zone = zinit("AIOP", sizeof (struct aioproclist), 0, 0, 1); aiocb_zone = zinit("AIOCB", sizeof (struct aiocblist), 0, 0, 1); aiol_zone = zinit("AIOL", AIO_LISTIO_MAX * sizeof (int), 0, 0, 1); aiolio_zone = zinit("AIOLIO", AIO_LISTIO_MAX * sizeof (struct aio_liojob), 0, 0, 1); aiod_timeout = AIOD_TIMEOUT_DEFAULT; aiod_lifetime = AIOD_LIFETIME_DEFAULT; jobrefid = 1; } /* * Init the per-process aioinfo structure. * The aioinfo limits are set per-process for user limit (resource) management. */ void aio_init_aioinfo(struct proc *p) { struct kaioinfo *ki; if (p->p_aioinfo == NULL) { ki = zalloc(kaio_zone); p->p_aioinfo = ki
1997-12-01 07:01:45 +00:00
#endif
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/* Remove our vmspace reference. */
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
vmspace_free(tmpvm);
clean up whitespace...
2004-08-13 17:43:53 +00:00
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
curcp = mycp;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_lock(&aio_job_mtx);
/*
* We have to restart to avoid race, we only sleep if
* no job can be selected, that should be
* curcp == mycp.
*/
continue;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_assert(&aio_job_mtx, MA_OWNED);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
TAILQ_INSERT_HEAD(&aio_freeproc, aiop, list);
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
aiop->aiothreadflags |= AIOP_FREE;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* If daemon is inactive for a long time, allow it to exit,
* thereby freeing resources.
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
*/
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (msleep(aiop->aiothread, &aio_job_mtx, PRIBIO, "aiordy",
aiod_lifetime)) {
o Remove the unused vestiges of JOBST_JOBQPROC and the per-thread jobtorun queue. o Use TAILQ_EMPTY() instead of TAILQ_FIRST(...) == NULL.
2002-01-20 18:59:58 +00:00
if (TAILQ_EMPTY(&aio_jobs)) {
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
if ((aiop->aiothreadflags & AIOP_FREE) &&
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
(num_aio_procs > target_aio_procs)) {
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
TAILQ_REMOVE(&aio_freeproc, aiop, list);
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
num_aio_procs--;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_unlock(&aio_job_mtx);
uma_zfree(aiop_zone, aiop);
free_unr(aiod_unr, id);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
#ifdef DIAGNOSTIC
if (mycp->p_vmspace->vm_refcnt <= 1) {
printf("AIOD: bad vm refcnt for"
" exiting daemon: %d\n",
mycp->p_vmspace->vm_refcnt);
}
Fix error handling for VCHR type I/O. Also, fix another spl problem, and remove alot of overly verbose debugging statements. ioproclist { int aioprocflags; /* AIO proc flags */ TAILQ_ENTRY(aioproclist) list; /* List of processes */ struct proc *aioproc; /* The AIO thread */ TAILQ_HEAD (,aiocblist) jobtorun; /* suggested job to run */ }; /* * data-structure for lio signal management */ struct aio_liojob { int lioj_flags; int lioj_buffer_count; int lioj_buffer_finished_count; int lioj_queue_count; int lioj_queue_finished_count; struct sigevent lioj_signal; /* signal on all I/O done */ TAILQ_ENTRY (aio_liojob) lioj_list; struct kaioinfo *lioj_ki; }; #define LIOJ_SIGNAL 0x1 /* signal on all done (lio) */ #define LIOJ_SIGNAL_POSTED 0x2 /* signal has been posted */ /* * per process aio data structure */ struct kaioinfo { int kaio_flags; /* per process kaio flags */ int kaio_maxactive_count; /* maximum number of AIOs */ int kaio_active_count; /* number of currently used AIOs */ int kaio_qallowed_count; /* maxiumu size of AIO queue */ int kaio_queue_count; /* size of AIO queue */ int kaio_ballowed_count; /* maximum number of buffers */ int kaio_queue_finished_count; /* number of daemon jobs finished */ int kaio_buffer_count; /* number of physio buffers */ int kaio_buffer_finished_count; /* count of I/O done */ struct proc *kaio_p; /* process that uses this kaio block */ TAILQ_HEAD (,aio_liojob) kaio_liojoblist; /* list of lio jobs */ TAILQ_HEAD (,aiocblist) kaio_jobqueue; /* job queue for process */ TAILQ_HEAD (,aiocblist) kaio_jobdone; /* done queue for process */ TAILQ_HEAD (,aiocblist) kaio_bufqueue; /* buffer job queue for process */ TAILQ_HEAD (,aiocblist) kaio_bufdone; /* buffer done queue for process */ }; #define KAIO_RUNDOWN 0x1 /* process is being run down */ #define KAIO_WAKEUP 0x2 /* wakeup process when there is a significant event */ TAILQ_HEAD (,aioproclist) aio_freeproc, aio_activeproc; TAILQ_HEAD(,aiocblist) aio_jobs; /* Async job list */ TAILQ_HEAD(,aiocblist) aio_bufjobs; /* Phys I/O job list */ TAILQ_HEAD(,aiocblist) aio_freejobs; /* Pool of free jobs */ static void aio_init_aioinfo(struct proc *p) ; static void aio_onceonly(void *) ; static int aio_free_entry(struct aiocblist *aiocbe); static void aio_process(struct aiocblist *aiocbe); static int aio_newproc(void) ; static int aio_aqueue(struct proc *p, struct aiocb *job, int type) ; static void aio_physwakeup(struct buf *bp); static int aio_fphysio(struct proc *p, struct aiocblist *aiocbe, int type); static int aio_qphysio(struct proc *p, struct aiocblist *iocb); static void aio_daemon(void *uproc); SYSINIT(aio, SI_SUB_VFS, SI_ORDER_ANY, aio_onceonly, NULL); static vm_zone_t kaio_zone=0, aiop_zone=0, aiocb_zone=0, aiol_zone=0, aiolio_zone=0; /* * Single AIOD vmspace shared amongst all of them */ static struct vmspace *aiovmspace = NULL; /* * Startup initialization */ void aio_onceonly(void *na) { TAILQ_INIT(&aio_freeproc); TAILQ_INIT(&aio_activeproc); TAILQ_INIT(&aio_jobs); TAILQ_INIT(&aio_bufjobs); TAILQ_INIT(&aio_freejobs); kaio_zone = zinit("AIO", sizeof (struct kaioinfo), 0, 0, 1); aiop_zone = zinit("AIOP", sizeof (struct aioproclist), 0, 0, 1); aiocb_zone = zinit("AIOCB", sizeof (struct aiocblist), 0, 0, 1); aiol_zone = zinit("AIOL", AIO_LISTIO_MAX * sizeof (int), 0, 0, 1); aiolio_zone = zinit("AIOLIO", AIO_LISTIO_MAX * sizeof (struct aio_liojob), 0, 0, 1); aiod_timeout = AIOD_TIMEOUT_DEFAULT; aiod_lifetime = AIOD_LIFETIME_DEFAULT; jobrefid = 1; } /* * Init the per-process aioinfo structure. * The aioinfo limits are set per-process for user limit (resource) management. */ void aio_init_aioinfo(struct proc *p) { struct kaioinfo *ki; if (p->p_aioinfo == NULL) { ki = zalloc(kaio_zone); p->p_aioinfo = ki
1997-12-01 07:01:45 +00:00
#endif
Rename the kthread_xxx (e.g. kthread_create()) calls to kproc_xxx as they actually make whole processes. Thos makes way for us to add REAL kthread_create() and friends that actually make theads. it turns out that most of these calls actually end up being moved back to the thread version when it's added. but we need to make this cosmetic change first. I'd LOVE to do this rename in 7.0 so that we can eventually MFC the new kthread_xxx() calls.
2007-10-20 23:23:23 +00:00
kproc_exit(0);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
}
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_unlock(&aio_job_mtx);
panic("shouldn't be here\n");
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
/*
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
* Create a new AIO daemon. This is mostly a kernel-thread fork routine. The
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* AIO daemon modifies its environment itself.
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
*/
static int
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
aio_newproc(int *start)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
int error;
Use the kthread API to create and destroy AIO daemons. Submitted by: jhb
2001-03-09 06:27:01 +00:00
struct proc *p;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
int id;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
id = alloc_unr(aiod_unr);
Rename the kthread_xxx (e.g. kthread_create()) calls to kproc_xxx as they actually make whole processes. Thos makes way for us to add REAL kthread_create() and friends that actually make theads. it turns out that most of these calls actually end up being moved back to the thread version when it's added. but we need to make this cosmetic change first. I'd LOVE to do this rename in 7.0 so that we can eventually MFC the new kthread_xxx() calls.
2007-10-20 23:23:23 +00:00
error = kproc_create(aio_daemon, (void *)(intptr_t)id, &p,
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
RFNOWAIT, 0, "aiod%d", id);
if (error == 0) {
/*
* Wait until daemon is started.
*/
sema_wait(&aio_newproc_sem);
mtx_lock(&aio_job_mtx);
num_aio_procs++;
if (start != NULL)
Add bracket.
2006-01-23 23:46:30 +00:00
(*start)--;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_unlock(&aio_job_mtx);
} else {
free_unr(aiod_unr, id);
}
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (error);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
/*
Eliminate the aio_freejobs list. Its purpose was to store free aiocb's allocated by zalloc(). In other words, zfree() was never called. Now, we call zfree(). Why eliminate this micro- optimization? At some later point, when we multithread the AIO system, we would need a mutex to synchronize access to aio_freejobs, making its use nearly indistinguishable in cost from zalloc() and zfree(). Remove unnecessary fhold() and fdrop() calls from aio_qphysio(), undo'ing a part of revision 1.86. The reference count on the file structure is already incremented by _aio_aqueue() before it calls aio_qphysio(). (Update the comments to document this fact.) Remove unnecessary casts from _aio_aqueue(), aio_read(), aio_write() and aio_waitcomplete(). Remove an unnecessary "return;" from aio_process(). Add "static" in various places.
2001-03-05 01:30:23 +00:00
* Try the high-performance, low-overhead physio method for eligible
* VCHR devices. This method doesn't use an aio helper thread, and
clean up whitespace...
2004-08-13 17:43:53 +00:00
* thus has very low overhead.
Eliminate the aio_freejobs list. Its purpose was to store free aiocb's allocated by zalloc(). In other words, zfree() was never called. Now, we call zfree(). Why eliminate this micro- optimization? At some later point, when we multithread the AIO system, we would need a mutex to synchronize access to aio_freejobs, making its use nearly indistinguishable in cost from zalloc() and zfree(). Remove unnecessary fhold() and fdrop() calls from aio_qphysio(), undo'ing a part of revision 1.86. The reference count on the file structure is already incremented by _aio_aqueue() before it calls aio_qphysio(). (Update the comments to document this fact.) Remove unnecessary casts from _aio_aqueue(), aio_read(), aio_write() and aio_waitcomplete(). Remove an unnecessary "return;" from aio_process(). Add "static" in various places.
2001-03-05 01:30:23 +00:00
*
1) Merge _aio_aqueue and aio_aqueue, check quota in aio_aqueue, so that lio_listio won't exceed the quota. 2) Remove lio_ref_count, it is no longer used.
2006-01-23 02:49:34 +00:00
* Assumes that the caller, aio_aqueue(), has incremented the file
Eliminate the aio_freejobs list. Its purpose was to store free aiocb's allocated by zalloc(). In other words, zfree() was never called. Now, we call zfree(). Why eliminate this micro- optimization? At some later point, when we multithread the AIO system, we would need a mutex to synchronize access to aio_freejobs, making its use nearly indistinguishable in cost from zalloc() and zfree(). Remove unnecessary fhold() and fdrop() calls from aio_qphysio(), undo'ing a part of revision 1.86. The reference count on the file structure is already incremented by _aio_aqueue() before it calls aio_qphysio(). (Update the comments to document this fact.) Remove unnecessary casts from _aio_aqueue(), aio_read(), aio_write() and aio_waitcomplete(). Remove an unnecessary "return;" from aio_process(). Add "static" in various places.
2001-03-05 01:30:23 +00:00
* structure's reference count, preventing its deallocation for the
clean up whitespace...
2004-08-13 17:43:53 +00:00
* duration of this call.
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
*/
Eliminate the aio_freejobs list. Its purpose was to store free aiocb's allocated by zalloc(). In other words, zfree() was never called. Now, we call zfree(). Why eliminate this micro- optimization? At some later point, when we multithread the AIO system, we would need a mutex to synchronize access to aio_freejobs, making its use nearly indistinguishable in cost from zalloc() and zfree(). Remove unnecessary fhold() and fdrop() calls from aio_qphysio(), undo'ing a part of revision 1.86. The reference count on the file structure is already incremented by _aio_aqueue() before it calls aio_qphysio(). (Update the comments to document this fact.) Remove unnecessary casts from _aio_aqueue(), aio_read(), aio_write() and aio_waitcomplete(). Remove an unnecessary "return;" from aio_process(). Add "static" in various places.
2001-03-05 01:30:23 +00:00
static int
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
aio_qphysio(struct proc *p, struct aiocblist *aiocbe)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
struct aiocb *cb;
struct file *fp;
struct buf *bp;
struct vnode *vp;
Fix a race with the vnode reclamation in the aio_qphysio(). Obtain the thread reference on the vp->v_rdev and use the returned struct cdev *dev instead of using vp->v_rdev. Call dev_strategy_csw() instead of dev_strategy(), since we now own the reference. Since the csw was already calculated, test d_flags to avoid mapping the buffer if the driver supports unmapped requests [*]. Suggested by: kan [*] Reviewed by: kan (previous version) Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2013-03-27 11:47:52 +00:00
struct cdevsw *csw;
struct cdev *dev;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
struct kaioinfo *ki;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
struct aioliojob *lj;
Fix a race with the vnode reclamation in the aio_qphysio(). Obtain the thread reference on the vp->v_rdev and use the returned struct cdev *dev instead of using vp->v_rdev. Call dev_strategy_csw() instead of dev_strategy(), since we now own the reference. Since the csw was already calculated, test d_flags to avoid mapping the buffer if the driver supports unmapped requests [*]. Suggested by: kan [*] Reviewed by: kan (previous version) Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2013-03-27 11:47:52 +00:00
int error, ref;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
cb = &aiocbe->uaiocb;
o Clearing p/td_retval[0] after aio_newproc() is unnecessary. (We stopped calling rfork() to create aio threads in revision 1.46.) o Don't recompute the FILE * when it's already stored in the kernel's AIOCB.
2002-02-12 17:40:41 +00:00
fp = aiocbe->fd_file;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
Add new system call - aio_mlock(). The name speaks for itself. It allows to perform the mlock(2) operation, which can consume a lot of time, under control of aio(4). Reviewed by: kib, jilles Sponsored by: Nginx, Inc.
2013-06-08 13:27:57 +00:00
if (fp == NULL || fp->f_type != DTYPE_VNODE)
Simplify and de-bogotify check for raw disk.
1999-11-07 13:09:09 +00:00
return (-1);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
Add a f_vnode field to struct file. Several of the subtypes have an associated vnode which is used for stuff like the f*() functions. By giving the vnode a speparate field, a number of checks for the specific subtype can be replaced simply with a check for f_vnode != NULL, and we can later free f_data up to subtype specific use. At this point in time, f_data still points to the vnode, so any code I might have overlooked will still work.
2003-06-22 08:41:43 +00:00
vp = fp->f_vnode;
Fix error handling for VCHR type I/O. Also, fix another spl problem, and remove alot of overly verbose debugging statements. ioproclist { int aioprocflags; /* AIO proc flags */ TAILQ_ENTRY(aioproclist) list; /* List of processes */ struct proc *aioproc; /* The AIO thread */ TAILQ_HEAD (,aiocblist) jobtorun; /* suggested job to run */ }; /* * data-structure for lio signal management */ struct aio_liojob { int lioj_flags; int lioj_buffer_count; int lioj_buffer_finished_count; int lioj_queue_count; int lioj_queue_finished_count; struct sigevent lioj_signal; /* signal on all I/O done */ TAILQ_ENTRY (aio_liojob) lioj_list; struct kaioinfo *lioj_ki; }; #define LIOJ_SIGNAL 0x1 /* signal on all done (lio) */ #define LIOJ_SIGNAL_POSTED 0x2 /* signal has been posted */ /* * per process aio data structure */ struct kaioinfo { int kaio_flags; /* per process kaio flags */ int kaio_maxactive_count; /* maximum number of AIOs */ int kaio_active_count; /* number of currently used AIOs */ int kaio_qallowed_count; /* maxiumu size of AIO queue */ int kaio_queue_count; /* size of AIO queue */ int kaio_ballowed_count; /* maximum number of buffers */ int kaio_queue_finished_count; /* number of daemon jobs finished */ int kaio_buffer_count; /* number of physio buffers */ int kaio_buffer_finished_count; /* count of I/O done */ struct proc *kaio_p; /* process that uses this kaio block */ TAILQ_HEAD (,aio_liojob) kaio_liojoblist; /* list of lio jobs */ TAILQ_HEAD (,aiocblist) kaio_jobqueue; /* job queue for process */ TAILQ_HEAD (,aiocblist) kaio_jobdone; /* done queue for process */ TAILQ_HEAD (,aiocblist) kaio_bufqueue; /* buffer job queue for process */ TAILQ_HEAD (,aiocblist) kaio_bufdone; /* buffer done queue for process */ }; #define KAIO_RUNDOWN 0x1 /* process is being run down */ #define KAIO_WAKEUP 0x2 /* wakeup process when there is a significant event */ TAILQ_HEAD (,aioproclist) aio_freeproc, aio_activeproc; TAILQ_HEAD(,aiocblist) aio_jobs; /* Async job list */ TAILQ_HEAD(,aiocblist) aio_bufjobs; /* Phys I/O job list */ TAILQ_HEAD(,aiocblist) aio_freejobs; /* Pool of free jobs */ static void aio_init_aioinfo(struct proc *p) ; static void aio_onceonly(void *) ; static int aio_free_entry(struct aiocblist *aiocbe); static void aio_process(struct aiocblist *aiocbe); static int aio_newproc(void) ; static int aio_aqueue(struct proc *p, struct aiocb *job, int type) ; static void aio_physwakeup(struct buf *bp); static int aio_fphysio(struct proc *p, struct aiocblist *aiocbe, int type); static int aio_qphysio(struct proc *p, struct aiocblist *iocb); static void aio_daemon(void *uproc); SYSINIT(aio, SI_SUB_VFS, SI_ORDER_ANY, aio_onceonly, NULL); static vm_zone_t kaio_zone=0, aiop_zone=0, aiocb_zone=0, aiol_zone=0, aiolio_zone=0; /* * Single AIOD vmspace shared amongst all of them */ static struct vmspace *aiovmspace = NULL; /* * Startup initialization */ void aio_onceonly(void *na) { TAILQ_INIT(&aio_freeproc); TAILQ_INIT(&aio_activeproc); TAILQ_INIT(&aio_jobs); TAILQ_INIT(&aio_bufjobs); TAILQ_INIT(&aio_freejobs); kaio_zone = zinit("AIO", sizeof (struct kaioinfo), 0, 0, 1); aiop_zone = zinit("AIOP", sizeof (struct aioproclist), 0, 0, 1); aiocb_zone = zinit("AIOCB", sizeof (struct aiocblist), 0, 0, 1); aiol_zone = zinit("AIOL", AIO_LISTIO_MAX * sizeof (int), 0, 0, 1); aiolio_zone = zinit("AIOLIO", AIO_LISTIO_MAX * sizeof (struct aio_liojob), 0, 0, 1); aiod_timeout = AIOD_TIMEOUT_DEFAULT; aiod_lifetime = AIOD_LIFETIME_DEFAULT; jobrefid = 1; } /* * Init the per-process aioinfo structure. * The aioinfo limits are set per-process for user limit (resource) management. */ void aio_init_aioinfo(struct proc *p) { struct kaioinfo *ki; if (p->p_aioinfo == NULL) { ki = zalloc(kaio_zone); p->p_aioinfo = ki
1997-12-01 07:01:45 +00:00
Fix vn_isdisk() usage to make AIO work on non-disk-files again, rather than just return ENOTBLK. PR: 16163 Submitted by: Adrian Chadd <adrian@FreeBSD.org>
2000-01-17 21:18:39 +00:00
/*
* If its not a disk, we don't want to return a positive error.
* It causes the aio code to not fall through to try the thread
* way when you're talking to a regular file.
*/
if (!vn_isdisk(vp, &error)) {
if (error == ENOTBLK)
return (-1);
else
return (error);
}
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
If block size is zero, use normal file operations to do I/O, this eliminates a divided-by-zero fault. Recommended by: phk
2006-02-22 00:05:12 +00:00
if (vp->v_bufobj.bo_bsize == 0)
return (-1);
Put the I/O block size in bufobj->bo_bsize. We keep si_bsize_phys around for now as that is the simplest way to pull the number out of disk device drivers in devfs_open(). The correct solution would be to do an ioctl(DIOCGSECTORSIZE), but the point is probably mooth when filesystems sit on GEOM, so don't bother for now.
2004-10-26 07:39:12 +00:00
if (cb->aio_nbytes % vp->v_bufobj.bo_bsize)
Simplify and de-bogotify check for raw disk.
1999-11-07 13:09:09 +00:00
return (-1);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
When aio_read/write() is used on a raw device, physical buffers are used for up to "vfs.aio.max_buf_aio" of the requests. If a request size is MAXPHYS, but the request base isn't page aligned, vmapbuf() will map the end of the user space buffer into the start of the kva allocated for the next physical buffer. Don't use a physical buffer in this case. (This change addresses problem report 25617.) When an aio_read/write() on a raw device has completed, timeout() is used to schedule a signal to the process. Thus, the reporting is delayed up to 10 ms (assuming hz is 100). The process might have terminated in the meantime, causing a trap 12 when attempting to deliver the signal. Thus, the timeout must be cancelled when removing the job. aio jobs in state JOBST_JOBQGLOBAL should be removed from the kaio_jobqueue list during process rundown. During process rundown, some aio jobs might move from one list to a different list that has already been "emptied", causing the rundown to be incomplete. Retry the rundown. A call to BUF_KERNPROC() is needed after obtaining a physical buffer to disassociate the lock from the running process since it can return to userland without releasing that lock. PR: 25617 Submitted by: tegge
2001-03-10 22:47:57 +00:00
if (cb->aio_nbytes >
MAXPHYS - (((vm_offset_t) cb->aio_buf) & PAGE_MASK))
Simplify and de-bogotify check for raw disk.
1999-11-07 13:09:09 +00:00
return (-1);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
ki = p->p_aioinfo;
clean up whitespace...
2004-08-13 17:43:53 +00:00
if (ki->kaio_buffer_count >= ki->kaio_ballowed_count)
Simplify and de-bogotify check for raw disk.
1999-11-07 13:09:09 +00:00
return (-1);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
Fix a race with the vnode reclamation in the aio_qphysio(). Obtain the thread reference on the vp->v_rdev and use the returned struct cdev *dev instead of using vp->v_rdev. Call dev_strategy_csw() instead of dev_strategy(), since we now own the reference. Since the csw was already calculated, test d_flags to avoid mapping the buffer if the driver supports unmapped requests [*]. Suggested by: kan [*] Reviewed by: kan (previous version) Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2013-03-27 11:47:52 +00:00
ref = 0;
csw = devvn_refthread(vp, &dev, &ref);
if (csw == NULL)
return (ENXIO);
if (cb->aio_nbytes > dev->si_iosize_max) {
error = -1;
goto unref;
}
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/* Create and build a buffer header for a transfer. */
This is a rather large commit that encompasses the new swapper, changes to the VM system to support the new swapper, VM bug fixes, several VM optimizations, and some additional revamping of the VM code. The specific bug fixes will be documented with additional forced commits. This commit is somewhat rough in regards to code cleanup issues. Reviewed by: "John S. Dyson" <root@dyson.iquest.net>, "David Greenman" <dg@root.com>
1999-01-21 08:29:12 +00:00
bp = (struct buf *)getpbuf(NULL);
When aio_read/write() is used on a raw device, physical buffers are used for up to "vfs.aio.max_buf_aio" of the requests. If a request size is MAXPHYS, but the request base isn't page aligned, vmapbuf() will map the end of the user space buffer into the start of the kva allocated for the next physical buffer. Don't use a physical buffer in this case. (This change addresses problem report 25617.) When an aio_read/write() on a raw device has completed, timeout() is used to schedule a signal to the process. Thus, the reporting is delayed up to 10 ms (assuming hz is 100). The process might have terminated in the meantime, causing a trap 12 when attempting to deliver the signal. Thus, the timeout must be cancelled when removing the job. aio jobs in state JOBST_JOBQGLOBAL should be removed from the kaio_jobqueue list during process rundown. During process rundown, some aio jobs might move from one list to a different list that has already been "emptied", causing the rundown to be incomplete. Retry the rundown. A call to BUF_KERNPROC() is needed after obtaining a physical buffer to disassociate the lock from the running process since it can return to userland without releasing that lock. PR: 25617 Submitted by: tegge
2001-03-10 22:47:57 +00:00
BUF_KERNPROC(bp);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
ki->kaio_count++;
ki->kaio_buffer_count++;
lj = aiocbe->lio;
if (lj)
lj->lioj_count++;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Get a copy of the kva from the physical buffer.
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
*/
Don't reinitialize fields that are already initialized by getpbuf().
2003-04-05 23:02:58 +00:00
error = 0;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
bp->b_bcount = cb->aio_nbytes;
bp->b_bufsize = cb->aio_nbytes;
bp->b_iodone = aio_physwakeup;
bp->b_saveaddr = bp->b_data;
o Eliminate unnecessary synchronization from filt_aiodetach(). o The manual page for kevent says that EVFILT_AIO returns under the same conditions as aio_error(). With that in mind, set the data field of the returned struct kevent to the value that would be returned by aio_error(). o Fix two compilation warnings.
2001-12-09 08:16:36 +00:00
bp->b_data = (void *)(uintptr_t)cb->aio_buf;
We need to initialize bp->b_offset and bp->b_iooffset becuase bp->b_blkno is ignored now.
2003-10-21 13:18:19 +00:00
bp->b_offset = cb->aio_offset;
bp->b_iooffset = cb->aio_offset;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
bp->b_blkno = btodb(cb->aio_offset);
o Remove useracc() calls from aio_qphysio(); they are redundant given the checks performed by vmapbuf(). Reviewed by: tegge
2003-04-04 06:26:28 +00:00
bp->b_iocmd = cb->aio_lio_opcode == LIO_WRITE ? BIO_WRITE : BIO_READ;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
Close the remaining user address mapping races for physical I/O, CAM, and AIO. Still TODO: streamline useracc() checks. Reviewed by: alc, tegge MFC after: 7 days
2003-01-20 17:46:48 +00:00
/*
* Bring buffer into kernel space.
*/
Change the way that unmapped I/O capability is advertised. The previous method was to set the D_UNMAPPED_IO flag in the cdevsw for the driver. The problem with this is that in many cases (e.g. sa(4)) there may be some instances of the driver that can handle unmapped I/O and some that can't. The isp(4) driver can handle unmapped I/O, but the esp(4) driver currently cannot. The cdevsw is shared among all driver instances. So instead of setting a flag on the cdevsw, set a flag on the cdev. This allows drivers to indicate support for unmapped I/O on a per-instance basis. sys/conf.h: Remove the D_UNMAPPED_IO cdevsw flag and replace it with an SI_UNMAPPED cdev flag. kern_physio.c: Look at the cdev SI_UNMAPPED flag to determine whether or not a particular driver can handle unmapped I/O. geom_dev.c: Set the SI_UNMAPPED flag for all GEOM cdevs. Since GEOM will create a temporary mapping when needed, setting SI_UNMAPPED unconditionally will work. Remove the D_UNMAPPED_IO flag. nvme_ns.c: Set the SI_UNMAPPED flag on cdevs created here if NVME_UNMAPPED_BIO_SUPPORT is enabled. vfs_aio.c: In aio_qphysio(), check the SI_UNMAPPED flag on a cdev instead of the D_UNMAPPED_IO flag on the cdevsw. sys/param.h: Bump __FreeBSD_version to 1000045 for the switch from setting the D_UNMAPPED_IO flag in the cdevsw to setting SI_UNMAPPED in the cdev. Reviewed by: kib, jimharris MFC after: 1 week Sponsored by: Spectra Logic
2013-08-15 22:52:39 +00:00
if (vmapbuf(bp, (dev->si_flags & SI_UNMAPPED) == 0) < 0) {
Close the remaining user address mapping races for physical I/O, CAM, and AIO. Still TODO: streamline useracc() checks. Reviewed by: alc, tegge MFC after: 7 days
2003-01-20 17:46:48 +00:00
error = EFAULT;
goto doerror;
}
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
aiocbe->bp = bp;
Fix asynchronous physio breakage introduced in rev 1.163. We cannnot use bp->b_caller2 because DEV_STRATEGY will overwrite it.
2003-09-10 15:48:51 +00:00
bp->b_caller1 = (void *)aiocbe;
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
TAILQ_INSERT_TAIL(&ki->kaio_bufqueue, aiocbe, plist);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TAILQ_INSERT_TAIL(&ki->kaio_all, aiocbe, allist);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
aiocbe->jobstate = JOBST_JOBQBUF;
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
cb->_aiocb_private.status = cb->aio_nbytes;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
atomic_add_int(&num_queue_count, 1);
atomic_add_int(&num_buf_aio, 1);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
bp->b_error = 0;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TASK_INIT(&aiocbe->biotask, 0, biohelper, aiocbe);
clean up whitespace...
2004-08-13 17:43:53 +00:00
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/* Perform transfer. */
Fix a race with the vnode reclamation in the aio_qphysio(). Obtain the thread reference on the vp->v_rdev and use the returned struct cdev *dev instead of using vp->v_rdev. Call dev_strategy_csw() instead of dev_strategy(), since we now own the reference. Since the csw was already calculated, test d_flags to avoid mapping the buffer if the driver supports unmapped requests [*]. Suggested by: kan [*] Reviewed by: kan (previous version) Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2013-03-27 11:47:52 +00:00
dev_strategy_csw(dev, csw, bp);
dev_relthread(dev, ref);
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (0);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
doerror:
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
ki->kaio_count--;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
ki->kaio_buffer_count--;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
if (lj)
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
lj->lioj_count--;
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
aiocbe->bp = NULL;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
This is a rather large commit that encompasses the new swapper, changes to the VM system to support the new swapper, VM bug fixes, several VM optimizations, and some additional revamping of the VM code. The specific bug fixes will be documented with additional forced commits. This commit is somewhat rough in regards to code cleanup issues. Reviewed by: "John S. Dyson" <root@dyson.iquest.net>, "David Greenman" <dg@root.com>
1999-01-21 08:29:12 +00:00
relpbuf(bp, NULL);
Fix a race with the vnode reclamation in the aio_qphysio(). Obtain the thread reference on the vp->v_rdev and use the returned struct cdev *dev instead of using vp->v_rdev. Call dev_strategy_csw() instead of dev_strategy(), since we now own the reference. Since the csw was already calculated, test d_flags to avoid mapping the buffer if the driver supports unmapped requests [*]. Suggested by: kan [*] Reviewed by: kan (previous version) Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
2013-03-27 11:47:52 +00:00
unref:
dev_relthread(dev, ref);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
return (error);
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Wake up aio requests that may be serviceable now.
*/
o Add missing synchronization (splnet()/splx()) in aio_free_entry(). o Move the definition of struct aiocblist from sys/aio.h to kern/vfs_aio.c. o Make aio_swake_cb() static.
2002-01-06 21:03:39 +00:00
static void
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
aio_swake_cb(struct socket *so, struct sockbuf *sb)
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
{
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
struct aiocblist *cb, *cbn;
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
int opcode;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
Rework socket upcalls to close some races with setup/teardown of upcalls. - Each socket upcall is now invoked with the appropriate socket buffer locked. It is not permissible to call soisconnected() with this lock held; however, so socket upcalls now return an integer value. The two possible values are SU_OK and SU_ISCONNECTED. If an upcall returns SU_ISCONNECTED, then the soisconnected() will be invoked on the socket after the socket buffer lock is dropped. - A new API is provided for setting and clearing socket upcalls. The API consists of soupcall_set() and soupcall_clear(). - To simplify locking, each socket buffer now has a separate upcall. - When a socket upcall returns SU_ISCONNECTED, the upcall is cleared from the receive socket buffer automatically. Note that a SO_SND upcall should never return SU_ISCONNECTED. - All this means that accept filters should now return SU_ISCONNECTED instead of calling soisconnected() directly. They also no longer need to explicitly clear the upcall on the new socket. - The HTTP accept filter still uses soupcall_set() to manage its internal state machine, but other accept filters no longer have any explicit knowlege of socket upcall internals aside from their return value. - The various RPC client upcalls currently drop the socket buffer lock while invoking soreceive() as a temporary band-aid. The plan for the future is to add a new flag to allow soreceive() to be called with the socket buffer locked. - The AIO callback for socket I/O is now also invoked with the socket buffer locked. Previously sowakeup() would drop the socket buffer lock only to call aio_swake() which immediately re-acquired the socket buffer lock for the duration of the function call. Discussed with: rwatson, rmacklem
2009-06-01 21:17:03 +00:00
SOCKBUF_LOCK_ASSERT(sb);
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
if (sb == &so->so_snd)
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
opcode = LIO_WRITE;
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
else
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
opcode = LIO_READ;
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
sb->sb_flags &= ~SB_AIO;
mtx_lock(&aio_job_mtx);
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
TAILQ_FOREACH_SAFE(cb, &so->so_aiojobq, list, cbn) {
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
if (opcode == cb->uaiocb.aio_lio_opcode) {
Fix a bogus panic.
2006-01-22 09:39:59 +00:00
if (cb->jobstate != JOBST_JOBQSOCK)
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
panic("invalid queue value");
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
/* XXX
* We don't have actual sockets backend yet,
* so we simply move the requests to the generic
* file I/O backend.
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
*/
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
TAILQ_REMOVE(&so->so_aiojobq, cb, list);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TAILQ_INSERT_TAIL(&aio_jobs, cb, list);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
aio_kick_nowait(cb->userproc);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
}
}
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
mtx_unlock(&aio_job_mtx);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
}
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
static int
convert_old_sigevent(struct osigevent *osig, struct sigevent *nsig)
{
/*
* Only SIGEV_NONE, SIGEV_SIGNAL, and SIGEV_KEVENT are
* supported by AIO with the old sigevent structure.
*/
nsig->sigev_notify = osig->sigev_notify;
switch (nsig->sigev_notify) {
case SIGEV_NONE:
break;
case SIGEV_SIGNAL:
nsig->sigev_signo = osig->__sigev_u.__sigev_signo;
break;
case SIGEV_KEVENT:
nsig->sigev_notify_kqueue =
osig->__sigev_u.__sigev_notify_kqueue;
nsig->sigev_value.sival_ptr = osig->sigev_value.sival_ptr;
break;
default:
return (EINVAL);
}
return (0);
}
static int
aiocb_copyin_old_sigevent(struct aiocb *ujob, struct aiocb *kjob)
{
struct oaiocb *ojob;
int error;
bzero(kjob, sizeof(struct aiocb));
error = copyin(ujob, kjob, sizeof(struct oaiocb));
if (error)
return (error);
ojob = (struct oaiocb *)kjob;
return (convert_old_sigevent(&ojob->aio_sigevent, &kjob->aio_sigevent));
}
static int
aiocb_copyin(struct aiocb *ujob, struct aiocb *kjob)
{
return (copyin(ujob, kjob, sizeof(struct aiocb)));
}
static long
aiocb_fetch_status(struct aiocb *ujob)
{
return (fuword(&ujob->_aiocb_private.status));
}
static long
aiocb_fetch_error(struct aiocb *ujob)
{
return (fuword(&ujob->_aiocb_private.error));
}
static int
aiocb_store_status(struct aiocb *ujob, long status)
{
return (suword(&ujob->_aiocb_private.status, status));
}
static int
aiocb_store_error(struct aiocb *ujob, long error)
{
return (suword(&ujob->_aiocb_private.error, error));
}
static int
aiocb_store_kernelinfo(struct aiocb *ujob, long jobref)
{
return (suword(&ujob->_aiocb_private.kernelinfo, jobref));
}
static int
aiocb_store_aiocb(struct aiocb **ujobp, struct aiocb *ujob)
{
return (suword(ujobp, (long)ujob));
}
static struct aiocb_ops aiocb_ops = {
.copyin = aiocb_copyin,
.fetch_status = aiocb_fetch_status,
.fetch_error = aiocb_fetch_error,
.store_status = aiocb_store_status,
.store_error = aiocb_store_error,
.store_kernelinfo = aiocb_store_kernelinfo,
.store_aiocb = aiocb_store_aiocb,
};
static struct aiocb_ops aiocb_ops_osigevent = {
.copyin = aiocb_copyin_old_sigevent,
.fetch_status = aiocb_fetch_status,
.fetch_error = aiocb_fetch_error,
.store_status = aiocb_store_status,
.store_error = aiocb_store_error,
.store_kernelinfo = aiocb_store_kernelinfo,
.store_aiocb = aiocb_store_aiocb,
};
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/*
* Queue a new AIO request. Choosing either the threaded or direct physio VCHR
* technique is done in this code.
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
*/
MFP4 (with some minor changes): Implement the linux_io_* syscalls (AIO). They are only enabled if the native AIO code is available (either compiled in to the kernel or as a module) at the time the functions are used. If the AIO stuff is not available there will be a ENOSYS. From the submitter: ---snip--- DESIGN NOTES: 1. Linux permits a process to own multiple AIO queues (distinguished by "context"), but FreeBSD creates only one single AIO queue per process. My code maintains a request queue (STAILQ of queue(3)) per "context", and throws all AIO requests of all contexts owned by a process into the single FreeBSD per-process AIO queue. When the process calls io_destroy(2), io_getevents(2), io_submit(2) and io_cancel(2), my code can pick out requests owned by the specified context from the single FreeBSD per-process AIO queue according to the per-context request queues maintained by my code. 2. The request queue maintained by my code stores contrast information between Linux IO control blocks (struct linux_iocb) and FreeBSD IO control blocks (struct aiocb). FreeBSD IO control block actually exists in userland memory space, required by FreeBSD native aio_XXXXXX(2). 3. It is quite troubling that the function io_getevents() of libaio-0.3.105 needs to use Linux-specific "struct aio_ring", which is a partial mirror of context in user space. I would rather take the address of context in kernel as the context ID, but the io_getevents() of libaio forces me to take the address of the "ring" in user space as the context ID. To my surprise, one comment line in the file "io_getevents.c" of libaio-0.3.105 reads: Ben will hate me for this REFERENCE: 1. Linux kernel source code: http://www.kernel.org/pub/linux/kernel/v2.6/ (include/linux/aio_abi.h, fs/aio.c) 2. Linux manual pages: http://www.kernel.org/pub/linux/docs/manpages/ (io_setup(2), io_destroy(2), io_getevents(2), io_submit(2), io_cancel(2)) 3. Linux Scalability Effort: http://lse.sourceforge.net/io/aio.html The design notes: http://lse.sourceforge.net/io/aionotes.txt 4. The package libaio, both source and binary: http://rpmfind.net/linux/rpm2html/search.php?query=libaio Simple transparent interface to Linux AIO system calls. 5. Libaio-oracle: http://oss.oracle.com/projects/libaio-oracle/ POSIX AIO implementation based on Linux AIO system calls (depending on libaio). ---snip--- Submitted by: Li, Xiao <intron@intron.ac>
2006-10-15 14:22:14 +00:00
int
1) Merge _aio_aqueue and aio_aqueue, check quota in aio_aqueue, so that lio_listio won't exceed the quota. 2) Remove lio_ref_count, it is no longer used.
2006-01-23 02:49:34 +00:00
aio_aqueue(struct thread *td, struct aiocb *job, struct aioliojob *lj,
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
int type, struct aiocb_ops *ops)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct proc *p = td->td_proc;
Change the cap_rights_t type from uint64_t to a structure that we can extend in the future in a backward compatible (API and ABI) way. The cap_rights_t represents capability rights. We used to use one bit to represent one right, but we are running out of spare bits. Currently the new structure provides place for 114 rights (so 50 more than the previous cap_rights_t), but it is possible to grow the structure to hold at least 285 rights, although we can make it even larger if 285 rights won't be enough. The structure definition looks like this: struct cap_rights { uint64_t cr_rights[CAP_RIGHTS_VERSION + 2]; }; The initial CAP_RIGHTS_VERSION is 0. The top two bits in the first element of the cr_rights[] array contain total number of elements in the array - 2. This means if those two bits are equal to 0, we have 2 array elements. The top two bits in all remaining array elements should be 0. The next five bits in all array elements contain array index. Only one bit is used and bit position in this five-bits range defines array index. This means there can be at most five array elements in the future. To define new right the CAPRIGHT() macro must be used. The macro takes two arguments - an array index and a bit to set, eg. #define CAP_PDKILL CAPRIGHT(1, 0x0000000000000800ULL) We still support aliases that combine few rights, but the rights have to belong to the same array element, eg: #define CAP_LOOKUP CAPRIGHT(0, 0x0000000000000400ULL) #define CAP_FCHMOD CAPRIGHT(0, 0x0000000000002000ULL) #define CAP_FCHMODAT (CAP_FCHMOD | CAP_LOOKUP) There is new API to manage the new cap_rights_t structure: cap_rights_t *cap_rights_init(cap_rights_t *rights, ...); void cap_rights_set(cap_rights_t *rights, ...); void cap_rights_clear(cap_rights_t *rights, ...); bool cap_rights_is_set(const cap_rights_t *rights, ...); bool cap_rights_is_valid(const cap_rights_t *rights); void cap_rights_merge(cap_rights_t *dst, const cap_rights_t *src); void cap_rights_remove(cap_rights_t *dst, const cap_rights_t *src); bool cap_rights_contains(const cap_rights_t *big, const cap_rights_t *little); Capability rights to the cap_rights_init(), cap_rights_set(), cap_rights_clear() and cap_rights_is_set() functions are provided by separating them with commas, eg: cap_rights_t rights; cap_rights_init(&rights, CAP_READ, CAP_WRITE, CAP_FSTAT); There is no need to terminate the list of rights, as those functions are actually macros that take care of the termination, eg: #define cap_rights_set(rights, ...) \ __cap_rights_set((rights), __VA_ARGS__, 0ULL) void __cap_rights_set(cap_rights_t *rights, ...); Thanks to using one bit as an array index we can assert in those functions that there are no two rights belonging to different array elements provided together. For example this is illegal and will be detected, because CAP_LOOKUP belongs to element 0 and CAP_PDKILL to element 1: cap_rights_init(&rights, CAP_LOOKUP | CAP_PDKILL); Providing several rights that belongs to the same array's element this way is correct, but is not advised. It should only be used for aliases definition. This commit also breaks compatibility with some existing Capsicum system calls, but I see no other way to do that. This should be fine as Capsicum is still experimental and this change is not going to 9.x. Sponsored by: The FreeBSD Foundation
2013-09-05 00:09:56 +00:00
cap_rights_t rights;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct file *fp;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
struct socket *so;
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
struct aiocblist *aiocbe, *cb;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct kaioinfo *ki;
Provide a new interface for the user of aio_read() and aio_write() to request a kevent upon completion of the I/O. Specifically, introduce a new type of sigevent notification, SIGEV_EVENT. If sigev_notify is SIGEV_EVENT, then sigev_notify_kqueue names the kqueue that should receive the event and sigev_value contains the "void *" is copied into the kevent's udata field. In contrast to the existing interface, this one: 1) works on the Alpha 2) avoids the extra copyin() call for the kevent because all of the information needed is in the sigevent and 3) could be applied to request a single kevent upon completion of an entire lio_listio(). Reviewed by: jlemon
2000-11-21 19:36:36 +00:00
struct kevent kev;
cover soreadable and sowriteable with the corresponding socketbuffer locks.
2004-10-01 05:54:06 +00:00
struct sockbuf *sb;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
int opcode;
int error;
hide kqueue_register from public view, and replace it w/ kqfd_register... this eliminates a possible race in aio registering a kevent..
2006-09-24 04:47:47 +00:00
int fd, kqfd;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
int jid;
If multiple threads call kevent() to get AIO events on same kqueue fd, it is possible that a single AIO event will be reported to multiple threads, it is not threading friendly, and the existing API can not control this behavior. Allocate a kevent flags field sigev_notify_kevent_flags for AIO event notification in sigevent, and allow user to pass EV_CLEAR, EV_DISPATCH or EV_ONESHOT to AIO kernel code, user can control whether the event should be cleared once it is retrieved by a thread. This change should be comptaible with existing application, because the field should have already been zero-filled, and no additional action will be taken by kernel. PR: kern/156567
2012-02-01 02:53:06 +00:00
u_short evflags;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
1) Merge _aio_aqueue and aio_aqueue, check quota in aio_aqueue, so that lio_listio won't exceed the quota. 2) Remove lio_ref_count, it is no longer used.
2006-01-23 02:49:34 +00:00
if (p->p_aioinfo == NULL)
aio_init_aioinfo(p);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
ki = p->p_aioinfo;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
ops->store_status(job, -1);
ops->store_error(job, 0);
ops->store_kernelinfo(job, -1);
1) Merge _aio_aqueue and aio_aqueue, check quota in aio_aqueue, so that lio_listio won't exceed the quota. 2) Remove lio_ref_count, it is no longer used.
2006-01-23 02:49:34 +00:00
if (num_queue_count >= max_queue_count ||
ki->kaio_count >= ki->kaio_qallowed_count) {
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
ops->store_error(job, EAGAIN);
1) Merge _aio_aqueue and aio_aqueue, check quota in aio_aqueue, so that lio_listio won't exceed the quota. 2) Remove lio_ref_count, it is no longer used.
2006-01-23 02:49:34 +00:00
return (EAGAIN);
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
aiocbe = uma_zalloc(aiocb_zone, M_WAITOK | M_ZERO);
Adapt vfs kqfilter to the shared vnode lock used by zfs write vop. Use vnode interlock to protect the knote fields [1]. The locking assumes that shared vnode lock is held, thus we get exclusive access to knote either by exclusive vnode lock protection, or by shared vnode lock + vnode interlock. Do not use kl_locked() method to assert either lock ownership or the fact that curthread does not own the lock. For shared locks, ownership is not recorded, e.g. VOP_ISLOCKED can return LK_SHARED for the shared lock not owned by curthread, causing false positives in kqueue subsystem assertions about knlist lock. Remove kl_locked method from knlist lock vector, and add two separate assertion methods kl_assert_locked and kl_assert_unlocked, that are supposed to use proper asserts. Change knlist_init accordingly. Add convenience function knlist_init_mtx to reduce number of arguments for typical knlist initialization. Submitted by: jhb [1] Noted by: jhb [2] Reviewed by: jhb Tested by: rnoland
2009-06-10 20:59:32 +00:00
knlist_init_mtx(&aiocbe->klist, AIO_MTX(ki));
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
error = ops->copyin(job, &aiocbe->uaiocb);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
if (error) {
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
ops->store_error(job, error);
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
uma_zfree(aiocb_zone, aiocbe);
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (error);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Verify all supported notification types.
2006-01-23 10:27:15 +00:00
Although aio_nbytes is size_t, later is is signed to casted types: to ssize_t in filesystem code and to int in buf code, thus supplying a negative argument leads to kernel panic later. To fix that check user supplied argument in the beginning of syscall. Submitted by: Maxim Dounin <mdounin mdounin.ru>, maxim@
2012-01-26 11:59:48 +00:00
/* XXX: aio_nbytes is later casted to signed types. */
Fix size check, that prevents getting negative after casting to a signed type Reviewed by: bde
2012-01-27 08:58:58 +00:00
if (aiocbe->uaiocb.aio_nbytes > INT_MAX) {
Although aio_nbytes is size_t, later is is signed to casted types: to ssize_t in filesystem code and to int in buf code, thus supplying a negative argument leads to kernel panic later. To fix that check user supplied argument in the beginning of syscall. Submitted by: Maxim Dounin <mdounin mdounin.ru>, maxim@
2012-01-26 11:59:48 +00:00
uma_zfree(aiocb_zone, aiocbe);
return (EINVAL);
}
Verify all supported notification types.
2006-01-23 10:27:15 +00:00
if (aiocbe->uaiocb.aio_sigevent.sigev_notify != SIGEV_KEVENT &&
aiocbe->uaiocb.aio_sigevent.sigev_notify != SIGEV_SIGNAL &&
aiocbe->uaiocb.aio_sigevent.sigev_notify != SIGEV_THREAD_ID &&
aiocbe->uaiocb.aio_sigevent.sigev_notify != SIGEV_NONE) {
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
ops->store_error(job, EINVAL);
Verify all supported notification types.
2006-01-23 10:27:15 +00:00
uma_zfree(aiocb_zone, aiocbe);
return (EINVAL);
}
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
Support sending realtime signal information via signal queue, realtime signal memory is pre-allocated, so kernel can always notify user code.
2005-11-03 05:25:26 +00:00
if ((aiocbe->uaiocb.aio_sigevent.sigev_notify == SIGEV_SIGNAL ||
aiocbe->uaiocb.aio_sigevent.sigev_notify == SIGEV_THREAD_ID) &&
Check validity of signal callback requested via aio routines. Also move the insertion of the request to after the request is validated, there's still looks like there may be some problems if an invalid address is passed to the aio routines, basically a possible leak or having a not completely initialized structure on the queue may still be possible. A new sig macro was made _SIG_VALID to check the validity of a signal, it would be advisable to use it from now on (in kern/kern_sig.c) rather than rolling your own. PR: kern/17152
2001-04-18 22:18:39 +00:00
!_SIG_VALID(aiocbe->uaiocb.aio_sigevent.sigev_signo)) {
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
uma_zfree(aiocb_zone, aiocbe);
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (EINVAL);
Check validity of signal callback requested via aio routines. Also move the insertion of the request to after the request is validated, there's still looks like there may be some problems if an invalid address is passed to the aio routines, basically a possible leak or having a not completely initialized structure on the queue may still be possible. A new sig macro was made _SIG_VALID to check the validity of a signal, it would be advisable to use it from now on (in kern/kern_sig.c) rather than rolling your own. PR: kern/17152
2001-04-18 22:18:39 +00:00
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Support sending realtime signal information via signal queue, realtime signal memory is pre-allocated, so kernel can always notify user code.
2005-11-03 05:25:26 +00:00
ksiginfo_init(&aiocbe->ksi);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/* Save userspace address of the job info. */
Fix error handling for VCHR type I/O. Also, fix another spl problem, and remove alot of overly verbose debugging statements. ioproclist { int aioprocflags; /* AIO proc flags */ TAILQ_ENTRY(aioproclist) list; /* List of processes */ struct proc *aioproc; /* The AIO thread */ TAILQ_HEAD (,aiocblist) jobtorun; /* suggested job to run */ }; /* * data-structure for lio signal management */ struct aio_liojob { int lioj_flags; int lioj_buffer_count; int lioj_buffer_finished_count; int lioj_queue_count; int lioj_queue_finished_count; struct sigevent lioj_signal; /* signal on all I/O done */ TAILQ_ENTRY (aio_liojob) lioj_list; struct kaioinfo *lioj_ki; }; #define LIOJ_SIGNAL 0x1 /* signal on all done (lio) */ #define LIOJ_SIGNAL_POSTED 0x2 /* signal has been posted */ /* * per process aio data structure */ struct kaioinfo { int kaio_flags; /* per process kaio flags */ int kaio_maxactive_count; /* maximum number of AIOs */ int kaio_active_count; /* number of currently used AIOs */ int kaio_qallowed_count; /* maxiumu size of AIO queue */ int kaio_queue_count; /* size of AIO queue */ int kaio_ballowed_count; /* maximum number of buffers */ int kaio_queue_finished_count; /* number of daemon jobs finished */ int kaio_buffer_count; /* number of physio buffers */ int kaio_buffer_finished_count; /* count of I/O done */ struct proc *kaio_p; /* process that uses this kaio block */ TAILQ_HEAD (,aio_liojob) kaio_liojoblist; /* list of lio jobs */ TAILQ_HEAD (,aiocblist) kaio_jobqueue; /* job queue for process */ TAILQ_HEAD (,aiocblist) kaio_jobdone; /* done queue for process */ TAILQ_HEAD (,aiocblist) kaio_bufqueue; /* buffer job queue for process */ TAILQ_HEAD (,aiocblist) kaio_bufdone; /* buffer done queue for process */ }; #define KAIO_RUNDOWN 0x1 /* process is being run down */ #define KAIO_WAKEUP 0x2 /* wakeup process when there is a significant event */ TAILQ_HEAD (,aioproclist) aio_freeproc, aio_activeproc; TAILQ_HEAD(,aiocblist) aio_jobs; /* Async job list */ TAILQ_HEAD(,aiocblist) aio_bufjobs; /* Phys I/O job list */ TAILQ_HEAD(,aiocblist) aio_freejobs; /* Pool of free jobs */ static void aio_init_aioinfo(struct proc *p) ; static void aio_onceonly(void *) ; static int aio_free_entry(struct aiocblist *aiocbe); static void aio_process(struct aiocblist *aiocbe); static int aio_newproc(void) ; static int aio_aqueue(struct proc *p, struct aiocb *job, int type) ; static void aio_physwakeup(struct buf *bp); static int aio_fphysio(struct proc *p, struct aiocblist *aiocbe, int type); static int aio_qphysio(struct proc *p, struct aiocblist *iocb); static void aio_daemon(void *uproc); SYSINIT(aio, SI_SUB_VFS, SI_ORDER_ANY, aio_onceonly, NULL); static vm_zone_t kaio_zone=0, aiop_zone=0, aiocb_zone=0, aiol_zone=0, aiolio_zone=0; /* * Single AIOD vmspace shared amongst all of them */ static struct vmspace *aiovmspace = NULL; /* * Startup initialization */ void aio_onceonly(void *na) { TAILQ_INIT(&aio_freeproc); TAILQ_INIT(&aio_activeproc); TAILQ_INIT(&aio_jobs); TAILQ_INIT(&aio_bufjobs); TAILQ_INIT(&aio_freejobs); kaio_zone = zinit("AIO", sizeof (struct kaioinfo), 0, 0, 1); aiop_zone = zinit("AIOP", sizeof (struct aioproclist), 0, 0, 1); aiocb_zone = zinit("AIOCB", sizeof (struct aiocblist), 0, 0, 1); aiol_zone = zinit("AIOL", AIO_LISTIO_MAX * sizeof (int), 0, 0, 1); aiolio_zone = zinit("AIOLIO", AIO_LISTIO_MAX * sizeof (struct aio_liojob), 0, 0, 1); aiod_timeout = AIOD_TIMEOUT_DEFAULT; aiod_lifetime = AIOD_LIFETIME_DEFAULT; jobrefid = 1; } /* * Init the per-process aioinfo structure. * The aioinfo limits are set per-process for user limit (resource) management. */ void aio_init_aioinfo(struct proc *p) { struct kaioinfo *ki; if (p->p_aioinfo == NULL) { ki = zalloc(kaio_zone); p->p_aioinfo = ki
1997-12-01 07:01:45 +00:00
aiocbe->uuaiocb = job;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/* Get the opcode. */
if (type != LIO_NOP)
Major cleanup and debugging of the new AIO/LIO code. The LIO code is now corrected. New tunables/instrumentation added. The code is now likely "good enough to use." I will add the userland support soon. The "high performance" mode for raw devices is still missing, and will be added next. POSIX system calls that now appear to work: aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, lio_listio. Missing, but to be added soon: aio_fsync.
1997-10-09 04:14:41 +00:00
aiocbe->uaiocb.aio_lio_opcode = type;
opcode = aiocbe->uaiocb.aio_lio_opcode;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Second-to-last commit implementing Capsicum capabilities in the FreeBSD kernel for FreeBSD 9.0: Add a new capability mask argument to fget(9) and friends, allowing system call code to declare what capabilities are required when an integer file descriptor is converted into an in-kernel struct file *. With options CAPABILITIES compiled into the kernel, this enforces capability protection; without, this change is effectively a no-op. Some cases require special handling, such as mmap(2), which must preserve information about the maximum rights at the time of mapping in the memory map so that they can later be enforced in mprotect(2) -- this is done by narrowing the rights in the existing max_protection field used for similar purposes with file permissions. In namei(9), we assert that the code is not reached from within capability mode, as we're not yet ready to enforce namespace capabilities there. This will follow in a later commit. Update two capability names: CAP_EVENT and CAP_KEVENT become CAP_POST_KEVENT and CAP_POLL_KEVENT to more accurately indicate what they represent. Approved by: re (bz) Submitted by: jonathan Sponsored by: Google Inc
2011-08-11 12:30:23 +00:00
/*
* Validate the opcode and fetch the file object for the specified
* file descriptor.
*
* XXXRW: Moved the opcode validation up here so that we don't
* retrieve a file descriptor without knowing what the capabiltity
* should be.
*/
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
fd = aiocbe->uaiocb.aio_fildes;
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
switch (opcode) {
case LIO_WRITE:
Change the cap_rights_t type from uint64_t to a structure that we can extend in the future in a backward compatible (API and ABI) way. The cap_rights_t represents capability rights. We used to use one bit to represent one right, but we are running out of spare bits. Currently the new structure provides place for 114 rights (so 50 more than the previous cap_rights_t), but it is possible to grow the structure to hold at least 285 rights, although we can make it even larger if 285 rights won't be enough. The structure definition looks like this: struct cap_rights { uint64_t cr_rights[CAP_RIGHTS_VERSION + 2]; }; The initial CAP_RIGHTS_VERSION is 0. The top two bits in the first element of the cr_rights[] array contain total number of elements in the array - 2. This means if those two bits are equal to 0, we have 2 array elements. The top two bits in all remaining array elements should be 0. The next five bits in all array elements contain array index. Only one bit is used and bit position in this five-bits range defines array index. This means there can be at most five array elements in the future. To define new right the CAPRIGHT() macro must be used. The macro takes two arguments - an array index and a bit to set, eg. #define CAP_PDKILL CAPRIGHT(1, 0x0000000000000800ULL) We still support aliases that combine few rights, but the rights have to belong to the same array element, eg: #define CAP_LOOKUP CAPRIGHT(0, 0x0000000000000400ULL) #define CAP_FCHMOD CAPRIGHT(0, 0x0000000000002000ULL) #define CAP_FCHMODAT (CAP_FCHMOD | CAP_LOOKUP) There is new API to manage the new cap_rights_t structure: cap_rights_t *cap_rights_init(cap_rights_t *rights, ...); void cap_rights_set(cap_rights_t *rights, ...); void cap_rights_clear(cap_rights_t *rights, ...); bool cap_rights_is_set(const cap_rights_t *rights, ...); bool cap_rights_is_valid(const cap_rights_t *rights); void cap_rights_merge(cap_rights_t *dst, const cap_rights_t *src); void cap_rights_remove(cap_rights_t *dst, const cap_rights_t *src); bool cap_rights_contains(const cap_rights_t *big, const cap_rights_t *little); Capability rights to the cap_rights_init(), cap_rights_set(), cap_rights_clear() and cap_rights_is_set() functions are provided by separating them with commas, eg: cap_rights_t rights; cap_rights_init(&rights, CAP_READ, CAP_WRITE, CAP_FSTAT); There is no need to terminate the list of rights, as those functions are actually macros that take care of the termination, eg: #define cap_rights_set(rights, ...) \ __cap_rights_set((rights), __VA_ARGS__, 0ULL) void __cap_rights_set(cap_rights_t *rights, ...); Thanks to using one bit as an array index we can assert in those functions that there are no two rights belonging to different array elements provided together. For example this is illegal and will be detected, because CAP_LOOKUP belongs to element 0 and CAP_PDKILL to element 1: cap_rights_init(&rights, CAP_LOOKUP | CAP_PDKILL); Providing several rights that belongs to the same array's element this way is correct, but is not advised. It should only be used for aliases definition. This commit also breaks compatibility with some existing Capsicum system calls, but I see no other way to do that. This should be fine as Capsicum is still experimental and this change is not going to 9.x. Sponsored by: The FreeBSD Foundation
2013-09-05 00:09:56 +00:00
error = fget_write(td, fd,
cap_rights_init(&rights, CAP_PWRITE), &fp);
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
break;
case LIO_READ:
Change the cap_rights_t type from uint64_t to a structure that we can extend in the future in a backward compatible (API and ABI) way. The cap_rights_t represents capability rights. We used to use one bit to represent one right, but we are running out of spare bits. Currently the new structure provides place for 114 rights (so 50 more than the previous cap_rights_t), but it is possible to grow the structure to hold at least 285 rights, although we can make it even larger if 285 rights won't be enough. The structure definition looks like this: struct cap_rights { uint64_t cr_rights[CAP_RIGHTS_VERSION + 2]; }; The initial CAP_RIGHTS_VERSION is 0. The top two bits in the first element of the cr_rights[] array contain total number of elements in the array - 2. This means if those two bits are equal to 0, we have 2 array elements. The top two bits in all remaining array elements should be 0. The next five bits in all array elements contain array index. Only one bit is used and bit position in this five-bits range defines array index. This means there can be at most five array elements in the future. To define new right the CAPRIGHT() macro must be used. The macro takes two arguments - an array index and a bit to set, eg. #define CAP_PDKILL CAPRIGHT(1, 0x0000000000000800ULL) We still support aliases that combine few rights, but the rights have to belong to the same array element, eg: #define CAP_LOOKUP CAPRIGHT(0, 0x0000000000000400ULL) #define CAP_FCHMOD CAPRIGHT(0, 0x0000000000002000ULL) #define CAP_FCHMODAT (CAP_FCHMOD | CAP_LOOKUP) There is new API to manage the new cap_rights_t structure: cap_rights_t *cap_rights_init(cap_rights_t *rights, ...); void cap_rights_set(cap_rights_t *rights, ...); void cap_rights_clear(cap_rights_t *rights, ...); bool cap_rights_is_set(const cap_rights_t *rights, ...); bool cap_rights_is_valid(const cap_rights_t *rights); void cap_rights_merge(cap_rights_t *dst, const cap_rights_t *src); void cap_rights_remove(cap_rights_t *dst, const cap_rights_t *src); bool cap_rights_contains(const cap_rights_t *big, const cap_rights_t *little); Capability rights to the cap_rights_init(), cap_rights_set(), cap_rights_clear() and cap_rights_is_set() functions are provided by separating them with commas, eg: cap_rights_t rights; cap_rights_init(&rights, CAP_READ, CAP_WRITE, CAP_FSTAT); There is no need to terminate the list of rights, as those functions are actually macros that take care of the termination, eg: #define cap_rights_set(rights, ...) \ __cap_rights_set((rights), __VA_ARGS__, 0ULL) void __cap_rights_set(cap_rights_t *rights, ...); Thanks to using one bit as an array index we can assert in those functions that there are no two rights belonging to different array elements provided together. For example this is illegal and will be detected, because CAP_LOOKUP belongs to element 0 and CAP_PDKILL to element 1: cap_rights_init(&rights, CAP_LOOKUP | CAP_PDKILL); Providing several rights that belongs to the same array's element this way is correct, but is not advised. It should only be used for aliases definition. This commit also breaks compatibility with some existing Capsicum system calls, but I see no other way to do that. This should be fine as Capsicum is still experimental and this change is not going to 9.x. Sponsored by: The FreeBSD Foundation
2013-09-05 00:09:56 +00:00
error = fget_read(td, fd,
cap_rights_init(&rights, CAP_PREAD), &fp);
Second-to-last commit implementing Capsicum capabilities in the FreeBSD kernel for FreeBSD 9.0: Add a new capability mask argument to fget(9) and friends, allowing system call code to declare what capabilities are required when an integer file descriptor is converted into an in-kernel struct file *. With options CAPABILITIES compiled into the kernel, this enforces capability protection; without, this change is effectively a no-op. Some cases require special handling, such as mmap(2), which must preserve information about the maximum rights at the time of mapping in the memory map so that they can later be enforced in mprotect(2) -- this is done by narrowing the rights in the existing max_protection field used for similar purposes with file permissions. In namei(9), we assert that the code is not reached from within capability mode, as we're not yet ready to enforce namespace capabilities there. This will follow in a later commit. Update two capability names: CAP_EVENT and CAP_KEVENT become CAP_POST_KEVENT and CAP_POLL_KEVENT to more accurately indicate what they represent. Approved by: re (bz) Submitted by: jonathan Sponsored by: Google Inc
2011-08-11 12:30:23 +00:00
break;
case LIO_SYNC:
Change the cap_rights_t type from uint64_t to a structure that we can extend in the future in a backward compatible (API and ABI) way. The cap_rights_t represents capability rights. We used to use one bit to represent one right, but we are running out of spare bits. Currently the new structure provides place for 114 rights (so 50 more than the previous cap_rights_t), but it is possible to grow the structure to hold at least 285 rights, although we can make it even larger if 285 rights won't be enough. The structure definition looks like this: struct cap_rights { uint64_t cr_rights[CAP_RIGHTS_VERSION + 2]; }; The initial CAP_RIGHTS_VERSION is 0. The top two bits in the first element of the cr_rights[] array contain total number of elements in the array - 2. This means if those two bits are equal to 0, we have 2 array elements. The top two bits in all remaining array elements should be 0. The next five bits in all array elements contain array index. Only one bit is used and bit position in this five-bits range defines array index. This means there can be at most five array elements in the future. To define new right the CAPRIGHT() macro must be used. The macro takes two arguments - an array index and a bit to set, eg. #define CAP_PDKILL CAPRIGHT(1, 0x0000000000000800ULL) We still support aliases that combine few rights, but the rights have to belong to the same array element, eg: #define CAP_LOOKUP CAPRIGHT(0, 0x0000000000000400ULL) #define CAP_FCHMOD CAPRIGHT(0, 0x0000000000002000ULL) #define CAP_FCHMODAT (CAP_FCHMOD | CAP_LOOKUP) There is new API to manage the new cap_rights_t structure: cap_rights_t *cap_rights_init(cap_rights_t *rights, ...); void cap_rights_set(cap_rights_t *rights, ...); void cap_rights_clear(cap_rights_t *rights, ...); bool cap_rights_is_set(const cap_rights_t *rights, ...); bool cap_rights_is_valid(const cap_rights_t *rights); void cap_rights_merge(cap_rights_t *dst, const cap_rights_t *src); void cap_rights_remove(cap_rights_t *dst, const cap_rights_t *src); bool cap_rights_contains(const cap_rights_t *big, const cap_rights_t *little); Capability rights to the cap_rights_init(), cap_rights_set(), cap_rights_clear() and cap_rights_is_set() functions are provided by separating them with commas, eg: cap_rights_t rights; cap_rights_init(&rights, CAP_READ, CAP_WRITE, CAP_FSTAT); There is no need to terminate the list of rights, as those functions are actually macros that take care of the termination, eg: #define cap_rights_set(rights, ...) \ __cap_rights_set((rights), __VA_ARGS__, 0ULL) void __cap_rights_set(cap_rights_t *rights, ...); Thanks to using one bit as an array index we can assert in those functions that there are no two rights belonging to different array elements provided together. For example this is illegal and will be detected, because CAP_LOOKUP belongs to element 0 and CAP_PDKILL to element 1: cap_rights_init(&rights, CAP_LOOKUP | CAP_PDKILL); Providing several rights that belongs to the same array's element this way is correct, but is not advised. It should only be used for aliases definition. This commit also breaks compatibility with some existing Capsicum system calls, but I see no other way to do that. This should be fine as Capsicum is still experimental and this change is not going to 9.x. Sponsored by: The FreeBSD Foundation
2013-09-05 00:09:56 +00:00
error = fget(td, fd, cap_rights_init(&rights, CAP_FSYNC), &fp);
Second-to-last commit implementing Capsicum capabilities in the FreeBSD kernel for FreeBSD 9.0: Add a new capability mask argument to fget(9) and friends, allowing system call code to declare what capabilities are required when an integer file descriptor is converted into an in-kernel struct file *. With options CAPABILITIES compiled into the kernel, this enforces capability protection; without, this change is effectively a no-op. Some cases require special handling, such as mmap(2), which must preserve information about the maximum rights at the time of mapping in the memory map so that they can later be enforced in mprotect(2) -- this is done by narrowing the rights in the existing max_protection field used for similar purposes with file permissions. In namei(9), we assert that the code is not reached from within capability mode, as we're not yet ready to enforce namespace capabilities there. This will follow in a later commit. Update two capability names: CAP_EVENT and CAP_KEVENT become CAP_POST_KEVENT and CAP_POLL_KEVENT to more accurately indicate what they represent. Approved by: re (bz) Submitted by: jonathan Sponsored by: Google Inc
2011-08-11 12:30:23 +00:00
break;
Add new system call - aio_mlock(). The name speaks for itself. It allows to perform the mlock(2) operation, which can consume a lot of time, under control of aio(4). Reviewed by: kib, jilles Sponsored by: Nginx, Inc.
2013-06-08 13:27:57 +00:00
case LIO_MLOCK:
fp = NULL;
break;
Second-to-last commit implementing Capsicum capabilities in the FreeBSD kernel for FreeBSD 9.0: Add a new capability mask argument to fget(9) and friends, allowing system call code to declare what capabilities are required when an integer file descriptor is converted into an in-kernel struct file *. With options CAPABILITIES compiled into the kernel, this enforces capability protection; without, this change is effectively a no-op. Some cases require special handling, such as mmap(2), which must preserve information about the maximum rights at the time of mapping in the memory map so that they can later be enforced in mprotect(2) -- this is done by narrowing the rights in the existing max_protection field used for similar purposes with file permissions. In namei(9), we assert that the code is not reached from within capability mode, as we're not yet ready to enforce namespace capabilities there. This will follow in a later commit. Update two capability names: CAP_EVENT and CAP_KEVENT become CAP_POST_KEVENT and CAP_POLL_KEVENT to more accurately indicate what they represent. Approved by: re (bz) Submitted by: jonathan Sponsored by: Google Inc
2011-08-11 12:30:23 +00:00
case LIO_NOP:
Change the cap_rights_t type from uint64_t to a structure that we can extend in the future in a backward compatible (API and ABI) way. The cap_rights_t represents capability rights. We used to use one bit to represent one right, but we are running out of spare bits. Currently the new structure provides place for 114 rights (so 50 more than the previous cap_rights_t), but it is possible to grow the structure to hold at least 285 rights, although we can make it even larger if 285 rights won't be enough. The structure definition looks like this: struct cap_rights { uint64_t cr_rights[CAP_RIGHTS_VERSION + 2]; }; The initial CAP_RIGHTS_VERSION is 0. The top two bits in the first element of the cr_rights[] array contain total number of elements in the array - 2. This means if those two bits are equal to 0, we have 2 array elements. The top two bits in all remaining array elements should be 0. The next five bits in all array elements contain array index. Only one bit is used and bit position in this five-bits range defines array index. This means there can be at most five array elements in the future. To define new right the CAPRIGHT() macro must be used. The macro takes two arguments - an array index and a bit to set, eg. #define CAP_PDKILL CAPRIGHT(1, 0x0000000000000800ULL) We still support aliases that combine few rights, but the rights have to belong to the same array element, eg: #define CAP_LOOKUP CAPRIGHT(0, 0x0000000000000400ULL) #define CAP_FCHMOD CAPRIGHT(0, 0x0000000000002000ULL) #define CAP_FCHMODAT (CAP_FCHMOD | CAP_LOOKUP) There is new API to manage the new cap_rights_t structure: cap_rights_t *cap_rights_init(cap_rights_t *rights, ...); void cap_rights_set(cap_rights_t *rights, ...); void cap_rights_clear(cap_rights_t *rights, ...); bool cap_rights_is_set(const cap_rights_t *rights, ...); bool cap_rights_is_valid(const cap_rights_t *rights); void cap_rights_merge(cap_rights_t *dst, const cap_rights_t *src); void cap_rights_remove(cap_rights_t *dst, const cap_rights_t *src); bool cap_rights_contains(const cap_rights_t *big, const cap_rights_t *little); Capability rights to the cap_rights_init(), cap_rights_set(), cap_rights_clear() and cap_rights_is_set() functions are provided by separating them with commas, eg: cap_rights_t rights; cap_rights_init(&rights, CAP_READ, CAP_WRITE, CAP_FSTAT); There is no need to terminate the list of rights, as those functions are actually macros that take care of the termination, eg: #define cap_rights_set(rights, ...) \ __cap_rights_set((rights), __VA_ARGS__, 0ULL) void __cap_rights_set(cap_rights_t *rights, ...); Thanks to using one bit as an array index we can assert in those functions that there are no two rights belonging to different array elements provided together. For example this is illegal and will be detected, because CAP_LOOKUP belongs to element 0 and CAP_PDKILL to element 1: cap_rights_init(&rights, CAP_LOOKUP | CAP_PDKILL); Providing several rights that belongs to the same array's element this way is correct, but is not advised. It should only be used for aliases definition. This commit also breaks compatibility with some existing Capsicum system calls, but I see no other way to do that. This should be fine as Capsicum is still experimental and this change is not going to 9.x. Sponsored by: The FreeBSD Foundation
2013-09-05 00:09:56 +00:00
error = fget(td, fd, cap_rights_init(&rights), &fp);
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
break;
default:
Second-to-last commit implementing Capsicum capabilities in the FreeBSD kernel for FreeBSD 9.0: Add a new capability mask argument to fget(9) and friends, allowing system call code to declare what capabilities are required when an integer file descriptor is converted into an in-kernel struct file *. With options CAPABILITIES compiled into the kernel, this enforces capability protection; without, this change is effectively a no-op. Some cases require special handling, such as mmap(2), which must preserve information about the maximum rights at the time of mapping in the memory map so that they can later be enforced in mprotect(2) -- this is done by narrowing the rights in the existing max_protection field used for similar purposes with file permissions. In namei(9), we assert that the code is not reached from within capability mode, as we're not yet ready to enforce namespace capabilities there. This will follow in a later commit. Update two capability names: CAP_EVENT and CAP_KEVENT become CAP_POST_KEVENT and CAP_POLL_KEVENT to more accurately indicate what they represent. Approved by: re (bz) Submitted by: jonathan Sponsored by: Google Inc
2011-08-11 12:30:23 +00:00
error = EINVAL;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
if (error) {
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
uma_zfree(aiocb_zone, aiocbe);
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
ops->store_error(job, error);
Return error from fget_write() rather than hardcoding EBADF now that fget_write() DTRT. Requested by: bde
2006-01-06 16:34:22 +00:00
return (error);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
if (opcode == LIO_SYNC && fp->f_vnode == NULL) {
error = EINVAL;
goto aqueue_fail;
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
if (opcode != LIO_SYNC && aiocbe->uaiocb.aio_offset == -1LL) {
Reduce the duplication of code for error handling in _aio_aqueue().
2002-04-07 07:17:59 +00:00
error = EINVAL;
goto aqueue_fail;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
aiocbe->fd_file = fp;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_lock(&aio_job_mtx);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
jid = jobrefid++;
aiocbe->seqno = jobseqno++;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_unlock(&aio_job_mtx);
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
error = ops->store_kernelinfo(job, jid);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (error) {
error = EINVAL;
goto aqueue_fail;
}
aiocbe->uaiocb._aiocb_private.kernelinfo = (void *)(intptr_t)jid;
clean up whitespace...
2004-08-13 17:43:53 +00:00
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
if (opcode == LIO_NOP) {
Keep the reference to the file acquired in _aio_aqueue() until the operation completes. The reference is released in aio_free_entry(). Submitted by: tegge
2002-03-31 20:17:56 +00:00
fdrop(fp, td);
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
uma_zfree(aiocb_zone, aiocbe);
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (0);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
hide kqueue_register from public view, and replace it w/ kqfd_register... this eliminates a possible race in aio registering a kevent..
2006-09-24 04:47:47 +00:00
if (aiocbe->uaiocb.aio_sigevent.sigev_notify != SIGEV_KEVENT)
Eliminate the original method of requesting notification of aio_read(2) and aio_write(2) completion through kevent(2). This method does not work on 64-bit architectures. It was deprecated in FreeBSD 4.4. See revisions 1.87 and 1.70.2.7. Change aio_physwakeup() to call psignal(9) directly rather than indirectly through a timeout(9). Discussed with: bde Correct a bug introduced in revision 1.65 that could result in premature delivery of a signal if an lio_listio(2) consisted of a mixture of direct/raw and queued I/O operations. Observed by: tegge Eliminate a field from struct kaioinfo that is now unused. Reviewed by: tegge
2005-06-04 19:16:33 +00:00
goto no_kqueue;
If multiple threads call kevent() to get AIO events on same kqueue fd, it is possible that a single AIO event will be reported to multiple threads, it is not threading friendly, and the existing API can not control this behavior. Allocate a kevent flags field sigev_notify_kevent_flags for AIO event notification in sigevent, and allow user to pass EV_CLEAR, EV_DISPATCH or EV_ONESHOT to AIO kernel code, user can control whether the event should be cleared once it is retrieved by a thread. This change should be comptaible with existing application, because the field should have already been zero-filled, and no additional action will be taken by kernel. PR: kern/156567
2012-02-01 02:53:06 +00:00
evflags = aiocbe->uaiocb.aio_sigevent.sigev_notify_kevent_flags;
if ((evflags & ~(EV_CLEAR | EV_DISPATCH | EV_ONESHOT)) != 0) {
error = EINVAL;
goto aqueue_fail;
}
hide kqueue_register from public view, and replace it w/ kqfd_register... this eliminates a possible race in aio registering a kevent..
2006-09-24 04:47:47 +00:00
kqfd = aiocbe->uaiocb.aio_sigevent.sigev_notify_kqueue;
Set the ident field of the struct kevent that is registered by _aio_aqueue() to the address of the user's aiocb rather than the kernel's aiocb. (In other words, prior to this change, the ident field returned by kevent(2) on completion of an AIO was effectively garbage.) Submitted by: Romer Gil <rgil@cs.rice.edu>
2002-08-06 19:01:08 +00:00
kev.ident = (uintptr_t)aiocbe->uuaiocb;
Provide a new interface for the user of aio_read() and aio_write() to request a kevent upon completion of the I/O. Specifically, introduce a new type of sigevent notification, SIGEV_EVENT. If sigev_notify is SIGEV_EVENT, then sigev_notify_kqueue names the kqueue that should receive the event and sigev_value contains the "void *" is copied into the kevent's udata field. In contrast to the existing interface, this one: 1) works on the Alpha 2) avoids the extra copyin() call for the kevent because all of the information needed is in the sigevent and 3) could be applied to request a single kevent upon completion of an entire lio_listio(). Reviewed by: jlemon
2000-11-21 19:36:36 +00:00
kev.filter = EVFILT_AIO;
If multiple threads call kevent() to get AIO events on same kqueue fd, it is possible that a single AIO event will be reported to multiple threads, it is not threading friendly, and the existing API can not control this behavior. Allocate a kevent flags field sigev_notify_kevent_flags for AIO event notification in sigevent, and allow user to pass EV_CLEAR, EV_DISPATCH or EV_ONESHOT to AIO kernel code, user can control whether the event should be cleared once it is retrieved by a thread. This change should be comptaible with existing application, because the field should have already been zero-filled, and no additional action will be taken by kernel. PR: kern/156567
2012-02-01 02:53:06 +00:00
kev.flags = EV_ADD | EV_ENABLE | EV_FLAG1 | evflags;
Set the ident field of the struct kevent that is registered by _aio_aqueue() to the address of the user's aiocb rather than the kernel's aiocb. (In other words, prior to this change, the ident field returned by kevent(2) on completion of an AIO was effectively garbage.) Submitted by: Romer Gil <rgil@cs.rice.edu>
2002-08-06 19:01:08 +00:00
kev.data = (intptr_t)aiocbe;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
kev.udata = aiocbe->uaiocb.aio_sigevent.sigev_value.sival_ptr;
hide kqueue_register from public view, and replace it w/ kqfd_register... this eliminates a possible race in aio registering a kevent..
2006-09-24 04:47:47 +00:00
error = kqfd_register(kqfd, &kev, td, 1);
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
aqueue_fail:
Provide a new interface for the user of aio_read() and aio_write() to request a kevent upon completion of the I/O. Specifically, introduce a new type of sigevent notification, SIGEV_EVENT. If sigev_notify is SIGEV_EVENT, then sigev_notify_kqueue names the kqueue that should receive the event and sigev_value contains the "void *" is copied into the kevent's udata field. In contrast to the existing interface, this one: 1) works on the Alpha 2) avoids the extra copyin() call for the kevent because all of the information needed is in the sigevent and 3) could be applied to request a single kevent upon completion of an entire lio_listio(). Reviewed by: jlemon
2000-11-21 19:36:36 +00:00
if (error) {
Add new system call - aio_mlock(). The name speaks for itself. It allows to perform the mlock(2) operation, which can consume a lot of time, under control of aio(4). Reviewed by: kib, jilles Sponsored by: Nginx, Inc.
2013-06-08 13:27:57 +00:00
if (fp)
fdrop(fp, td);
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
uma_zfree(aiocb_zone, aiocbe);
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
ops->store_error(job, error);
Provide a new interface for the user of aio_read() and aio_write() to request a kevent upon completion of the I/O. Specifically, introduce a new type of sigevent notification, SIGEV_EVENT. If sigev_notify is SIGEV_EVENT, then sigev_notify_kqueue names the kqueue that should receive the event and sigev_value contains the "void *" is copied into the kevent's udata field. In contrast to the existing interface, this one: 1) works on the Alpha 2) avoids the extra copyin() call for the kevent because all of the information needed is in the sigevent and 3) could be applied to request a single kevent upon completion of an entire lio_listio(). Reviewed by: jlemon
2000-11-21 19:36:36 +00:00
goto done;
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
}
Provide a new interface for the user of aio_read() and aio_write() to request a kevent upon completion of the I/O. Specifically, introduce a new type of sigevent notification, SIGEV_EVENT. If sigev_notify is SIGEV_EVENT, then sigev_notify_kqueue names the kqueue that should receive the event and sigev_value contains the "void *" is copied into the kevent's udata field. In contrast to the existing interface, this one: 1) works on the Alpha 2) avoids the extra copyin() call for the kevent because all of the information needed is in the sigevent and 3) could be applied to request a single kevent upon completion of an entire lio_listio(). Reviewed by: jlemon
2000-11-21 19:36:36 +00:00
no_kqueue:
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
ops->store_error(job, EINPROGRESS);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
aiocbe->uaiocb._aiocb_private.error = EINPROGRESS;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
aiocbe->userproc = p;
Do a bit more work in the aio code to simulate the credential environment of the original AIO request: save and restore the active thread credential as well as using the file credential, since MAC (and some other bits of the system) rely on the thread credential instead of/as well as the file credential. In brief: cache td->td_ucred when the AIO operation is queued, temporarily set and restore the kernel thread credential, and release the credential when done. Similar to ktrace credential management. Reviewed by: alc Approved by: re Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
2002-11-07 20:46:37 +00:00
aiocbe->cred = crhold(td->td_ucred);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
aiocbe->jobflags = 0;
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
aiocbe->lio = lj;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
if (opcode == LIO_SYNC)
goto queueit;
Add new system call - aio_mlock(). The name speaks for itself. It allows to perform the mlock(2) operation, which can consume a lot of time, under control of aio(4). Reviewed by: kib, jilles Sponsored by: Nginx, Inc.
2013-06-08 13:27:57 +00:00
if (fp && fp->f_type == DTYPE_SOCKET) {
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/*
* Alternate queueing for socket ops: Reach down into the
* descriptor to get the socket data. Then check to see if the
* socket is ready to be read or written (based on the requested
* operation).
*
* If it is not ready for io, then queue the aiocbe on the
* socket, and set the flags so we get a call when sbnotify()
* happens.
cover soreadable and sowriteable with the corresponding socketbuffer locks.
2004-10-01 05:54:06 +00:00
*
* Note if opcode is neither LIO_WRITE nor LIO_READ we lock
* and unlock the snd sockbuf for no reason.
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
*/
Bow to the whining masses and change a union back into void *. Retain removal of unnecessary casts and throw in some minor cleanups to see if anyone complains, just for the hell of it.
2003-01-13 00:33:17 +00:00
so = fp->f_data;
cover soreadable and sowriteable with the corresponding socketbuffer locks.
2004-10-01 05:54:06 +00:00
sb = (opcode == LIO_READ) ? &so->so_rcv : &so->so_snd;
SOCKBUF_LOCK(sb);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
if (((opcode == LIO_READ) && (!soreadable(so))) || ((opcode ==
LIO_WRITE) && (!sowriteable(so)))) {
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
sb->sb_flags |= SB_AIO;
mtx_lock(&aio_job_mtx);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
TAILQ_INSERT_TAIL(&so->so_aiojobq, aiocbe, list);
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
mtx_unlock(&aio_job_mtx);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TAILQ_INSERT_TAIL(&ki->kaio_all, aiocbe, allist);
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
TAILQ_INSERT_TAIL(&ki->kaio_jobqueue, aiocbe, plist);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
aiocbe->jobstate = JOBST_JOBQSOCK;
ki->kaio_count++;
if (lj)
lj->lioj_count++;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
cover soreadable and sowriteable with the corresponding socketbuffer locks.
2004-10-01 05:54:06 +00:00
SOCKBUF_UNLOCK(sb);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
atomic_add_int(&num_queue_count, 1);
This patchset fixes a large number of file descriptor race conditions. Pre-rfork code assumed inherent locking of a process's file descriptor array. However, with the advent of rfork() the file descriptor table could be shared between processes. This patch closes over a dozen serious race conditions related to one thread manipulating the table (e.g. closing or dup()ing a descriptor) while another is blocked in an open(), close(), fcntl(), read(), write(), etc... PR: kern/11629 Discussed with: Alexander Viro <viro@math.psu.edu>
2000-11-18 21:01:04 +00:00
error = 0;
goto done;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
}
cover soreadable and sowriteable with the corresponding socketbuffer locks.
2004-10-01 05:54:06 +00:00
SOCKBUF_UNLOCK(sb);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
}
if ((error = aio_qphysio(p, aiocbe)) == 0)
This patchset fixes a large number of file descriptor race conditions. Pre-rfork code assumed inherent locking of a process's file descriptor array. However, with the advent of rfork() the file descriptor table could be shared between processes. This patch closes over a dozen serious race conditions related to one thread manipulating the table (e.g. closing or dup()ing a descriptor) while another is blocked in an open(), close(), fcntl(), read(), write(), etc... PR: kern/11629 Discussed with: Alexander Viro <viro@math.psu.edu>
2000-11-18 21:01:04 +00:00
goto done;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
#if 0
This patchset fixes a large number of file descriptor race conditions. Pre-rfork code assumed inherent locking of a process's file descriptor array. However, with the advent of rfork() the file descriptor table could be shared between processes. This patch closes over a dozen serious race conditions related to one thread manipulating the table (e.g. closing or dup()ing a descriptor) while another is blocked in an open(), close(), fcntl(), read(), write(), etc... PR: kern/11629 Discussed with: Alexander Viro <viro@math.psu.edu>
2000-11-18 21:01:04 +00:00
if (error > 0) {
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
aiocbe->uaiocb._aiocb_private.error = error;
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
ops->store_error(job, error);
This patchset fixes a large number of file descriptor race conditions. Pre-rfork code assumed inherent locking of a process's file descriptor array. However, with the advent of rfork() the file descriptor table could be shared between processes. This patch closes over a dozen serious race conditions related to one thread manipulating the table (e.g. closing or dup()ing a descriptor) while another is blocked in an open(), close(), fcntl(), read(), write(), etc... PR: kern/11629 Discussed with: Alexander Viro <viro@math.psu.edu>
2000-11-18 21:01:04 +00:00
goto done;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
#endif
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
queueit:
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/* No buffer for daemon I/O. */
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
aiocbe->bp = NULL;
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
atomic_add_int(&num_queue_count, 1);
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
ki->kaio_count++;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
if (lj)
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
lj->lioj_count++;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
TAILQ_INSERT_TAIL(&ki->kaio_jobqueue, aiocbe, plist);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TAILQ_INSERT_TAIL(&ki->kaio_all, aiocbe, allist);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
if (opcode == LIO_SYNC) {
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
TAILQ_FOREACH(cb, &ki->kaio_jobqueue, plist) {
if (cb->fd_file == aiocbe->fd_file &&
cb->uaiocb.aio_lio_opcode != LIO_SYNC &&
cb->seqno < aiocbe->seqno) {
cb->jobflags |= AIOCBLIST_CHECKSYNC;
aiocbe->pending++;
}
}
TAILQ_FOREACH(cb, &ki->kaio_bufqueue, plist) {
if (cb->fd_file == aiocbe->fd_file &&
cb->uaiocb.aio_lio_opcode != LIO_SYNC &&
cb->seqno < aiocbe->seqno) {
cb->jobflags |= AIOCBLIST_CHECKSYNC;
aiocbe->pending++;
}
}
if (aiocbe->pending != 0) {
TAILQ_INSERT_TAIL(&ki->kaio_syncqueue, aiocbe, list);
aiocbe->jobstate = JOBST_JOBQSYNC;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
goto done;
}
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
}
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
mtx_lock(&aio_job_mtx);
TAILQ_INSERT_TAIL(&aio_jobs, aiocbe, list);
aiocbe->jobstate = JOBST_JOBQGLOBAL;
aio_kick_nowait(p);
mtx_unlock(&aio_job_mtx);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
error = 0;
done:
return (error);
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
static void
aio_kick_nowait(struct proc *userp)
{
struct kaioinfo *ki = userp->p_aioinfo;
struct aiothreadlist *aiop;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
mtx_assert(&aio_job_mtx, MA_OWNED);
if ((aiop = TAILQ_FIRST(&aio_freeproc)) != NULL) {
TAILQ_REMOVE(&aio_freeproc, aiop, list);
aiop->aiothreadflags &= ~AIOP_FREE;
wakeup(aiop->aiothread);
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
} else if (((num_aio_resv_start + num_aio_procs) < max_aio_procs) &&
((ki->kaio_active_count + num_aio_resv_start) <
ki->kaio_maxactive_count)) {
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
taskqueue_enqueue(taskqueue_aiod_bio, &ki->kaio_task);
}
}
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
static int
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
aio_kick(struct proc *userp)
{
struct kaioinfo *ki = userp->p_aioinfo;
struct aiothreadlist *aiop;
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
int error, ret = 0;
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
mtx_assert(&aio_job_mtx, MA_OWNED);
o Correct the initialization of aiolio_zone: Each entry was 16 times larger than necessary. o Move a rarely-used goto label inside a critical section so that we don't perform an splnet() for which there is no corresponding splx(). o Remove unnecessary splnet()/splx() around accesses to kaioinfo::kaio_jobdone in aio_return(). o Use TAILQ_FOREACH for simple cases of iteration over kaioinfo::kaio_jobdone.
2002-01-14 07:26:33 +00:00
retryproc:
Fix warnings in preparation for adding -Wall -Wcast-qual to the kernel compile
1999-01-27 21:50:00 +00:00
if ((aiop = TAILQ_FIRST(&aio_freeproc)) != NULL) {
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
TAILQ_REMOVE(&aio_freeproc, aiop, list);
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
aiop->aiothreadflags &= ~AIOP_FREE;
wakeup(aiop->aiothread);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
} else if (((num_aio_resv_start + num_aio_procs) < max_aio_procs) &&
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
((ki->kaio_active_count + num_aio_resv_start) <
ki->kaio_maxactive_count)) {
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
num_aio_resv_start++;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_unlock(&aio_job_mtx);
error = aio_newproc(&num_aio_resv_start);
mtx_lock(&aio_job_mtx);
if (error) {
num_aio_resv_start--;
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
goto retryproc;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
}
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
} else {
ret = -1;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
return (ret);
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
static void
aio_kick_helper(void *context, int pending)
{
struct proc *userp = context;
mtx_lock(&aio_job_mtx);
1. Move code for scanning pending I/O from aio_fsync to aio_aqueue, it has less overhead. 2. Avoid scheduling task if maximum number of I/O threads is reached.
2006-03-24 00:50:06 +00:00
while (--pending >= 0) {
if (aio_kick(userp))
break;
}
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
mtx_unlock(&aio_job_mtx);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Support the aio_return system call, as a side-effect, kernel resources are
* released.
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
*/
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
static int
kern_aio_return(struct thread *td, struct aiocb *uaiocb, struct aiocb_ops *ops)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct proc *p = td->td_proc;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
struct aiocblist *cb;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct kaioinfo *ki;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
int status, error;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Restructure aio_return() to eliminate duplicated code and facilitate Giant push down.
2002-04-08 04:57:56 +00:00
ki = p->p_aioinfo;
if (ki == NULL)
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (EINVAL);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TAILQ_FOREACH(cb, &ki->kaio_done, plist) {
if (cb->uuaiocb == uaiocb)
Restructure aio_return() to eliminate duplicated code and facilitate Giant push down.
2002-04-08 04:57:56 +00:00
break;
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
}
Restructure aio_return() to eliminate duplicated code and facilitate Giant push down.
2002-04-08 04:57:56 +00:00
if (cb != NULL) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
MPASS(cb->jobstate == JOBST_JOBFINISHED);
status = cb->uaiocb._aiocb_private.status;
error = cb->uaiocb._aiocb_private.error;
td->td_retval[0] = status;
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
if (cb->uaiocb.aio_lio_opcode == LIO_WRITE) {
- Move rusage from being per-process in struct pstats to per-thread in td_ru. This removes the requirement for per-process synchronization in statclock() and mi_switch(). This was previously supported by sched_lock which is going away. All modifications to rusage are now done in the context of the owning thread. reads proceed without locks. - Aggregate exiting threads rusage in thread_exit() such that the exiting thread's rusage is not lost. - Provide a new routine, rufetch() to fetch an aggregate of all rusage structures from all threads in a process. This routine must be used in any place requiring a rusage from a process prior to it's exit. The exited process's rusage is still available via p_ru. - Aggregate tick statistics only on demand via rufetch() or when a thread exits. Tick statistics are kept in the thread and protected by sched_lock until it exits. Initial patch by: attilio Reviewed by: attilio, bde (some objections), arch (mostly silent)
2007-06-01 01:12:45 +00:00
td->td_ru.ru_oublock += cb->outputcharge;
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
cb->outputcharge = 0;
} else if (cb->uaiocb.aio_lio_opcode == LIO_READ) {
- Move rusage from being per-process in struct pstats to per-thread in td_ru. This removes the requirement for per-process synchronization in statclock() and mi_switch(). This was previously supported by sched_lock which is going away. All modifications to rusage are now done in the context of the owning thread. reads proceed without locks. - Aggregate exiting threads rusage in thread_exit() such that the exiting thread's rusage is not lost. - Provide a new routine, rufetch() to fetch an aggregate of all rusage structures from all threads in a process. This routine must be used in any place requiring a rusage from a process prior to it's exit. The exited process's rusage is still available via p_ru. - Aggregate tick statistics only on demand via rufetch() or when a thread exits. Tick statistics are kept in the thread and protected by sched_lock until it exits. Initial patch by: attilio Reviewed by: attilio, bde (some objections), arch (mostly silent)
2007-06-01 01:12:45 +00:00
td->td_ru.ru_inblock += cb->inputcharge;
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
cb->inputcharge = 0;
}
Restructure aio_return() to eliminate duplicated code and facilitate Giant push down.
2002-04-08 04:57:56 +00:00
aio_free_entry(cb);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
ops->store_error(uaiocb, error);
ops->store_status(uaiocb, status);
in aio_aqueue, store same return code into job->_aiocb_private.error. in aio_return, unlock proc lock before suword.
2006-01-26 08:37:02 +00:00
} else {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
error = EINVAL;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
in aio_aqueue, store same return code into job->_aiocb_private.error. in aio_return, unlock proc lock before suword.
2006-01-26 08:37:02 +00:00
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
return (error);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
int
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_aio_return(struct thread *td, struct aio_return_args *uap)
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
{
return (kern_aio_return(td, uap->aiocbp, &aiocb_ops));
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Allow a process to wakeup when any of the I/O requests are completed.
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
*/
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
static int
kern_aio_suspend(struct thread *td, int njoblist, struct aiocb **ujoblist,
struct timespec *ts)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct proc *p = td->td_proc;
Remove a bunch of variables which were unused both in GENERIC and LINT. Found by: -Wunused
1997-11-07 08:53:44 +00:00
struct timeval atv;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct kaioinfo *ki;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
struct aiocblist *cb, *cbfirst;
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
int error, i, timo;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
timo = 0;
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
if (ts) {
if (ts->tv_nsec < 0 || ts->tv_nsec >= 1000000000)
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
return (EINVAL);
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
TIMESPEC_TO_TIMEVAL(&atv, ts);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
if (itimerfix(&atv))
return (EINVAL);
Eradicate the variable "time" from the kernel, using various measures. "time" wasn't a atomic variable, so splfoo() protection were needed around any access to it, unless you just wanted the seconds part. Most uses of time.tv_sec now uses the new variable time_second instead. gettime() changed to getmicrotime(0. Remove a couple of unneeded splfoo() protections, the new getmicrotime() is atomic, (until Bruce sets a breakpoint in it). A couple of places needed random data, so use read_random() instead of mucking about with time which isn't random. Add a new nfs_curusec() function. Mark a couple of bogosities involving the now disappeard time variable. Update ffs_update() to avoid the weird "== &time" checks, by fixing the one remaining call that passwd &time as args. Change profiling in ncr.c to use ticks instead of time. Resolution is the same. Add new function "tvtohz()" to avoid the bogus "splfoo(), add time, call hzto() which subtracts time" sequences. Reviewed by: bde
1998-03-30 09:56:58 +00:00
timo = tvtohz(&atv);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
ki = p->p_aioinfo;
if (ki == NULL)
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (EAGAIN);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
if (njoblist == 0)
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (0);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
for (;;) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
cbfirst = NULL;
error = 0;
TAILQ_FOREACH(cb, &ki->kaio_all, allist) {
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
for (i = 0; i < njoblist; i++) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (cb->uuaiocb == ujoblist[i]) {
if (cbfirst == NULL)
cbfirst = cb;
if (cb->jobstate == JOBST_JOBFINISHED)
goto RETURN;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
}
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
/* All tasks were finished. */
if (cbfirst == NULL)
break;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
ki->kaio_flags |= KAIO_WAKEUP;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
error = msleep(&p->p_aioinfo, AIO_MTX(ki), PRIBIO | PCATCH,
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
"aiospn", timo);
if (error == ERESTART)
error = EINTR;
if (error)
break;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
RETURN:
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
return (error);
}
int
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_aio_suspend(struct thread *td, struct aio_suspend_args *uap)
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
{
struct timespec ts, *tsp;
struct aiocb **ujoblist;
int error;
if (uap->nent < 0 || uap->nent > AIO_LISTIO_MAX)
return (EINVAL);
if (uap->timeout) {
/* Get timespec struct. */
if ((error = copyin(uap->timeout, &ts, sizeof(ts))) != 0)
return (error);
tsp = &ts;
} else
tsp = NULL;
ujoblist = uma_zalloc(aiol_zone, M_WAITOK);
error = copyin(uap->aiocbp, ujoblist, uap->nent * sizeof(ujoblist[0]));
if (error == 0)
error = kern_aio_suspend(td, uap->nent, ujoblist, tsp);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
uma_zfree(aiol_zone, ujoblist);
return (error);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
/*
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
* aio_cancel cancels any non-physio aio operations not currently in
* progress.
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
*/
int
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_aio_cancel(struct thread *td, struct aio_cancel_args *uap)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct proc *p = td->td_proc;
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
struct kaioinfo *ki;
struct aiocblist *cbe, *cbn;
struct file *fp;
struct socket *so;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
int error;
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
int remove;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
int cancelled = 0;
int notcancelled = 0;
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
struct vnode *vp;
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
/* Lookup file object. */
The fget() function now takes pointer to cap_rights_t, so change 0 to NULL.
2013-09-05 11:59:23 +00:00
error = fget(td, uap->fd, NULL, &fp);
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
if (error)
return (error);
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
ki = p->p_aioinfo;
if (ki == NULL)
goto done;
clean up whitespace...
2004-08-13 17:43:53 +00:00
if (fp->f_type == DTYPE_VNODE) {
Add a f_vnode field to struct file. Several of the subtypes have an associated vnode which is used for stuff like the f*() functions. By giving the vnode a speparate field, a number of checks for the specific subtype can be replaced simply with a check for f_vnode != NULL, and we can later free f_data up to subtype specific use. At this point in time, f_data still points to the vnode, so any code I might have overlooked will still work.
2003-06-22 08:41:43 +00:00
vp = fp->f_vnode;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (vn_isdisk(vp, &error)) {
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
fdrop(fp, td);
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
td->td_retval[0] = AIO_NOTCANCELED;
clean up whitespace...
2004-08-13 17:43:53 +00:00
return (0);
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
}
}
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
TAILQ_FOREACH_SAFE(cbe, &ki->kaio_jobqueue, plist, cbn) {
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
if ((uap->fd == cbe->uaiocb.aio_fildes) &&
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
((uap->aiocbp == NULL) ||
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
(uap->aiocbp == cbe->uuaiocb))) {
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
remove = 0;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
mtx_lock(&aio_job_mtx);
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
if (cbe->jobstate == JOBST_JOBQGLOBAL) {
TAILQ_REMOVE(&aio_jobs, cbe, list);
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
remove = 1;
} else if (cbe->jobstate == JOBST_JOBQSOCK) {
MPASS(fp->f_type == DTYPE_SOCKET);
so = fp->f_data;
TAILQ_REMOVE(&so->so_aiojobq, cbe, list);
remove = 1;
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
} else if (cbe->jobstate == JOBST_JOBQSYNC) {
TAILQ_REMOVE(&ki->kaio_syncqueue, cbe, list);
remove = 1;
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
}
mtx_unlock(&aio_job_mtx);
if (remove) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TAILQ_REMOVE(&ki->kaio_jobqueue, cbe, plist);
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
cbe->uaiocb._aiocb_private.status = -1;
cbe->uaiocb._aiocb_private.error = ECANCELED;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
aio_bio_done_notify(p, cbe, DONE_QUEUE);
cancelled++;
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
} else {
notcancelled++;
}
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
if (uap->aiocbp != NULL)
break;
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
}
}
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
o Make a correction to the last change: In aio_cancel(2) return AIO_ALLDONE instead of EINVAL if p->p_aioinfo is NULL.
2002-08-11 19:04:17 +00:00
done:
Various and sundry cleanups: - Use curthread for calls to knlist_delete() and add a big comment explaining why as well as appropriate assertions. - Use TAILQ_FOREACH and TAILQ_FOREACH_SAFE instead of handrolling them. - Use fget() family of functions to lookup file objects instead of grovelling around in file descriptor tables. - Destroy the aio_freeproc mutex if we are unloaded. Tested on: i386
2005-11-08 17:43:05 +00:00
fdrop(fp, td);
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
if (uap->aiocbp != NULL) {
if (cancelled) {
td->td_retval[0] = AIO_CANCELED;
return (0);
}
}
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
if (notcancelled) {
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
td->td_retval[0] = AIO_NOTCANCELED;
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (0);
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
}
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
if (cancelled) {
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
td->td_retval[0] = AIO_CANCELED;
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (0);
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
}
Add locking annotation and comments about socket, pipe, fifo problem. Temporarily fix a locking problem for socket I/O.
2006-01-24 07:24:24 +00:00
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
td->td_retval[0] = AIO_ALLDONE;
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (0);
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
}
/*
Further system call comment cleanup: - Remove also "MP SAFE" after prior "MPSAFE" pass. (suggested by bde) - Remove extra blank lines in some cases. - Add extra blank lines in some cases. - Remove no-op comments consisting solely of the function name, the word "syscall", or the system call name. - Add punctuation. - Re-wrap some comments.
2007-03-05 13:10:58 +00:00
* aio_error is implemented in the kernel level for compatibility purposes
* only. For a user mode async implementation, it would be best to do it in
* a userland subroutine.
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
*/
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
static int
kern_aio_error(struct thread *td, struct aiocb *aiocbp, struct aiocb_ops *ops)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct proc *p = td->td_proc;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct aiocblist *cb;
struct kaioinfo *ki;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
int status;
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
ki = p->p_aioinfo;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (ki == NULL) {
td->td_retval[0] = EINVAL;
return (0);
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
}
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
TAILQ_FOREACH(cb, &ki->kaio_all, allist) {
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
if (cb->uuaiocb == aiocbp) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (cb->jobstate == JOBST_JOBFINISHED)
td->td_retval[0] =
cb->uaiocb._aiocb_private.error;
else
td->td_retval[0] = EINPROGRESS;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (0);
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
}
}
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
/*
1) Merge _aio_aqueue and aio_aqueue, check quota in aio_aqueue, so that lio_listio won't exceed the quota. 2) Remove lio_ref_count, it is no longer used.
2006-01-23 02:49:34 +00:00
* Hack for failure of aio_aqueue.
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
*/
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
status = ops->fetch_status(aiocbp);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (status == -1) {
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
td->td_retval[0] = ops->fetch_error(aiocbp);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
return (0);
}
td->td_retval[0] = EINVAL;
return (0);
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
}
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
int
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_aio_error(struct thread *td, struct aio_error_args *uap)
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
{
return (kern_aio_error(td, uap->aiocbp, &aiocb_ops));
}
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/* syscall - asynchronous read from a file (REALTIME) */
Fix sigevent's POSIX incompatible problem by adding member fields sigev_notify_function and sigev_notify_attributes. AIO syscalls use sigevent, so they have to be adjusted. Reviewed by: alc
2005-10-30 02:12:49 +00:00
int
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_oaio_read(struct thread *td, struct oaio_read_args *uap)
Fix sigevent's POSIX incompatible problem by adding member fields sigev_notify_function and sigev_notify_attributes. AIO syscalls use sigevent, so they have to be adjusted. Reviewed by: alc
2005-10-30 02:12:49 +00:00
{
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
return (aio_aqueue(td, (struct aiocb *)uap->aiocbp, NULL, LIO_READ,
&aiocb_ops_osigevent));
Fix sigevent's POSIX incompatible problem by adding member fields sigev_notify_function and sigev_notify_attributes. AIO syscalls use sigevent, so they have to be adjusted. Reviewed by: alc
2005-10-30 02:12:49 +00:00
}
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
int
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_aio_read(struct thread *td, struct aio_read_args *uap)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
return (aio_aqueue(td, uap->aiocbp, NULL, LIO_READ, &aiocb_ops));
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
}
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/* syscall - asynchronous write to a file (REALTIME) */
Fix sigevent's POSIX incompatible problem by adding member fields sigev_notify_function and sigev_notify_attributes. AIO syscalls use sigevent, so they have to be adjusted. Reviewed by: alc
2005-10-30 02:12:49 +00:00
int
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_oaio_write(struct thread *td, struct oaio_write_args *uap)
Fix sigevent's POSIX incompatible problem by adding member fields sigev_notify_function and sigev_notify_attributes. AIO syscalls use sigevent, so they have to be adjusted. Reviewed by: alc
2005-10-30 02:12:49 +00:00
{
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
return (aio_aqueue(td, (struct aiocb *)uap->aiocbp, NULL, LIO_WRITE,
&aiocb_ops_osigevent));
Fix sigevent's POSIX incompatible problem by adding member fields sigev_notify_function and sigev_notify_attributes. AIO syscalls use sigevent, so they have to be adjusted. Reviewed by: alc
2005-10-30 02:12:49 +00:00
}
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
int
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_aio_write(struct thread *td, struct aio_write_args *uap)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
Make AIO a loadable module. Remove the explicit call to aio_proc_rundown() from exit1(), instead AIO will use at_exit(9). Add functions at_exec(9), rm_at_exec(9) which function nearly the same as at_exec(9) and rm_at_exec(9), these functions are called on behalf of modules at the time of execve(2) after the image activator has run. Use a modified version of tegge's suggestion via at_exec(9) to close an exploitable race in AIO. Fix SYSCALL_MODULE_HELPER such that it's archetecuterally neutral, the problem was that one had to pass it a paramater indicating the number of arguments which were actually the number of "int". Fix it by using an inline version of the AS macro against the syscall arguments. (AS should be available globally but we'll get to that later.) Add a primative system for dynamically adding kqueue ops, it's really not as sophisticated as it should be, but I'll discuss with jlemon when he's around.
2001-12-29 07:13:47 +00:00
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
return (aio_aqueue(td, uap->aiocbp, NULL, LIO_WRITE, &aiocb_ops));
Fix sigevent's POSIX incompatible problem by adding member fields sigev_notify_function and sigev_notify_attributes. AIO syscalls use sigevent, so they have to be adjusted. Reviewed by: alc
2005-10-30 02:12:49 +00:00
}
Add new system call - aio_mlock(). The name speaks for itself. It allows to perform the mlock(2) operation, which can consume a lot of time, under control of aio(4). Reviewed by: kib, jilles Sponsored by: Nginx, Inc.
2013-06-08 13:27:57 +00:00
int
sys_aio_mlock(struct thread *td, struct aio_mlock_args *uap)
{
return (aio_aqueue(td, uap->aiocbp, NULL, LIO_MLOCK, &aiocb_ops));
}
Fix sigevent's POSIX incompatible problem by adding member fields sigev_notify_function and sigev_notify_attributes. AIO syscalls use sigevent, so they have to be adjusted. Reviewed by: alc
2005-10-30 02:12:49 +00:00
static int
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
kern_lio_listio(struct thread *td, int mode, struct aiocb * const *uacb_list,
struct aiocb **acb_list, int nent, struct sigevent *sig,
struct aiocb_ops *ops)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct proc *p = td->td_proc;
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
struct aiocb *iocb;
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
struct kaioinfo *ki;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
struct aioliojob *lj;
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
struct kevent kev;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
int error;
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
int nerror;
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
int i;
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
if ((mode != LIO_NOWAIT) && (mode != LIO_WAIT))
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (EINVAL);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Allowing nent < 0 in aio_suspend() and lio_listio() is just asking for trouble. Return EINVAL instead.
2003-01-12 09:40:23 +00:00
if (nent < 0 || nent > AIO_LISTIO_MAX)
style(9) fixes, mostly add parens around return arguments.
2003-01-13 15:06:05 +00:00
return (EINVAL);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
if (p->p_aioinfo == NULL)
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
aio_init_aioinfo(p);
ki = p->p_aioinfo;
Back out M_* changes, per decision of the TRB. Approved by: trb
2003-02-19 05:47:46 +00:00
lj = uma_zalloc(aiolio_zone, M_WAITOK);
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
lj->lioj_flags = 0;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
lj->lioj_count = 0;
lj->lioj_finished_count = 0;
Adapt vfs kqfilter to the shared vnode lock used by zfs write vop. Use vnode interlock to protect the knote fields [1]. The locking assumes that shared vnode lock is held, thus we get exclusive access to knote either by exclusive vnode lock protection, or by shared vnode lock + vnode interlock. Do not use kl_locked() method to assert either lock ownership or the fact that curthread does not own the lock. For shared locks, ownership is not recorded, e.g. VOP_ISLOCKED can return LK_SHARED for the shared lock not owned by curthread, causing false positives in kqueue subsystem assertions about knlist lock. Remove kl_locked method from knlist lock vector, and add two separate assertion methods kl_assert_locked and kl_assert_unlocked, that are supposed to use proper asserts. Change knlist_init accordingly. Add convenience function knlist_init_mtx to reduce number of arguments for typical knlist initialization. Submitted by: jhb [1] Noted by: jhb [2] Reviewed by: jhb Tested by: rnoland
2009-06-10 20:59:32 +00:00
knlist_init_mtx(&lj->klist, AIO_MTX(ki));
Support sending realtime signal information via signal queue, realtime signal memory is pre-allocated, so kernel can always notify user code.
2005-11-03 05:25:26 +00:00
ksiginfo_init(&lj->lioj_ksi);
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
/*
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
* Setup signal.
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
*/
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
if (sig && (mode == LIO_NOWAIT)) {
bcopy(sig, &lj->lioj_signal, sizeof(lj->lioj_signal));
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
if (lj->lioj_signal.sigev_notify == SIGEV_KEVENT) {
/* Assume only new style KEVENT */
kev.filter = EVFILT_LIO;
kev.flags = EV_ADD | EV_ENABLE | EV_FLAG1;
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
kev.ident = (uintptr_t)uacb_list; /* something unique */
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
kev.data = (intptr_t)lj;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
/* pass user defined sigval data */
kev.udata = lj->lioj_signal.sigev_value.sival_ptr;
hide kqueue_register from public view, and replace it w/ kqfd_register... this eliminates a possible race in aio registering a kevent..
2006-09-24 04:47:47 +00:00
error = kqfd_register(
lj->lioj_signal.sigev_notify_kqueue, &kev, td, 1);
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
if (error) {
uma_zfree(aiolio_zone, lj);
return (error);
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
} else if (lj->lioj_signal.sigev_notify == SIGEV_NONE) {
;
Verify all supported notification types.
2006-01-23 10:27:15 +00:00
} else if (lj->lioj_signal.sigev_notify == SIGEV_SIGNAL ||
lj->lioj_signal.sigev_notify == SIGEV_THREAD_ID) {
if (!_SIG_VALID(lj->lioj_signal.sigev_signo)) {
uma_zfree(aiolio_zone, lj);
return EINVAL;
}
lj->lioj_flags |= LIOJ_SIGNAL;
} else {
Remove references to vm_zone.h and switch over to the new uma API. Also, remove maxsockets. If you look carefully you'll notice that the old zone allocator never honored this anyway.
2002-03-20 04:09:59 +00:00
uma_zfree(aiolio_zone, lj);
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
return EINVAL;
Check validity of signal callback requested via aio routines. Also move the insertion of the request to after the request is validated, there's still looks like there may be some problems if an invalid address is passed to the aio routines, basically a possible leak or having a not completely initialized structure on the queue may still be possible. A new sig macro was made _SIG_VALID to check the validity of a signal, it would be advisable to use it from now on (in kern/kern_sig.c) rather than rolling your own. PR: kern/17152
2001-04-18 22:18:39 +00:00
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
}
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Check validity of signal callback requested via aio routines. Also move the insertion of the request to after the request is validated, there's still looks like there may be some problems if an invalid address is passed to the aio routines, basically a possible leak or having a not completely initialized structure on the queue may still be possible. A new sig macro was made _SIG_VALID to check the validity of a signal, it would be advisable to use it from now on (in kern/kern_sig.c) rather than rolling your own. PR: kern/17152
2001-04-18 22:18:39 +00:00
TAILQ_INSERT_TAIL(&ki->kaio_liojoblist, lj, lioj_list);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
/*
* Add extra aiocb count to avoid the lio to be freed
* by other threads doing aio_waitcomplete or aio_return,
* and prevent event from being sent until we have queued
* all tasks.
*/
lj->lioj_count = 1;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
/*
* Get pointers to the list of I/O requests.
*/
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
nerror = 0;
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
for (i = 0; i < nent; i++) {
iocb = acb_list[i];
if (iocb != NULL) {
error = aio_aqueue(td, iocb, lj, LIO_NOP, ops);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (error != 0)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
nerror++;
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
error = 0;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
if (mode == LIO_WAIT) {
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
while (lj->lioj_count - 1 != lj->lioj_finished_count) {
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
ki->kaio_flags |= KAIO_WAKEUP;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
error = msleep(&p->p_aioinfo, AIO_MTX(ki),
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
PRIBIO | PCATCH, "aiospn", 0);
if (error == ERESTART)
error = EINTR;
if (error)
break;
}
} else {
if (lj->lioj_count - 1 == lj->lioj_finished_count) {
if (lj->lioj_signal.sigev_notify == SIGEV_KEVENT) {
lj->lioj_flags |= LIOJ_KEVENT_POSTED;
KNOTE_LOCKED(&lj->klist, 1);
}
if ((lj->lioj_flags & (LIOJ_SIGNAL|LIOJ_SIGNAL_POSTED))
== LIOJ_SIGNAL
&& (lj->lioj_signal.sigev_notify == SIGEV_SIGNAL ||
lj->lioj_signal.sigev_notify == SIGEV_THREAD_ID)) {
aio_sendsig(p, &lj->lioj_signal,
&lj->lioj_ksi);
lj->lioj_flags |= LIOJ_SIGNAL_POSTED;
}
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
}
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
lj->lioj_count--;
if (lj->lioj_count == 0) {
TAILQ_REMOVE(&ki->kaio_liojoblist, lj, lioj_list);
knlist_delete(&lj->klist, curthread, 1);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
PROC_LOCK(p);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
sigqueue_take(&lj->lioj_ksi);
PROC_UNLOCK(p);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
uma_zfree(aiolio_zone, lj);
} else
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
This is an upgrade so that the kernel supports the AIO calls from POSIX.4. Additionally, there is some initial code that supports LIO. This code supports AIO/LIO for all types of file descriptors, with few if any restrictions. There will be a followup very soon that will support significantly more efficient operation for VCHR type files (raw.) This code is also dependent on some kernel features that don't work under SMP yet. After I commit the changes to the kernel to support proper address space sharing on SMP, this code will also work under SMP.
1997-07-06 02:40:43 +00:00
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (nerror)
return (EIO);
return (error);
Add initial AIO/LIO kernel thread support files. This is preliminary, and further features will be added.
1997-06-16 00:27:26 +00:00
}
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
/* syscall - list directed I/O (REALTIME) */
int
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_olio_listio(struct thread *td, struct olio_listio_args *uap)
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
{
struct aiocb **acb_list;
struct sigevent *sigp, sig;
struct osigevent osig;
int error, nent;
if ((uap->mode != LIO_NOWAIT) && (uap->mode != LIO_WAIT))
return (EINVAL);
nent = uap->nent;
if (nent < 0 || nent > AIO_LISTIO_MAX)
return (EINVAL);
if (uap->sig && (uap->mode == LIO_NOWAIT)) {
error = copyin(uap->sig, &osig, sizeof(osig));
if (error)
return (error);
error = convert_old_sigevent(&osig, &sig);
if (error)
return (error);
sigp = &sig;
} else
sigp = NULL;
acb_list = malloc(sizeof(struct aiocb *) * nent, M_LIO, M_WAITOK);
error = copyin(uap->acb_list, acb_list, nent * sizeof(acb_list[0]));
if (error == 0)
error = kern_lio_listio(td, uap->mode,
(struct aiocb * const *)uap->acb_list, acb_list, nent, sigp,
&aiocb_ops_osigevent);
free(acb_list, M_LIO);
return (error);
}
/* syscall - list directed I/O (REALTIME) */
int
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_lio_listio(struct thread *td, struct lio_listio_args *uap)
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
{
struct aiocb **acb_list;
struct sigevent *sigp, sig;
int error, nent;
if ((uap->mode != LIO_NOWAIT) && (uap->mode != LIO_WAIT))
return (EINVAL);
nent = uap->nent;
if (nent < 0 || nent > AIO_LISTIO_MAX)
return (EINVAL);
if (uap->sig && (uap->mode == LIO_NOWAIT)) {
error = copyin(uap->sig, &sig, sizeof(sig));
if (error)
return (error);
sigp = &sig;
} else
sigp = NULL;
acb_list = malloc(sizeof(struct aiocb *) * nent, M_LIO, M_WAITOK);
error = copyin(uap->acb_list, acb_list, nent * sizeof(acb_list[0]));
if (error == 0)
error = kern_lio_listio(td, uap->mode, uap->acb_list, acb_list,
nent, sigp, &aiocb_ops);
free(acb_list, M_LIO);
return (error);
}
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
/*
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
* Called from interrupt thread for physio, we should return as fast
* as possible, so we schedule a biohelper task.
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
*/
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
static void
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
aio_physwakeup(struct buf *bp)
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
{
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
struct aiocblist *aiocbe;
Fix asynchronous physio breakage introduced in rev 1.163. We cannnot use bp->b_caller2 because DEV_STRATEGY will overwrite it.
2003-09-10 15:48:51 +00:00
aiocbe = (struct aiocblist *)bp->b_caller1;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
taskqueue_enqueue(taskqueue_aiod_bio, &aiocbe->biotask);
}
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
/*
* Task routine to perform heavy tasks, process wakeup, and signals.
*/
static void
biohelper(void *context, int pending)
{
struct aiocblist *aiocbe = context;
struct buf *bp;
struct proc *userp;
1. Remove aio entry from lists earlier in aio_free_entry, so other threads can not see it if we unlock the proc lock (this can happen in knlist_delete). Don't do wakeup, it is not necessary. 2. Decrease kaio_buffer_count in biohelper rather than doing it in aio_bio_done_notify. 3. In aio_bio_done_notify, don't send notification if KAIO_RUNDOWN was set, because the process is already in single thread mode. 4. Use assignment to initialize aiothreadflags. 5. AIOCBLIST_RUNDOWN is not useful, axe the code using it. 6. use LIO_NOP instead of zero.
2006-02-26 12:56:23 +00:00
struct kaioinfo *ki;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
int nblks;
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
bp = aiocbe->bp;
userp = aiocbe->userproc;
1. Remove aio entry from lists earlier in aio_free_entry, so other threads can not see it if we unlock the proc lock (this can happen in knlist_delete). Don't do wakeup, it is not necessary. 2. Decrease kaio_buffer_count in biohelper rather than doing it in aio_bio_done_notify. 3. In aio_bio_done_notify, don't send notification if KAIO_RUNDOWN was set, because the process is already in single thread mode. 4. Use assignment to initialize aiothreadflags. 5. AIOCBLIST_RUNDOWN is not useful, axe the code using it. 6. use LIO_NOP instead of zero.
2006-02-26 12:56:23 +00:00
ki = userp->p_aioinfo;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
aiocbe->uaiocb._aiocb_private.status -= bp->b_resid;
aiocbe->uaiocb._aiocb_private.error = 0;
if (bp->b_ioflags & BIO_ERROR)
aiocbe->uaiocb._aiocb_private.error = bp->b_error;
nblks = btodb(aiocbe->uaiocb.aio_nbytes);
if (aiocbe->uaiocb.aio_lio_opcode == LIO_WRITE)
aiocbe->outputcharge += nblks;
else
aiocbe->inputcharge += nblks;
aiocbe->bp = NULL;
TAILQ_REMOVE(&userp->p_aioinfo->kaio_bufqueue, aiocbe, plist);
1. Remove aio entry from lists earlier in aio_free_entry, so other threads can not see it if we unlock the proc lock (this can happen in knlist_delete). Don't do wakeup, it is not necessary. 2. Decrease kaio_buffer_count in biohelper rather than doing it in aio_bio_done_notify. 3. In aio_bio_done_notify, don't send notification if KAIO_RUNDOWN was set, because the process is already in single thread mode. 4. Use assignment to initialize aiothreadflags. 5. AIOCBLIST_RUNDOWN is not useful, axe the code using it. 6. use LIO_NOP instead of zero.
2006-02-26 12:56:23 +00:00
ki->kaio_buffer_count--;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
aio_bio_done_notify(userp, aiocbe, DONE_BUF);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
Finish up the vast majority of the AIO/LIO functionality. Proper signal support was missing in the previous version of the AIO code. More tunables added, and very efficient support for VCHR files has been added. Kernel threads are not used for VCHR files, all work for such files is done for the requesting process directly. Some attempt has been made to charge the requesting process for resource utilization, but more work is needed. aio_fsync is still missing (but the original fsync system call can be used for now.) aio_cancel is essentially a noop, but that is okay per POSIX. More aio_cancel functionality can be added later, if it is found to be needed. The functions implemented include: aio_read, aio_write, lio_listio, aio_error, aio_return, aio_cancel, aio_suspend. The code has been implemented to support the POSIX spec 1003.1b (formerly known as POSIX 1003.4 spec) features of the above. The async I/O features are truly async, with the VCHR mode of operation being essentially the same as physio (for appropriate files) for maximum efficiency. This code also supports the signal capability, is highly tunable, allowing management of resource usage, and has been written to allow a per process usage quota. Both the O'Reilly POSIX.4 book and the actual POSIX 1003.1b document were the reference specs used. Any filedescriptor can be used with these new system calls. I know of no exceptions where these system calls will not work. (TTY's will also probably work.)
1997-11-30 04:36:31 +00:00
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
/* Release mapping into kernel space. */
vunmapbuf(bp);
relpbuf(bp, NULL);
atomic_subtract_int(&num_buf_aio, 1);
Fix and complete the AIO syscalls. There are some performance enhancements coming up soon, but the code is functional. Docs will be forthcoming.
1997-11-29 01:33:10 +00:00
}
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/* syscall - wait for the next completion of an aio request */
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
static int
kern_aio_waitcomplete(struct thread *td, struct aiocb **aiocbp,
struct timespec *ts, struct aiocb_ops *ops)
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
{
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct proc *p = td->td_proc;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
struct timeval atv;
struct kaioinfo *ki;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
struct aiocblist *cb;
struct aiocb *uuaiocb;
int error, status, timo;
clean up whitespace...
2004-08-13 17:43:53 +00:00
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
ops->store_aiocb(aiocbp, NULL);
Add the VFS_AIO config option and leave it off by default. Unless the VFS_AIO option is specified, all aio-related syscalls return ENOSYS. The aio code is very fragile right now, and is unsuitable for default inclusion in a production shell box. Approved by: jkh
2000-02-23 07:44:25 +00:00
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
timo = 0;
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
if (ts) {
if ((ts->tv_nsec < 0) || (ts->tv_nsec >= 1000000000))
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
return (EINVAL);
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
TIMESPEC_TO_TIMEVAL(&atv, ts);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
if (itimerfix(&atv))
return (EINVAL);
timo = tvtohz(&atv);
}
Initialize ki to p->p_aioinfo after we know it's going to be referencing a valid kaioinfo structure. This avoids a potential NULL pointer dereference. Found with: Coverity Prevent(tm) MFC after: 2 weeks
2006-01-15 01:55:45 +00:00
if (p->p_aioinfo == NULL)
In aio_waitcomplete, do not return EAGAIN if no other threads have started aio, instead, initialize aio management structure if it hasn't been done, the reason to adjust this behavior is to make it a bit friendly for threaded program, consider two threads, one submits aio_write, and another just calls aio_waitcomplete to wait any I/O to be completed and recycle the aio requests, before submitter doing any I/O, the recycler wants to wait in kernel. This also fixes inconsistency with other aio syscalls.
2005-11-08 23:48:32 +00:00
aio_init_aioinfo(p);
Initialize ki to p->p_aioinfo after we know it's going to be referencing a valid kaioinfo structure. This avoids a potential NULL pointer dereference. Found with: Coverity Prevent(tm) MFC after: 2 weeks
2006-01-15 01:55:45 +00:00
ki = p->p_aioinfo;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
error = 0;
cb = NULL;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_LOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
while ((cb = TAILQ_FIRST(&ki->kaio_done)) == NULL) {
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
ki->kaio_flags |= KAIO_WAKEUP;
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
error = msleep(&p->p_aioinfo, AIO_MTX(ki), PRIBIO | PCATCH,
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
"aiowc", timo);
1. Remove aio entry from lists earlier in aio_free_entry, so other threads can not see it if we unlock the proc lock (this can happen in knlist_delete). Don't do wakeup, it is not necessary. 2. Decrease kaio_buffer_count in biohelper rather than doing it in aio_bio_done_notify. 3. In aio_bio_done_notify, don't send notification if KAIO_RUNDOWN was set, because the process is already in single thread mode. 4. Use assignment to initialize aiothreadflags. 5. AIOCBLIST_RUNDOWN is not useful, axe the code using it. 6. use LIO_NOP instead of zero.
2006-02-26 12:56:23 +00:00
if (timo && error == ERESTART)
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
error = EINTR;
if (error)
break;
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
}
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (cb != NULL) {
MPASS(cb->jobstate == JOBST_JOBFINISHED);
uuaiocb = cb->uuaiocb;
status = cb->uaiocb._aiocb_private.status;
error = cb->uaiocb._aiocb_private.error;
td->td_retval[0] = status;
if (cb->uaiocb.aio_lio_opcode == LIO_WRITE) {
- Move rusage from being per-process in struct pstats to per-thread in td_ru. This removes the requirement for per-process synchronization in statclock() and mi_switch(). This was previously supported by sched_lock which is going away. All modifications to rusage are now done in the context of the owning thread. reads proceed without locks. - Aggregate exiting threads rusage in thread_exit() such that the exiting thread's rusage is not lost. - Provide a new routine, rufetch() to fetch an aggregate of all rusage structures from all threads in a process. This routine must be used in any place requiring a rusage from a process prior to it's exit. The exited process's rusage is still available via p_ru. - Aggregate tick statistics only on demand via rufetch() or when a thread exits. Tick statistics are kept in the thread and protected by sched_lock until it exits. Initial patch by: attilio Reviewed by: attilio, bde (some objections), arch (mostly silent)
2007-06-01 01:12:45 +00:00
td->td_ru.ru_oublock += cb->outputcharge;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
cb->outputcharge = 0;
} else if (cb->uaiocb.aio_lio_opcode == LIO_READ) {
- Move rusage from being per-process in struct pstats to per-thread in td_ru. This removes the requirement for per-process synchronization in statclock() and mi_switch(). This was previously supported by sched_lock which is going away. All modifications to rusage are now done in the context of the owning thread. reads proceed without locks. - Aggregate exiting threads rusage in thread_exit() such that the exiting thread's rusage is not lost. - Provide a new routine, rufetch() to fetch an aggregate of all rusage structures from all threads in a process. This routine must be used in any place requiring a rusage from a process prior to it's exit. The exited process's rusage is still available via p_ru. - Aggregate tick statistics only on demand via rufetch() or when a thread exits. Tick statistics are kept in the thread and protected by sched_lock until it exits. Initial patch by: attilio Reviewed by: attilio, bde (some objections), arch (mostly silent)
2007-06-01 01:12:45 +00:00
td->td_ru.ru_inblock += cb->inputcharge;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
cb->inputcharge = 0;
}
aio_free_entry(cb);
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
ops->store_aiocb(aiocbp, uuaiocb);
ops->store_error(uuaiocb, error);
ops->store_status(uuaiocb, status);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
} else
Use a dedicated mutex to protect aio queues, the movation is to reduce lock contention with other parts.
2006-05-09 00:10:11 +00:00
AIO_UNLOCK(ki);
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
return (error);
Add aio_waitcomplete(). Make aio work correctly for socket descriptors. Make gratuitous style(9) fixes (me, not the submitter) to make the aio code more readable. PR: kern/12053 Submitted by: Chris Sedore <cmsedore@maxwell.syr.edu>
2000-01-14 02:53:29 +00:00
}
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
int
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_aio_waitcomplete(struct thread *td, struct aio_waitcomplete_args *uap)
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
{
struct timespec ts, *tsp;
int error;
if (uap->timeout) {
/* Get timespec struct. */
error = copyin(uap->timeout, &ts, sizeof(ts));
if (error)
return (error);
tsp = &ts;
} else
tsp = NULL;
return (kern_aio_waitcomplete(td, uap->aiocbp, tsp, &aiocb_ops));
}
static int
kern_aio_fsync(struct thread *td, int op, struct aiocb *aiocbp,
struct aiocb_ops *ops)
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
{
struct proc *p = td->td_proc;
struct kaioinfo *ki;
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
if (op != O_SYNC) /* XXX lack of O_DSYNC */
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
return (EINVAL);
ki = p->p_aioinfo;
if (ki == NULL)
aio_init_aioinfo(p);
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
return (aio_aqueue(td, aiocbp, NULL, LIO_SYNC, ops));
}
int
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
sys_aio_fsync(struct thread *td, struct aio_fsync_args *uap)
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
{
return (kern_aio_fsync(td, uap->op, uap->aiocbp, &aiocb_ops));
Implement aio_fsync() syscall.
2006-03-23 08:46:42 +00:00
}
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/* kqueue attach function */
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
static int
filt_aioattach(struct knote *kn)
{
Set the ident field of the struct kevent that is registered by _aio_aqueue() to the address of the user's aiocb rather than the kernel's aiocb. (In other words, prior to this change, the ident field returned by kevent(2) on completion of an AIO was effectively garbage.) Submitted by: Romer Gil <rgil@cs.rice.edu>
2002-08-06 19:01:08 +00:00
struct aiocblist *aiocbe = (struct aiocblist *)kn->kn_sdata;
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
/*
* The aiocbe pointer must be validated before using it, so
* registration is restricted to the kernel; the user cannot
* set EV_FLAG1.
*/
if ((kn->kn_flags & EV_FLAG1) == 0)
return (EPERM);
When asked to use kqueue, AIO stores its internal state in the `kn_sdata' member of the newly registered knote. The problem is that this member is overwritten by a call to kevent(2) with the EV_ADD flag, targetted at the same kevent/knote. For instance, a userland application may set the pointer to NULL, leading to a panic. A testcase was provided by the submitter. PR: kern/118911 Submitted by: MOROHOSHI Akihiko <moro@remus.dti.ne.jp> MFC after: 1 day
2008-01-24 17:10:19 +00:00
kn->kn_ptr.p_aio = aiocbe;
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
kn->kn_flags &= ~EV_FLAG1;
Add locking to the kqueue subsystem. This also makes the kqueue subsystem a more complete subsystem, and removes the knowlege of how things are implemented from the drivers. Include locking around filter ops, so a module like aio will know when not to be unloaded if there are outstanding knotes using it's filter ops. Currently, it uses the MTX_DUPOK even though it is not always safe to aquire duplicate locks. Witness currently doesn't support the ability to discover if a dup lock is ok (in some cases). Reviewed by: green, rwatson (both earlier versions)
2004-08-15 06:24:42 +00:00
knlist_add(&aiocbe->klist, kn, 0);
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
return (0);
}
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/* kqueue detach function */
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
static void
filt_aiodetach(struct knote *kn)
{
When detaching an AIO or LIO requests grab the lock and tell knlist_remove that we have the lock now. This cleans up a locking panic ASSERT when knlist_empty is called without a lock when INVARIANTS etc. are turned. Reviewed by: kib jhb MFC after: 1 week
2012-01-30 19:19:22 +00:00
struct knlist *knl;
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
When detaching an AIO or LIO requests grab the lock and tell knlist_remove that we have the lock now. This cleans up a locking panic ASSERT when knlist_empty is called without a lock when INVARIANTS etc. are turned. Reviewed by: kib jhb MFC after: 1 week
2012-01-30 19:19:22 +00:00
knl = &kn->kn_ptr.p_aio->klist;
knl->kl_lock(knl->kl_lockarg);
if (!knlist_empty(knl))
knlist_remove(knl, kn, 1);
knl->kl_unlock(knl->kl_lockarg);
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
}
Document all functions, global and static variables, and sysctls. Includes some minor whitespace changes, and re-ordering to be able to document properly (e.g, grouping of variables and the SYSCTL macro calls for them, where the documentation has been added.) Reviewed by: phk (but all errors are mine)
2002-03-05 15:38:49 +00:00
/* kqueue filter function */
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
/*ARGSUSED*/
static int
filt_aio(struct knote *kn, long hint)
{
When asked to use kqueue, AIO stores its internal state in the `kn_sdata' member of the newly registered knote. The problem is that this member is overwritten by a call to kevent(2) with the EV_ADD flag, targetted at the same kevent/knote. For instance, a userland application may set the pointer to NULL, leading to a panic. A testcase was provided by the submitter. PR: kern/118911 Submitted by: MOROHOSHI Akihiko <moro@remus.dti.ne.jp> MFC after: 1 day
2008-01-24 17:10:19 +00:00
struct aiocblist *aiocbe = kn->kn_ptr.p_aio;
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
o Eliminate unnecessary synchronization from filt_aiodetach(). o The manual page for kevent says that EVFILT_AIO returns under the same conditions as aio_error(). With that in mind, set the data field of the returned struct kevent to the value that would be returned by aio_error(). o Fix two compilation warnings.
2001-12-09 08:16:36 +00:00
kn->kn_data = aiocbe->uaiocb._aiocb_private.error;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
if (aiocbe->jobstate != JOBST_JOBFINISHED)
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
return (0);
clean up whitespace...
2004-08-13 17:43:53 +00:00
kn->kn_flags |= EV_EOF;
Introduce kqueue() and kevent(), a kernel event notification facility.
2000-04-16 18:53:38 +00:00
return (1);
}
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
/* kqueue attach function */
static int
filt_lioattach(struct knote *kn)
{
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
struct aioliojob * lj = (struct aioliojob *)kn->kn_sdata;
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
/*
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
* The aioliojob pointer must be validated before using it, so
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
* registration is restricted to the kernel; the user cannot
* set EV_FLAG1.
*/
if ((kn->kn_flags & EV_FLAG1) == 0)
return (EPERM);
When asked to use kqueue, AIO stores its internal state in the `kn_sdata' member of the newly registered knote. The problem is that this member is overwritten by a call to kevent(2) with the EV_ADD flag, targetted at the same kevent/knote. For instance, a userland application may set the pointer to NULL, leading to a panic. A testcase was provided by the submitter. PR: kern/118911 Submitted by: MOROHOSHI Akihiko <moro@remus.dti.ne.jp> MFC after: 1 day
2008-01-24 17:10:19 +00:00
kn->kn_ptr.p_lio = lj;
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
kn->kn_flags &= ~EV_FLAG1;
knlist_add(&lj->klist, kn, 0);
return (0);
}
/* kqueue detach function */
static void
filt_liodetach(struct knote *kn)
{
When detaching an AIO or LIO requests grab the lock and tell knlist_remove that we have the lock now. This cleans up a locking panic ASSERT when knlist_empty is called without a lock when INVARIANTS etc. are turned. Reviewed by: kib jhb MFC after: 1 week
2012-01-30 19:19:22 +00:00
struct knlist *knl;
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
When detaching an AIO or LIO requests grab the lock and tell knlist_remove that we have the lock now. This cleans up a locking panic ASSERT when knlist_empty is called without a lock when INVARIANTS etc. are turned. Reviewed by: kib jhb MFC after: 1 week
2012-01-30 19:19:22 +00:00
knl = &kn->kn_ptr.p_lio->klist;
knl->kl_lock(knl->kl_lockarg);
if (!knlist_empty(knl))
knlist_remove(knl, kn, 1);
knl->kl_unlock(knl->kl_lockarg);
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
}
/* kqueue filter function */
/*ARGSUSED*/
static int
filt_lio(struct knote *kn, long hint)
{
When asked to use kqueue, AIO stores its internal state in the `kn_sdata' member of the newly registered knote. The problem is that this member is overwritten by a call to kevent(2) with the EV_ADD flag, targetted at the same kevent/knote. For instance, a userland application may set the pointer to NULL, leading to a panic. A testcase was provided by the submitter. PR: kern/118911 Submitted by: MOROHOSHI Akihiko <moro@remus.dti.ne.jp> MFC after: 1 day
2008-01-24 17:10:19 +00:00
struct aioliojob * lj = kn->kn_ptr.p_lio;
Make aio code MP safe.
2006-01-22 05:59:27 +00:00
Add in kqueue support to LIO event notification and fix how it handled notifications when LIO operations completed. These were the problems with LIO event complete notification: - Move all LIO/AIO event notification into one general function so we don't have bugs in different data paths. This unification got rid of several notification bugs one of which if kqueue was used a SIGILL could get sent to the process. - Change the LIO event accounting to count all AIO request that could have been split across the fast path and daemon mode. The prior accounting only kept track of AIO op's in that mode and not the entire list of operations. This could cause a bogus LIO event complete notification to occur when all of the fast path AIO op's completed and not the AIO op's that ended up queued for the daemon. Suggestions from: alc
2005-10-12 17:51:31 +00:00
return (lj->lioj_flags & LIOJ_KEVENT_POSTED);
}
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
Provide groundwork for 32-bit binary compatibility on non-x86 platforms, for upcoming 64-bit PowerPC and MIPS support. This renames the COMPAT_IA32 option to COMPAT_FREEBSD32, removes some IA32-specific code from MI parts of the kernel and enhances the freebsd32 compatibility code to support big-endian platforms. Reviewed by: kib, jhb
2010-03-11 14:49:06 +00:00
#ifdef COMPAT_FREEBSD32
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
struct __aiocb_private32 {
int32_t status;
int32_t error;
uint32_t kernelinfo;
};
typedef struct oaiocb32 {
int aio_fildes; /* File descriptor */
uint64_t aio_offset __packed; /* File offset for I/O */
uint32_t aio_buf; /* I/O buffer in process space */
uint32_t aio_nbytes; /* Number of bytes for I/O */
struct osigevent32 aio_sigevent; /* Signal to deliver */
int aio_lio_opcode; /* LIO opcode */
int aio_reqprio; /* Request priority -- ignored */
struct __aiocb_private32 _aiocb_private;
} oaiocb32_t;
typedef struct aiocb32 {
int32_t aio_fildes; /* File descriptor */
uint64_t aio_offset __packed; /* File offset for I/O */
uint32_t aio_buf; /* I/O buffer in process space */
uint32_t aio_nbytes; /* Number of bytes for I/O */
int __spare__[2];
uint32_t __spare2__;
int aio_lio_opcode; /* LIO opcode */
int aio_reqprio; /* Request priority -- ignored */
struct __aiocb_private32 _aiocb_private;
struct sigevent32 aio_sigevent; /* Signal to deliver */
} aiocb32_t;
static int
convert_old_sigevent32(struct osigevent32 *osig, struct sigevent *nsig)
{
/*
* Only SIGEV_NONE, SIGEV_SIGNAL, and SIGEV_KEVENT are
* supported by AIO with the old sigevent structure.
*/
CP(*osig, *nsig, sigev_notify);
switch (nsig->sigev_notify) {
case SIGEV_NONE:
break;
case SIGEV_SIGNAL:
nsig->sigev_signo = osig->__sigev_u.__sigev_signo;
break;
case SIGEV_KEVENT:
nsig->sigev_notify_kqueue =
osig->__sigev_u.__sigev_notify_kqueue;
PTRIN_CP(*osig, *nsig, sigev_value.sival_ptr);
break;
default:
return (EINVAL);
}
return (0);
}
static int
aiocb32_copyin_old_sigevent(struct aiocb *ujob, struct aiocb *kjob)
{
struct oaiocb32 job32;
int error;
bzero(kjob, sizeof(struct aiocb));
error = copyin(ujob, &job32, sizeof(job32));
if (error)
return (error);
CP(job32, *kjob, aio_fildes);
CP(job32, *kjob, aio_offset);
PTRIN_CP(job32, *kjob, aio_buf);
CP(job32, *kjob, aio_nbytes);
CP(job32, *kjob, aio_lio_opcode);
CP(job32, *kjob, aio_reqprio);
CP(job32, *kjob, _aiocb_private.status);
CP(job32, *kjob, _aiocb_private.error);
PTRIN_CP(job32, *kjob, _aiocb_private.kernelinfo);
return (convert_old_sigevent32(&job32.aio_sigevent,
&kjob->aio_sigevent));
}
static int
aiocb32_copyin(struct aiocb *ujob, struct aiocb *kjob)
{
struct aiocb32 job32;
int error;
error = copyin(ujob, &job32, sizeof(job32));
if (error)
return (error);
CP(job32, *kjob, aio_fildes);
CP(job32, *kjob, aio_offset);
PTRIN_CP(job32, *kjob, aio_buf);
CP(job32, *kjob, aio_nbytes);
CP(job32, *kjob, aio_lio_opcode);
CP(job32, *kjob, aio_reqprio);
CP(job32, *kjob, _aiocb_private.status);
CP(job32, *kjob, _aiocb_private.error);
PTRIN_CP(job32, *kjob, _aiocb_private.kernelinfo);
return (convert_sigevent32(&job32.aio_sigevent, &kjob->aio_sigevent));
}
static long
aiocb32_fetch_status(struct aiocb *ujob)
{
struct aiocb32 *ujob32;
ujob32 = (struct aiocb32 *)ujob;
return (fuword32(&ujob32->_aiocb_private.status));
}
static long
aiocb32_fetch_error(struct aiocb *ujob)
{
struct aiocb32 *ujob32;
ujob32 = (struct aiocb32 *)ujob;
return (fuword32(&ujob32->_aiocb_private.error));
}
static int
aiocb32_store_status(struct aiocb *ujob, long status)
{
struct aiocb32 *ujob32;
ujob32 = (struct aiocb32 *)ujob;
return (suword32(&ujob32->_aiocb_private.status, status));
}
static int
aiocb32_store_error(struct aiocb *ujob, long error)
{
struct aiocb32 *ujob32;
ujob32 = (struct aiocb32 *)ujob;
return (suword32(&ujob32->_aiocb_private.error, error));
}
static int
aiocb32_store_kernelinfo(struct aiocb *ujob, long jobref)
{
struct aiocb32 *ujob32;
ujob32 = (struct aiocb32 *)ujob;
return (suword32(&ujob32->_aiocb_private.kernelinfo, jobref));
}
static int
aiocb32_store_aiocb(struct aiocb **ujobp, struct aiocb *ujob)
{
return (suword32(ujobp, (long)ujob));
}
static struct aiocb_ops aiocb32_ops = {
.copyin = aiocb32_copyin,
.fetch_status = aiocb32_fetch_status,
.fetch_error = aiocb32_fetch_error,
.store_status = aiocb32_store_status,
.store_error = aiocb32_store_error,
.store_kernelinfo = aiocb32_store_kernelinfo,
.store_aiocb = aiocb32_store_aiocb,
};
static struct aiocb_ops aiocb32_ops_osigevent = {
.copyin = aiocb32_copyin_old_sigevent,
.fetch_status = aiocb32_fetch_status,
.fetch_error = aiocb32_fetch_error,
.store_status = aiocb32_store_status,
.store_error = aiocb32_store_error,
.store_kernelinfo = aiocb32_store_kernelinfo,
.store_aiocb = aiocb32_store_aiocb,
};
int
freebsd32_aio_return(struct thread *td, struct freebsd32_aio_return_args *uap)
{
return (kern_aio_return(td, (struct aiocb *)uap->aiocbp, &aiocb32_ops));
}
int
freebsd32_aio_suspend(struct thread *td, struct freebsd32_aio_suspend_args *uap)
{
struct timespec32 ts32;
struct timespec ts, *tsp;
struct aiocb **ujoblist;
uint32_t *ujoblist32;
int error, i;
if (uap->nent < 0 || uap->nent > AIO_LISTIO_MAX)
return (EINVAL);
if (uap->timeout) {
/* Get timespec struct. */
if ((error = copyin(uap->timeout, &ts32, sizeof(ts32))) != 0)
return (error);
CP(ts32, ts, tv_sec);
CP(ts32, ts, tv_nsec);
tsp = &ts;
} else
tsp = NULL;
ujoblist = uma_zalloc(aiol_zone, M_WAITOK);
ujoblist32 = (uint32_t *)ujoblist;
error = copyin(uap->aiocbp, ujoblist32, uap->nent *
sizeof(ujoblist32[0]));
if (error == 0) {
for (i = uap->nent; i > 0; i--)
ujoblist[i] = PTRIN(ujoblist32[i]);
error = kern_aio_suspend(td, uap->nent, ujoblist, tsp);
}
uma_zfree(aiol_zone, ujoblist);
return (error);
}
int
freebsd32_aio_cancel(struct thread *td, struct freebsd32_aio_cancel_args *uap)
{
In order to maximize the re-usability of kernel code in user space this patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
2011-09-16 13:58:51 +00:00
return (sys_aio_cancel(td, (struct aio_cancel_args *)uap));
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
}
int
freebsd32_aio_error(struct thread *td, struct freebsd32_aio_error_args *uap)
{
return (kern_aio_error(td, (struct aiocb *)uap->aiocbp, &aiocb32_ops));
}
int
freebsd32_oaio_read(struct thread *td, struct freebsd32_oaio_read_args *uap)
{
return (aio_aqueue(td, (struct aiocb *)uap->aiocbp, NULL, LIO_READ,
&aiocb32_ops_osigevent));
}
int
freebsd32_aio_read(struct thread *td, struct freebsd32_aio_read_args *uap)
{
return (aio_aqueue(td, (struct aiocb *)uap->aiocbp, NULL, LIO_READ,
&aiocb32_ops));
}
int
freebsd32_oaio_write(struct thread *td, struct freebsd32_oaio_write_args *uap)
{
return (aio_aqueue(td, (struct aiocb *)uap->aiocbp, NULL, LIO_WRITE,
&aiocb32_ops_osigevent));
}
int
freebsd32_aio_write(struct thread *td, struct freebsd32_aio_write_args *uap)
{
return (aio_aqueue(td, (struct aiocb *)uap->aiocbp, NULL, LIO_WRITE,
&aiocb32_ops));
}
Add new system call - aio_mlock(). The name speaks for itself. It allows to perform the mlock(2) operation, which can consume a lot of time, under control of aio(4). Reviewed by: kib, jilles Sponsored by: Nginx, Inc.
2013-06-08 13:27:57 +00:00
int
freebsd32_aio_mlock(struct thread *td, struct freebsd32_aio_mlock_args *uap)
{
return (aio_aqueue(td, (struct aiocb *)uap->aiocbp, NULL, LIO_MLOCK,
&aiocb32_ops));
}
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
int
freebsd32_aio_waitcomplete(struct thread *td,
struct freebsd32_aio_waitcomplete_args *uap)
{
Use the correct type for the timeout parameter to the 32-bit compat version aio_waitcomplete(). Reminded by: bz Submitted by: jamie MFC after: 3 days
2009-01-23 13:23:17 +00:00
struct timespec32 ts32;
- Add 32-bit compat system calls for VFS_AIO. The system calls live in the aio code and are registered via the recently added SYSCALL32_*() helpers. - Since the aio code likes to invoke fuword and suword a lot down in the "bowels" of system calls, add a structure holding a set of operations for things like storing errors, copying in the aiocb structure, storing status, etc. The 32-bit system calls use a separate operations vector to handle fuword32 vs fuword, etc. Also, the oldsigevent handling is now done by having seperate operation vectors with different aiocb copyin routines. - Split out kern_foo() functions for the various AIO system calls so the 32-bit front ends can manage things like copying in and converting timespec structures, etc. - For both the native and 32-bit aio_suspend() and lio_listio() calls, just use copyin() to read the array of aiocb pointers instead of using a for loop that iterated over fuword/fuword32. The error handling in the old case was incomplete (lio_listio() just ignored any aiocb's that it got an EFAULT trying to read rather than reporting an error), and possibly slower. MFC after: 1 month
2008-12-10 20:56:19 +00:00
struct timespec ts, *tsp;
int error;
if (uap->timeout) {
/* Get timespec struct. */
error = copyin(uap->timeout, &ts32, sizeof(ts32));
if (error)
return (error);
CP(ts32, ts, tv_sec);
CP(ts32, ts, tv_nsec);
tsp = &ts;
} else
tsp = NULL;
return (kern_aio_waitcomplete(td, (struct aiocb **)uap->aiocbp, tsp,
&aiocb32_ops));
}
int
freebsd32_aio_fsync(struct thread *td, struct freebsd32_aio_fsync_args *uap)
{
return (kern_aio_fsync(td, uap->op, (struct aiocb *)uap->aiocbp,
&aiocb32_ops));
}
int
freebsd32_olio_listio(struct thread *td, struct freebsd32_olio_listio_args *uap)
{
struct aiocb **acb_list;
struct sigevent *sigp, sig;
struct osigevent32 osig;
uint32_t *acb_list32;
int error, i, nent;
if ((uap->mode != LIO_NOWAIT) && (uap->mode != LIO_WAIT))
return (EINVAL);
nent = uap->nent;
if (nent < 0 || nent > AIO_LISTIO_MAX)
return (EINVAL);
if (uap->sig && (uap->mode == LIO_NOWAIT)) {
error = copyin(uap->sig, &osig, sizeof(osig));
if (error)
return (error);
error = convert_old_sigevent32(&osig, &sig);
if (error)
return (error);
sigp = &sig;
} else
sigp = NULL;
acb_list32 = malloc(sizeof(uint32_t) * nent, M_LIO, M_WAITOK);
error = copyin(uap->acb_list, acb_list32, nent * sizeof(uint32_t));
if (error) {
free(acb_list32, M_LIO);
return (error);
}
acb_list = malloc(sizeof(struct aiocb *) * nent, M_LIO, M_WAITOK);
for (i = 0; i < nent; i++)
acb_list[i] = PTRIN(acb_list32[i]);
free(acb_list32, M_LIO);
error = kern_lio_listio(td, uap->mode,
(struct aiocb * const *)uap->acb_list, acb_list, nent, sigp,
&aiocb32_ops_osigevent);
free(acb_list, M_LIO);
return (error);
}
int
freebsd32_lio_listio(struct thread *td, struct freebsd32_lio_listio_args *uap)
{
struct aiocb **acb_list;
struct sigevent *sigp, sig;
struct sigevent32 sig32;
uint32_t *acb_list32;
int error, i, nent;
if ((uap->mode != LIO_NOWAIT) && (uap->mode != LIO_WAIT))
return (EINVAL);
nent = uap->nent;
if (nent < 0 || nent > AIO_LISTIO_MAX)
return (EINVAL);
if (uap->sig && (uap->mode == LIO_NOWAIT)) {
error = copyin(uap->sig, &sig32, sizeof(sig32));
if (error)
return (error);
error = convert_sigevent32(&sig32, &sig);
if (error)
return (error);
sigp = &sig;
} else
sigp = NULL;
acb_list32 = malloc(sizeof(uint32_t) * nent, M_LIO, M_WAITOK);
error = copyin(uap->acb_list, acb_list32, nent * sizeof(uint32_t));
if (error) {
free(acb_list32, M_LIO);
return (error);
}
acb_list = malloc(sizeof(struct aiocb *) * nent, M_LIO, M_WAITOK);
for (i = 0; i < nent; i++)
acb_list[i] = PTRIN(acb_list32[i]);
free(acb_list32, M_LIO);
error = kern_lio_listio(td, uap->mode,
(struct aiocb * const *)uap->acb_list, acb_list, nent, sigp,
&aiocb32_ops);
free(acb_list, M_LIO);
return (error);
}
#endif
Reference in New Issue Copy Permalink