d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
99e690772a
freebsd-dev/sys/compat/linuxkpi/common/include/linux/srcu.h

49 lines
1.9 KiB
C
Raw Normal View History

Implement sleepable RCU mechanism using shared exclusive locks. MFC after: 1 week Sponsored by: Mellanox Technologies
2015-12-21 11:03:12 +00:00
/*-
Replace dummy implementation of RCU in the LinuxKPI with one based on the in-kernel concurrency kit's ck_epoch API. Factor RCU hlist_xxx() functions into own rculist.h header file. Obtained from: kmacy @ MFC after: 1 week Sponsored by: Mellanox Technologies
2017-02-21 18:04:21 +00:00
* Copyright (c) 2015-2017 Mellanox Technologies, Ltd.
Implement sleepable RCU mechanism using shared exclusive locks. MFC after: 1 week Sponsored by: Mellanox Technologies
2015-12-21 11:03:12 +00:00
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice unmodified, this list of conditions, and the following
* disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
* IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
* IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
* THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
* $FreeBSD$
*/
Replace dummy implementation of RCU in the LinuxKPI with one based on the in-kernel concurrency kit's ck_epoch API. Factor RCU hlist_xxx() functions into own rculist.h header file. Obtained from: kmacy @ MFC after: 1 week Sponsored by: Mellanox Technologies
2017-02-21 18:04:21 +00:00
Implement sleepable RCU mechanism using shared exclusive locks. MFC after: 1 week Sponsored by: Mellanox Technologies
2015-12-21 11:03:12 +00:00
#ifndef _LINUX_SRCU_H_
#define _LINUX_SRCU_H_
Update the LinuxKPI RCU and SRCU wrappers for the concurrency kit, CK. - Optimise the RCU implementation to not allocate and free ck_epoch_records during runtime. Instead allocate two sets of ck_epoch_records per CPU for general purpose use. The first set is only used for reader locks and the second set is only used for synchronization and barriers and is protected with a regular mutex to prevent simultaneous issues. - Move the task structure away from the rcu_head structure and into the per-CPU structures. This allows the size of the rcu_head structure to be reduced down to the size of two pointers. - Fix a bug where the linux_rcu_barrier() function only waited for one per-CPU epoch record to be completed instead of all. - Use a critical section or a mutex to protect ck_epoch_begin() and ck_epoch_end() depending on RCU or SRCU type. All the ck_epoch_xxx() functions, except ck_epoch_register(), ck_epoch_unregister() and ck_epoch_recycle() are not re-entrant and needs a critical section or a mutex to operate in the LinuxKPI, after inspecting the CK implementation of the above mentioned functions. The simultaneous issues arise from per-CPU epoch records being shared between multiple threads depending on the amount of taskswitching and how many threads are involved with the RCU and SRCU operations. - Properly free all epoch records by using safe list traversal at LinuxKPI module unload. It turns out the ck_epoch_recycle() always have the records on an internal list and use a flag in the epoch record to track allocated and free entries. This would lead to use after free during module unload. - Remove redundant synchronize_rcu() call from the linux_compat_uninit() function. Let the linux_rcu_runtime_uninit() function do the final rcu_barrier() instead. MFC after: 1 week Sponsored by: Mellanox Technologies
2017-03-03 16:28:03 +00:00
struct srcu_epoch_record;
Implement sleepable RCU mechanism using shared exclusive locks. MFC after: 1 week Sponsored by: Mellanox Technologies
2015-12-21 11:03:12 +00:00
struct srcu_struct {
Update the LinuxKPI RCU and SRCU wrappers for the concurrency kit, CK. - Optimise the RCU implementation to not allocate and free ck_epoch_records during runtime. Instead allocate two sets of ck_epoch_records per CPU for general purpose use. The first set is only used for reader locks and the second set is only used for synchronization and barriers and is protected with a regular mutex to prevent simultaneous issues. - Move the task structure away from the rcu_head structure and into the per-CPU structures. This allows the size of the rcu_head structure to be reduced down to the size of two pointers. - Fix a bug where the linux_rcu_barrier() function only waited for one per-CPU epoch record to be completed instead of all. - Use a critical section or a mutex to protect ck_epoch_begin() and ck_epoch_end() depending on RCU or SRCU type. All the ck_epoch_xxx() functions, except ck_epoch_register(), ck_epoch_unregister() and ck_epoch_recycle() are not re-entrant and needs a critical section or a mutex to operate in the LinuxKPI, after inspecting the CK implementation of the above mentioned functions. The simultaneous issues arise from per-CPU epoch records being shared between multiple threads depending on the amount of taskswitching and how many threads are involved with the RCU and SRCU operations. - Properly free all epoch records by using safe list traversal at LinuxKPI module unload. It turns out the ck_epoch_recycle() always have the records on an internal list and use a flag in the epoch record to track allocated and free entries. This would lead to use after free during module unload. - Remove redundant synchronize_rcu() call from the linux_compat_uninit() function. Let the linux_rcu_runtime_uninit() function do the final rcu_barrier() instead. MFC after: 1 week Sponsored by: Mellanox Technologies
2017-03-03 16:28:03 +00:00
struct srcu_epoch_record *ss_epoch_record;
Implement sleepable RCU mechanism using shared exclusive locks. MFC after: 1 week Sponsored by: Mellanox Technologies
2015-12-21 11:03:12 +00:00
};
Implement srcu_dereference() macro in the LinuxKPI. MFC after: 1 week Sponsored by: Mellanox Technologies
2017-02-24 14:40:15 +00:00
#define srcu_dereference(ptr,srcu) ((__typeof(*(ptr)) *)(ptr))
Replace dummy implementation of RCU in the LinuxKPI with one based on the in-kernel concurrency kit's ck_epoch API. Factor RCU hlist_xxx() functions into own rculist.h header file. Obtained from: kmacy @ MFC after: 1 week Sponsored by: Mellanox Technologies
2017-02-21 18:04:21 +00:00
/* prototypes */
Implement sleepable RCU mechanism using shared exclusive locks. MFC after: 1 week Sponsored by: Mellanox Technologies
2015-12-21 11:03:12 +00:00
Replace dummy implementation of RCU in the LinuxKPI with one based on the in-kernel concurrency kit's ck_epoch API. Factor RCU hlist_xxx() functions into own rculist.h header file. Obtained from: kmacy @ MFC after: 1 week Sponsored by: Mellanox Technologies
2017-02-21 18:04:21 +00:00
extern int srcu_read_lock(struct srcu_struct *);
extern void srcu_read_unlock(struct srcu_struct *, int index);
extern void synchronize_srcu(struct srcu_struct *);
Update the LinuxKPI RCU and SRCU wrappers for the concurrency kit, CK. - Optimise the RCU implementation to not allocate and free ck_epoch_records during runtime. Instead allocate two sets of ck_epoch_records per CPU for general purpose use. The first set is only used for reader locks and the second set is only used for synchronization and barriers and is protected with a regular mutex to prevent simultaneous issues. - Move the task structure away from the rcu_head structure and into the per-CPU structures. This allows the size of the rcu_head structure to be reduced down to the size of two pointers. - Fix a bug where the linux_rcu_barrier() function only waited for one per-CPU epoch record to be completed instead of all. - Use a critical section or a mutex to protect ck_epoch_begin() and ck_epoch_end() depending on RCU or SRCU type. All the ck_epoch_xxx() functions, except ck_epoch_register(), ck_epoch_unregister() and ck_epoch_recycle() are not re-entrant and needs a critical section or a mutex to operate in the LinuxKPI, after inspecting the CK implementation of the above mentioned functions. The simultaneous issues arise from per-CPU epoch records being shared between multiple threads depending on the amount of taskswitching and how many threads are involved with the RCU and SRCU operations. - Properly free all epoch records by using safe list traversal at LinuxKPI module unload. It turns out the ck_epoch_recycle() always have the records on an internal list and use a flag in the epoch record to track allocated and free entries. This would lead to use after free during module unload. - Remove redundant synchronize_rcu() call from the linux_compat_uninit() function. Let the linux_rcu_runtime_uninit() function do the final rcu_barrier() instead. MFC after: 1 week Sponsored by: Mellanox Technologies
2017-03-03 16:28:03 +00:00
extern void srcu_barrier(struct srcu_struct *);
Replace dummy implementation of RCU in the LinuxKPI with one based on the in-kernel concurrency kit's ck_epoch API. Factor RCU hlist_xxx() functions into own rculist.h header file. Obtained from: kmacy @ MFC after: 1 week Sponsored by: Mellanox Technologies
2017-02-21 18:04:21 +00:00
extern int init_srcu_struct(struct srcu_struct *);
extern void cleanup_srcu_struct(struct srcu_struct *);
Implement sleepable RCU mechanism using shared exclusive locks. MFC after: 1 week Sponsored by: Mellanox Technologies
2015-12-21 11:03:12 +00:00
#endif /* _LINUX_SRCU_H_ */
Reference in New Issue Copy Permalink