Commit Graph

6 Commits

Author SHA1 Message Date
Hans Petter Selasky
f3de9af633 Fix problem regarding priority inversion when using the concurrency
kit, CK, in the LinuxKPI.

When threads are pinned to a CPU core or when there is only one CPU,
it can happen that a higher priority thread can call the CK
synchronize function while a lower priority thread holds the read
lock. Because the CK's synchronize is a simple wait loop this can lead
to a deadlock situation. To solve this problem use the recently
introduced CK's wait callback function.

When detecting a CK blocking condition figure out the lowest priority
among the blockers and update the calling thread's priority and
yield. If another CPU core is holding the read lock, pin the thread to
the blocked CPU core and update the priority. The calling threads
priority and CPU bindings are restored before return.

If a thread holding a CK read lock is detected to be sleeping, pause()
will be used instead of yield().

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-19 13:03:29 +00:00
Hans Petter Selasky
e0db0ddb39 Remove duplicate prototype in the LinuxKPI to fix compilation warning.
Reported by:		emaste @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-04 20:06:47 +00:00
Hans Petter Selasky
1f827dab9e Update the LinuxKPI RCU and SRCU wrappers for the concurrency kit, CK.
- Optimise the RCU implementation to not allocate and free
ck_epoch_records during runtime. Instead allocate two sets of
ck_epoch_records per CPU for general purpose use. The first set is
only used for reader locks and the second set is only used for
synchronization and barriers and is protected with a regular mutex to
prevent simultaneous issues.

- Move the task structure away from the rcu_head structure and into
the per-CPU structures. This allows the size of the rcu_head structure
to be reduced down to the size of two pointers.

- Fix a bug where the linux_rcu_barrier() function only waited for one
per-CPU epoch record to be completed instead of all.

- Use a critical section or a mutex to protect ck_epoch_begin() and
ck_epoch_end() depending on RCU or SRCU type. All the ck_epoch_xxx()
functions, except ck_epoch_register(), ck_epoch_unregister() and
ck_epoch_recycle() are not re-entrant and needs a critical section or
a mutex to operate in the LinuxKPI, after inspecting the CK
implementation of the above mentioned functions. The simultaneous
issues arise from per-CPU epoch records being shared between multiple
threads depending on the amount of taskswitching and how many threads
are involved with the RCU and SRCU operations.

- Properly free all epoch records by using safe list traversal at
LinuxKPI module unload. It turns out the ck_epoch_recycle() always
have the records on an internal list and use a flag in the epoch
record to track allocated and free entries. This would lead to use
after free during module unload.

- Remove redundant synchronize_rcu() call from the
linux_compat_uninit() function. Let the linux_rcu_runtime_uninit()
function do the final rcu_barrier() instead.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-03 16:28:03 +00:00
Hans Petter Selasky
959d6165a2 Implement srcu_dereference() macro in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-24 14:40:15 +00:00
Hans Petter Selasky
1a01b4e566 Replace dummy implementation of RCU in the LinuxKPI with one based on
the in-kernel concurrency kit's ck_epoch API. Factor RCU hlist_xxx()
functions into own rculist.h header file.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-21 18:04:21 +00:00
Hans Petter Selasky
c094330345 Implement sleepable RCU mechanism using shared exclusive locks.
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2015-12-21 11:03:12 +00:00