numam-dpdk/lib/librte_stack/rte_stack_lf.h

113 lines
2.6 KiB
C
Raw Normal View History

stack: add lock-free implementation This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF. The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter. The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns. The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table. This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem. The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created. Signed-off-by: Gage Eads <gage.eads@intel.com> Reviewed-by: Olivier Matz <olivier.matz@6wind.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
2019-04-03 23:20:17 +00:00
/* SPDX-License-Identifier: BSD-3-Clause
* Copyright(c) 2019 Intel Corporation
*/
#ifndef _RTE_STACK_LF_H_
#define _RTE_STACK_LF_H_
#if !(defined(RTE_ARCH_X86_64) || defined(RTE_ARCH_ARM64))
#include "rte_stack_lf_stubs.h"
#else
#ifdef RTE_USE_C11_MEM_MODEL
#include "rte_stack_lf_c11.h"
#else
stack: add lock-free implementation This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF. The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter. The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns. The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table. This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem. The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created. Signed-off-by: Gage Eads <gage.eads@intel.com> Reviewed-by: Olivier Matz <olivier.matz@6wind.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
2019-04-03 23:20:17 +00:00
#include "rte_stack_lf_generic.h"
#endif
#endif
stack: add lock-free implementation This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF. The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter. The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns. The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table. This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem. The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created. Signed-off-by: Gage Eads <gage.eads@intel.com> Reviewed-by: Olivier Matz <olivier.matz@6wind.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
2019-04-03 23:20:17 +00:00
/**
* @internal Push several objects on the lock-free stack (MT-safe).
*
* @param s
* A pointer to the stack structure.
* @param obj_table
* A pointer to a table of void * pointers (objects).
* @param n
* The number of objects to push on the stack from the obj_table.
* @return
* Actual number of objects enqueued.
*/
enforce experimental tag at beginning of declarations Putting a '__attribute__((deprecated))' in the middle of a function prototype does not result in the expected result with gcc (while clang is fine with this syntax). $ cat deprecated.c void * __attribute__((deprecated)) incorrect() { return 0; } __attribute__((deprecated)) void *correct(void) { return 0; } int main(int argc, char *argv[]) { incorrect(); correct(); return 0; } $ gcc -o deprecated.o -c deprecated.c deprecated.c: In function ‘main’: deprecated.c:3:1: warning: ‘correct’ is deprecated (declared at deprecated.c:2) [-Wdeprecated-declarations] int main(int argc, char *argv[]) { incorrect(); correct(); return 0; } ^ Move the tag on a separate line and make it the first thing of function prototypes. This is not perfect but we will trust reviewers to catch the other not so easy to detect patterns. sed -i \ -e '/^\([^#].*\)\?__rte_experimental */{' \ -e 's//\1/; s/ *$//; i\' \ -e __rte_experimental \ -e '/^$/d}' \ $(git grep -l __rte_experimental -- '*.h') Special mention for rte_mbuf_data_addr_default(): There is either a bug or a (not yet understood) issue with gcc. gcc won't drop this inline when unused and rte_mbuf_data_addr_default() calls rte_mbuf_buf_addr() which itself is experimental. This results in a build warning when not accepting experimental apis from sources just including rte_mbuf.h. For this specific case, we hide the call to rte_mbuf_buf_addr() under the ALLOW_EXPERIMENTAL_API flag. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: David Marchand <david.marchand@redhat.com>
2019-06-29 11:58:53 +00:00
__rte_experimental
static __rte_always_inline unsigned int
stack: add lock-free implementation This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF. The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter. The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns. The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table. This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem. The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created. Signed-off-by: Gage Eads <gage.eads@intel.com> Reviewed-by: Olivier Matz <olivier.matz@6wind.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
2019-04-03 23:20:17 +00:00
__rte_stack_lf_push(struct rte_stack *s,
void * const *obj_table,
unsigned int n)
{
struct rte_stack_lf_elem *tmp, *first, *last = NULL;
unsigned int i;
if (unlikely(n == 0))
return 0;
/* Pop n free elements */
first = __rte_stack_lf_pop_elems(&s->stack_lf.free, n, NULL, &last);
if (unlikely(first == NULL))
return 0;
/* Construct the list elements */
for (tmp = first, i = 0; i < n; i++, tmp = tmp->next)
tmp->data = obj_table[n - i - 1];
/* Push them to the used list */
__rte_stack_lf_push_elems(&s->stack_lf.used, first, last, n);
return n;
}
/**
* @internal Pop several objects from the lock-free stack (MT-safe).
*
* @param s
* A pointer to the stack structure.
* @param obj_table
* A pointer to a table of void * pointers (objects).
* @param n
* The number of objects to pull from the stack.
* @return
* - Actual number of objects popped.
*/
enforce experimental tag at beginning of declarations Putting a '__attribute__((deprecated))' in the middle of a function prototype does not result in the expected result with gcc (while clang is fine with this syntax). $ cat deprecated.c void * __attribute__((deprecated)) incorrect() { return 0; } __attribute__((deprecated)) void *correct(void) { return 0; } int main(int argc, char *argv[]) { incorrect(); correct(); return 0; } $ gcc -o deprecated.o -c deprecated.c deprecated.c: In function ‘main’: deprecated.c:3:1: warning: ‘correct’ is deprecated (declared at deprecated.c:2) [-Wdeprecated-declarations] int main(int argc, char *argv[]) { incorrect(); correct(); return 0; } ^ Move the tag on a separate line and make it the first thing of function prototypes. This is not perfect but we will trust reviewers to catch the other not so easy to detect patterns. sed -i \ -e '/^\([^#].*\)\?__rte_experimental */{' \ -e 's//\1/; s/ *$//; i\' \ -e __rte_experimental \ -e '/^$/d}' \ $(git grep -l __rte_experimental -- '*.h') Special mention for rte_mbuf_data_addr_default(): There is either a bug or a (not yet understood) issue with gcc. gcc won't drop this inline when unused and rte_mbuf_data_addr_default() calls rte_mbuf_buf_addr() which itself is experimental. This results in a build warning when not accepting experimental apis from sources just including rte_mbuf.h. For this specific case, we hide the call to rte_mbuf_buf_addr() under the ALLOW_EXPERIMENTAL_API flag. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: David Marchand <david.marchand@redhat.com>
2019-06-29 11:58:53 +00:00
__rte_experimental
static __rte_always_inline unsigned int
stack: add lock-free implementation This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF. The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter. The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns. The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table. This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem. The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created. Signed-off-by: Gage Eads <gage.eads@intel.com> Reviewed-by: Olivier Matz <olivier.matz@6wind.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
2019-04-03 23:20:17 +00:00
__rte_stack_lf_pop(struct rte_stack *s, void **obj_table, unsigned int n)
{
struct rte_stack_lf_elem *first, *last = NULL;
if (unlikely(n == 0))
return 0;
/* Pop n used elements */
first = __rte_stack_lf_pop_elems(&s->stack_lf.used,
n, obj_table, &last);
if (unlikely(first == NULL))
return 0;
/* Push the list elements to the free list */
__rte_stack_lf_push_elems(&s->stack_lf.free, first, last, n);
return n;
}
/**
* @internal Initialize a lock-free stack.
*
* @param s
* A pointer to the stack structure.
* @param count
* The size of the stack.
*/
void
rte_stack_lf_init(struct rte_stack *s, unsigned int count);
/**
* @internal Return the memory required for a lock-free stack.
*
* @param count
* The size of the stack.
* @return
* The bytes to allocate for a lock-free stack.
*/
ssize_t
rte_stack_lf_get_memsize(unsigned int count);
#endif /* _RTE_STACK_LF_H_ */