d/numam-dpdk
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
5d6c73dd59
numam-dpdk/lib/librte_hash/rte_cuckoo_hash.c

2139 lines
57 KiB
C
Raw Normal View History

lib: use SPDX tag for Intel copyright files Replace the BSD license header with the SPDX tag for files with only an Intel copyright on them. Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
2017-12-19 15:49:03 +00:00
/* SPDX-License-Identifier: BSD-3-Clause
* Copyright(c) 2010-2016 Intel Corporation
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
* Copyright(c) 2018 Arm Limited
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
*/
#include <string.h>
#include <stdint.h>
#include <errno.h>
#include <stdio.h>
#include <stdarg.h>
#include <sys/queue.h>
#include <rte_common.h>
#include <rte_memory.h> /* for definition of RTE_CACHE_LINE_SIZE */
#include <rte_log.h>
#include <rte_prefetch.h>
#include <rte_branch_prediction.h>
#include <rte_malloc.h>
#include <rte_eal.h>
#include <rte_eal_memconfig.h>
#include <rte_per_lcore.h>
#include <rte_errno.h>
#include <rte_string_fns.h>
#include <rte_cpuflags.h>
#include <rte_rwlock.h>
#include <rte_spinlock.h>
#include <rte_ring.h>
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
#include <rte_compat.h>
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
#include "rte_hash.h"
hash: add scalable multi-writer insertion with Intel TSX This patch introduced scalable multi-writer Cuckoo Hash insertion based on a split Cuckoo Search and Move operation using Intel TSX. It can do scalable hash insertion with 22 cores with little performance loss and negligible TSX abortion rate. * Added an extra rte_hash flag definition to switch default single writer Cuckoo Hash behavior to multiwriter. - If HTM is available, it would use hardware feature for concurrency. - If HTM is not available, it would fall back to spinlock. * Created a rte_cuckoo_hash_x86.h file to hold all x86-arch related cuckoo_hash functions. And rte_cuckoo_hash.c uses compile time flag to select x86 file or other platform-specific implementations. While HTM check is still done at runtime (same idea with RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT) * Moved rte_hash private struct definitions to rte_cuckoo_hash.h, to allow rte_cuckoo_hash_x86.h or future platform dependent functions to include. * Following new functions are created for consistent names when new platform TM support are added. - rte_hash_cuckoo_move_insert_mw_tm: do insertion with bucket movement. - rte_hash_cuckoo_insert_mw_tm: do insertion without bucket movement. * One extra multi-writer test case is added. Signed-off-by: Wei Shen <wei1.shen@intel.com> Signed-off-by: Sameh Gobriel <sameh.gobriel@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-06-16 22:14:14 +00:00
#include "rte_cuckoo_hash.h"
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
#define FOR_EACH_BUCKET(CURRENT_BKT, START_BUCKET) \
for (CURRENT_BKT = START_BUCKET; \
CURRENT_BKT != NULL; \
CURRENT_BKT = CURRENT_BKT->next)
hash: replace libc memcmp with optimized functions for arm64 The following measurements shows improvement over the default libc memcmp function Length(B) by X% over libc memcmp 16 149.57% 32 122.7% 48 104.96% 64 98.21% 80 93.75% 96 90.55% 112 110.48% 128 137.24% Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2015-11-23 18:45:35 +00:00
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
TAILQ_HEAD(rte_hash_list, rte_tailq_entry);
static struct rte_tailq_elem rte_hash_tailq = {
.name = "RTE_HASH",
};
EAL_REGISTER_TAILQ(rte_hash_tailq)
struct rte_hash *
rte_hash_find_existing(const char *name)
{
struct rte_hash *h = NULL;
struct rte_tailq_entry *te;
struct rte_hash_list *hash_list;
hash_list = RTE_TAILQ_CAST(rte_hash_tailq.head, rte_hash_list);
rte_rwlock_read_lock(RTE_EAL_TAILQ_RWLOCK);
TAILQ_FOREACH(te, hash_list, next) {
h = (struct rte_hash *) te->data;
if (strncmp(name, h->name, RTE_HASH_NAMESIZE) == 0)
break;
}
rte_rwlock_read_unlock(RTE_EAL_TAILQ_RWLOCK);
if (te == NULL) {
rte_errno = ENOENT;
return NULL;
}
return h;
}
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
static inline struct rte_hash_bucket *
rte_hash_get_last_bkt(struct rte_hash_bucket *lst_bkt)
{
while (lst_bkt->next != NULL)
lst_bkt = lst_bkt->next;
return lst_bkt;
}
hash: customize compare function Give user a chance to customize the hash key compare function. The default rte_hash_cmp_eq function is set in the rte_hash_create function, but these builtin ones may not good enough, so the user may call this to override the default one. Signed-off-by: Yu Nemo Wenbin <yuwb_bjy@ctbri.com.cn> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2015-12-04 03:11:41 +00:00
void rte_hash_set_cmp_func(struct rte_hash *h, rte_hash_cmp_eq_t func)
{
hash: fix custom compare Set cmp_jump_table_idx to KEY_CUSTOM in rte_hash_cmp_eq so that the custom function we are setting in rte_hash_set_cmp_func properly works. The custom function is only called by rte_hash_cmp_eq if cmp_jump_table_idx is set to KEY_CUSTOM. Fixes: 95da2f8e9c61 ("hash: customize compare function") Signed-off-by: Masoud Hasanifard <masoudhasanifard@gmail.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-09-17 09:43:24 +00:00
h->cmp_jump_table_idx = KEY_CUSTOM;
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
h->rte_hash_custom_cmp_eq = func;
}
static inline int
rte_hash_cmp_eq(const void *key1, const void *key2, const struct rte_hash *h)
{
if (h->cmp_jump_table_idx == KEY_CUSTOM)
return h->rte_hash_custom_cmp_eq(key1, key2, h->key_len);
else
return cmp_jump_table[h->cmp_jump_table_idx](key1, key2, h->key_len);
hash: customize compare function Give user a chance to customize the hash key compare function. The default rte_hash_cmp_eq function is set in the rte_hash_create function, but these builtin ones may not good enough, so the user may call this to override the default one. Signed-off-by: Yu Nemo Wenbin <yuwb_bjy@ctbri.com.cn> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2015-12-04 03:11:41 +00:00
}
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
/*
* We use higher 16 bits of hash as the signature value stored in table.
* We use the lower bits for the primary bucket
* location. Then we XOR primary bucket location and the signature
* to get the secondary bucket location. This is same as
* proposed in Bin Fan, et al's paper
* "MemC3: Compact and Concurrent MemCache with Dumber Caching and
* Smarter Hashing". The benefit to use
* XOR is that one could derive the alternative bucket location
* by only using the current bucket location and the signature.
*/
static inline uint16_t
get_short_sig(const hash_sig_t hash)
{
return hash >> 16;
}
static inline uint32_t
get_prim_bucket_index(const struct rte_hash *h, const hash_sig_t hash)
{
return hash & h->bucket_bitmask;
}
static inline uint32_t
get_alt_bucket_index(const struct rte_hash *h,
uint32_t cur_bkt_idx, uint16_t sig)
{
return (cur_bkt_idx ^ sig) & h->bucket_bitmask;
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
struct rte_hash *
rte_hash_create(const struct rte_hash_parameters *params)
{
struct rte_hash *h = NULL;
struct rte_tailq_entry *te = NULL;
struct rte_hash_list *hash_list;
struct rte_ring *r = NULL;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
struct rte_ring *r_ext = NULL;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
char hash_name[RTE_HASH_NAMESIZE];
hash: free internal ring when freeing hash Since freeing a ring is now possible, then when freeing a hash table, its internal ring can be freed as well. Therefore when a new table, with the same name as a previously freed table, is created, there is no need to look up the already allocated ring. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-02 16:07:13 +00:00
void *k = NULL;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
void *buckets = NULL;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
void *buckets_ext = NULL;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
char ring_name[RTE_RING_NAMESIZE];
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
char ext_ring_name[RTE_RING_NAMESIZE];
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
unsigned num_key_slots;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
unsigned i;
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
unsigned int hw_trans_mem_support = 0, use_local_cache = 0;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
unsigned int ext_table_support = 0;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
unsigned int readwrite_concur_support = 0;
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
unsigned int writer_takes_lock = 0;
hash: support no free on delete rte_hash_lookup_xxx APIs return the index of slot in the key store. Application(reader) can use that index to reference other data structures in its scope. Because of this, the index should not be freed till the application completes using the index. RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is introduced to support this. When this flag is enabled rte_hash_del_xxx APIs do not free the key-store index/internal memory associated with the deleted entry. The new API rte_hash_free_key_with_position should be called to free the key-store index/internal memory after calling rte_hash_del_xxx APIs. Suggested-by: Yipeng Wang <yipeng1.wang@intel.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:30 +00:00
unsigned int no_free_on_del = 0;
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
uint32_t *tbl_chng_cnt = NULL;
unsigned int readwrite_concur_lf_support = 0;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
hash: select cuckoo function at run-time Compile-time function selection can potentially lead to lower performance on generic builds done by distros. Replaced compile time flag checks with run-time function selection. Signed-off-by: Elza Mathew <elza.mathew@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2017-11-06 18:04:02 +00:00
rte_hash_function default_hash_func = (rte_hash_function)rte_jhash;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash_list = RTE_TAILQ_CAST(rte_hash_tailq.head, rte_hash_list);
if (params == NULL) {
RTE_LOG(ERR, HASH, "rte_hash_create has no parameters\n");
return NULL;
}
/* Check for valid parameters */
if ((params->entries > RTE_HASH_ENTRIES_MAX) ||
(params->entries < RTE_HASH_BUCKET_ENTRIES) ||
(params->key_len == 0)) {
rte_errno = EINVAL;
RTE_LOG(ERR, HASH, "rte_hash_create has invalid parameters\n");
return NULL;
}
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
/* Validate correct usage of extra options */
if ((params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY) &&
(params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF)) {
rte_errno = EINVAL;
RTE_LOG(ERR, HASH, "rte_hash_create: choose rw concurrency or "
"rw concurrency lock free\n");
return NULL;
}
if ((params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) &&
(params->extra_flag & RTE_HASH_EXTRA_FLAGS_EXT_TABLE)) {
rte_errno = EINVAL;
RTE_LOG(ERR, HASH, "rte_hash_create: extendable bucket "
"feature not supported with rw concurrency "
"lock free\n");
return NULL;
}
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
/* Check extra flags field to check extra options. */
if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT)
hw_trans_mem_support = 1;
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD) {
use_local_cache = 1;
writer_takes_lock = 1;
}
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY) {
readwrite_concur_support = 1;
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
writer_takes_lock = 1;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
}
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_EXT_TABLE)
ext_table_support = 1;
hash: support no free on delete rte_hash_lookup_xxx APIs return the index of slot in the key store. Application(reader) can use that index to reference other data structures in its scope. Because of this, the index should not be freed till the application completes using the index. RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is introduced to support this. When this flag is enabled rte_hash_del_xxx APIs do not free the key-store index/internal memory associated with the deleted entry. The new API rte_hash_free_key_with_position should be called to free the key-store index/internal memory after calling rte_hash_del_xxx APIs. Suggested-by: Yipeng Wang <yipeng1.wang@intel.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:30 +00:00
if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL)
no_free_on_del = 1;
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
readwrite_concur_lf_support = 1;
/* Enable not freeing internal memory/index on delete */
no_free_on_del = 1;
}
hash: fix race condition at creation To avoid a race condition while creating a new hash object, the list has to be locked before the lookup, and released only once the new object is added in the list. As the lock is held by the rte_ring_create(), move its creation at the beginning of the function and only take the lock after the ring is created to avoid a deadlock. Fixes: 48a3991196 ("hash: replace with cuckoo hash implementation") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:28:00 +00:00
/* Store all keys and leave the first entry as a dummy entry for lookup_bulk */
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (use_local_cache)
hash: fix race condition at creation To avoid a race condition while creating a new hash object, the list has to be locked before the lookup, and released only once the new object is added in the list. As the lock is held by the rte_ring_create(), move its creation at the beginning of the function and only take the lock after the ring is created to avoid a deadlock. Fixes: 48a3991196 ("hash: replace with cuckoo hash implementation") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:28:00 +00:00
/*
* Increase number of slots by total number of indices
* that can be stored in the lcore caches
* except for the first cache
*/
num_key_slots = params->entries + (RTE_MAX_LCORE - 1) *
hash: fix key slot size accuracy This commit calculates the needed key slot size more accurately. The previous local cache fix requires the free slot ring to be larger than actually needed. The calculation of the value is inaccurate. Fixes: 5915699153d7 ("hash: fix scaling by reducing contention") Cc: stable@dpdk.org Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:56 +00:00
(LCORE_CACHE_SIZE - 1) + 1;
hash: fix race condition at creation To avoid a race condition while creating a new hash object, the list has to be locked before the lookup, and released only once the new object is added in the list. As the lock is held by the rte_ring_create(), move its creation at the beginning of the function and only take the lock after the ring is created to avoid a deadlock. Fixes: 48a3991196 ("hash: replace with cuckoo hash implementation") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:28:00 +00:00
else
num_key_slots = params->entries + 1;
snprintf(ring_name, sizeof(ring_name), "HT_%s", params->name);
hash: fix ring size Ring stores the free slots available to be used in the key table. The ring size was being increased by 1, because of the dummy slot, used for key misses, but this is not actually stored in the ring, so there is no need to increase it. Fixes: 5915699153d7 ("hash: fix scaling by reducing contention") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com>
2016-08-26 21:30:07 +00:00
/* Create ring (Dummy slot index is not enqueued) */
hash: fix key slot size accuracy This commit calculates the needed key slot size more accurately. The previous local cache fix requires the free slot ring to be larger than actually needed. The calculation of the value is inaccurate. Fixes: 5915699153d7 ("hash: fix scaling by reducing contention") Cc: stable@dpdk.org Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:56 +00:00
r = rte_ring_create(ring_name, rte_align32pow2(num_key_slots),
hash: fix race condition at creation To avoid a race condition while creating a new hash object, the list has to be locked before the lookup, and released only once the new object is added in the list. As the lock is held by the rte_ring_create(), move its creation at the beginning of the function and only take the lock after the ring is created to avoid a deadlock. Fixes: 48a3991196 ("hash: replace with cuckoo hash implementation") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:28:00 +00:00
params->socket_id, 0);
if (r == NULL) {
RTE_LOG(ERR, HASH, "memory allocation failed\n");
goto err;
}
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
const uint32_t num_buckets = rte_align32pow2(params->entries) /
RTE_HASH_BUCKET_ENTRIES;
/* Create ring for extendable buckets. */
if (ext_table_support) {
snprintf(ext_ring_name, sizeof(ext_ring_name), "HT_EXT_%s",
params->name);
r_ext = rte_ring_create(ext_ring_name,
rte_align32pow2(num_buckets + 1),
params->socket_id, 0);
if (r_ext == NULL) {
RTE_LOG(ERR, HASH, "ext buckets memory allocation "
"failed\n");
goto err;
}
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
snprintf(hash_name, sizeof(hash_name), "HT_%s", params->name);
hash: fix race condition at creation To avoid a race condition while creating a new hash object, the list has to be locked before the lookup, and released only once the new object is added in the list. As the lock is held by the rte_ring_create(), move its creation at the beginning of the function and only take the lock after the ring is created to avoid a deadlock. Fixes: 48a3991196 ("hash: replace with cuckoo hash implementation") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:28:00 +00:00
rte_rwlock_write_lock(RTE_EAL_TAILQ_RWLOCK);
/* guarantee there's no existing: this is normally already checked
* by ring creation above */
TAILQ_FOREACH(te, hash_list, next) {
h = (struct rte_hash *) te->data;
if (strncmp(params->name, h->name, RTE_HASH_NAMESIZE) == 0)
break;
}
h = NULL;
if (te != NULL) {
hash: fix allocation of an existing object Change rte_hash*_create() functions to return NULL and set rte_errno to EEXIST when the object name already exists. This is the behavior described in the API documentation in the header file. These functions were returning a pointer to the existing object in that case, but it is a problem as the caller did not know if the object had to be freed or not. Doing this change also makes the hash API more consistent with the other APIs (mempool, rings, ...). Fixes: 916e4f4f4e ("memory: fix for multi process support") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:27:59 +00:00
rte_errno = EEXIST;
hash: fix race condition at creation To avoid a race condition while creating a new hash object, the list has to be locked before the lookup, and released only once the new object is added in the list. As the lock is held by the rte_ring_create(), move its creation at the beginning of the function and only take the lock after the ring is created to avoid a deadlock. Fixes: 48a3991196 ("hash: replace with cuckoo hash implementation") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:28:00 +00:00
te = NULL;
goto err_unlock;
hash: fix allocation of an existing object Change rte_hash*_create() functions to return NULL and set rte_errno to EEXIST when the object name already exists. This is the behavior described in the API documentation in the header file. These functions were returning a pointer to the existing object in that case, but it is a problem as the caller did not know if the object had to be freed or not. Doing this change also makes the hash API more consistent with the other APIs (mempool, rings, ...). Fixes: 916e4f4f4e ("memory: fix for multi process support") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:27:59 +00:00
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
te = rte_zmalloc("HASH_TAILQ_ENTRY", sizeof(*te), 0);
if (te == NULL) {
RTE_LOG(ERR, HASH, "tailq entry allocation failed\n");
hash: fix race condition at creation To avoid a race condition while creating a new hash object, the list has to be locked before the lookup, and released only once the new object is added in the list. As the lock is held by the rte_ring_create(), move its creation at the beginning of the function and only take the lock after the ring is created to avoid a deadlock. Fixes: 48a3991196 ("hash: replace with cuckoo hash implementation") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:28:00 +00:00
goto err_unlock;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
h = (struct rte_hash *)rte_zmalloc_socket(hash_name, sizeof(struct rte_hash),
RTE_CACHE_LINE_SIZE, params->socket_id);
if (h == NULL) {
RTE_LOG(ERR, HASH, "memory allocation failed\n");
hash: fix race condition at creation To avoid a race condition while creating a new hash object, the list has to be locked before the lookup, and released only once the new object is added in the list. As the lock is held by the rte_ring_create(), move its creation at the beginning of the function and only take the lock after the ring is created to avoid a deadlock. Fixes: 48a3991196 ("hash: replace with cuckoo hash implementation") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:28:00 +00:00
goto err_unlock;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
buckets = rte_zmalloc_socket(NULL,
num_buckets * sizeof(struct rte_hash_bucket),
RTE_CACHE_LINE_SIZE, params->socket_id);
if (buckets == NULL) {
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
RTE_LOG(ERR, HASH, "buckets memory allocation failed\n");
hash: fix race condition at creation To avoid a race condition while creating a new hash object, the list has to be locked before the lookup, and released only once the new object is added in the list. As the lock is held by the rte_ring_create(), move its creation at the beginning of the function and only take the lock after the ring is created to avoid a deadlock. Fixes: 48a3991196 ("hash: replace with cuckoo hash implementation") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:28:00 +00:00
goto err_unlock;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
/* Allocate same number of extendable buckets */
if (ext_table_support) {
buckets_ext = rte_zmalloc_socket(NULL,
num_buckets * sizeof(struct rte_hash_bucket),
RTE_CACHE_LINE_SIZE, params->socket_id);
if (buckets_ext == NULL) {
RTE_LOG(ERR, HASH, "ext buckets memory allocation "
"failed\n");
goto err_unlock;
}
/* Populate ext bkt ring. We reserve 0 similar to the
* key-data slot, just in case in future we want to
* use bucket index for the linked list and 0 means NULL
* for next bucket
*/
for (i = 1; i <= num_buckets; i++)
rte_ring_sp_enqueue(r_ext, (void *)((uintptr_t) i));
}
hash: fix key store element alignment Fix the key store array element alignment such that every array element is aligned on KEY_ALIGNMENT boundary. This is required to make 'pdata' in 'struct rte_hash_key' align on its natural boundary for atomic load/store. Fixes: 473d1bebce43 ("hash: allow to store data in hash table") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:31 +00:00
const uint32_t key_entry_size =
RTE_ALIGN(sizeof(struct rte_hash_key) + params->key_len,
KEY_ALIGNMENT);
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
const uint64_t key_tbl_size = (uint64_t) key_entry_size * num_key_slots;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
k = rte_zmalloc_socket(NULL, key_tbl_size,
RTE_CACHE_LINE_SIZE, params->socket_id);
if (k == NULL) {
RTE_LOG(ERR, HASH, "memory allocation failed\n");
hash: fix race condition at creation To avoid a race condition while creating a new hash object, the list has to be locked before the lookup, and released only once the new object is added in the list. As the lock is held by the rte_ring_create(), move its creation at the beginning of the function and only take the lock after the ring is created to avoid a deadlock. Fixes: 48a3991196 ("hash: replace with cuckoo hash implementation") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:28:00 +00:00
goto err_unlock;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
tbl_chng_cnt = rte_zmalloc_socket(NULL, sizeof(uint32_t),
RTE_CACHE_LINE_SIZE, params->socket_id);
if (tbl_chng_cnt == NULL) {
RTE_LOG(ERR, HASH, "memory allocation failed\n");
goto err_unlock;
}
hash: fix build for non-x86 arch Hash library uses optimized compare functions that use x86 intrinsics, therefore non-x86 systems could not build the library. In that case, the compare function is set to the generic memcmp. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Reported-by: Zhigang Lu <zlu@ezchip.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Zhigang Lu <zlu@ezchip.com>
2015-07-17 09:17:58 +00:00
/*
* If x86 architecture is used, select appropriate compare function,
hash: add scalable multi-writer insertion with Intel TSX This patch introduced scalable multi-writer Cuckoo Hash insertion based on a split Cuckoo Search and Move operation using Intel TSX. It can do scalable hash insertion with 22 cores with little performance loss and negligible TSX abortion rate. * Added an extra rte_hash flag definition to switch default single writer Cuckoo Hash behavior to multiwriter. - If HTM is available, it would use hardware feature for concurrency. - If HTM is not available, it would fall back to spinlock. * Created a rte_cuckoo_hash_x86.h file to hold all x86-arch related cuckoo_hash functions. And rte_cuckoo_hash.c uses compile time flag to select x86 file or other platform-specific implementations. While HTM check is still done at runtime (same idea with RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT) * Moved rte_hash private struct definitions to rte_cuckoo_hash.h, to allow rte_cuckoo_hash_x86.h or future platform dependent functions to include. * Following new functions are created for consistent names when new platform TM support are added. - rte_hash_cuckoo_move_insert_mw_tm: do insertion with bucket movement. - rte_hash_cuckoo_insert_mw_tm: do insertion without bucket movement. * One extra multi-writer test case is added. Signed-off-by: Wei Shen <wei1.shen@intel.com> Signed-off-by: Sameh Gobriel <sameh.gobriel@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-06-16 22:14:14 +00:00
* which may use x86 intrinsics, otherwise use memcmp
hash: fix build for non-x86 arch Hash library uses optimized compare functions that use x86 intrinsics, therefore non-x86 systems could not build the library. In that case, the compare function is set to the generic memcmp. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Reported-by: Zhigang Lu <zlu@ezchip.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Zhigang Lu <zlu@ezchip.com>
2015-07-17 09:17:58 +00:00
*/
hash: use common x86 flag Instead of using RTE_ARCH_X86_64, RTE_ARCH_X86_32 and RTE_ARCH_I686, use directly RTTE_ARCH_X86 Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-01 15:03:48 +00:00
#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM64)
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
/* Select function to compare keys */
switch (params->key_len) {
case 16:
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
h->cmp_jump_table_idx = KEY_16_BYTES;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
break;
case 32:
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
h->cmp_jump_table_idx = KEY_32_BYTES;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
break;
case 48:
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
h->cmp_jump_table_idx = KEY_48_BYTES;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
break;
case 64:
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
h->cmp_jump_table_idx = KEY_64_BYTES;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
break;
case 80:
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
h->cmp_jump_table_idx = KEY_80_BYTES;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
break;
case 96:
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
h->cmp_jump_table_idx = KEY_96_BYTES;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
break;
case 112:
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
h->cmp_jump_table_idx = KEY_112_BYTES;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
break;
case 128:
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
h->cmp_jump_table_idx = KEY_128_BYTES;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
break;
default:
/* If key is not multiple of 16, use generic memcmp */
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
h->cmp_jump_table_idx = KEY_OTHER_BYTES;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
hash: fix build for non-x86 arch Hash library uses optimized compare functions that use x86 intrinsics, therefore non-x86 systems could not build the library. In that case, the compare function is set to the generic memcmp. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Reported-by: Zhigang Lu <zlu@ezchip.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Zhigang Lu <zlu@ezchip.com>
2015-07-17 09:17:58 +00:00
#else
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
h->cmp_jump_table_idx = KEY_OTHER_BYTES;
hash: fix build for non-x86 arch Hash library uses optimized compare functions that use x86 intrinsics, therefore non-x86 systems could not build the library. In that case, the compare function is set to the generic memcmp. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Reported-by: Zhigang Lu <zlu@ezchip.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Zhigang Lu <zlu@ezchip.com>
2015-07-17 09:17:58 +00:00
#endif
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (use_local_cache) {
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
h->local_free_slots = rte_zmalloc_socket(NULL,
sizeof(struct lcore_cache) * RTE_MAX_LCORE,
RTE_CACHE_LINE_SIZE, params->socket_id);
}
hash: select cuckoo function at run-time Compile-time function selection can potentially lead to lower performance on generic builds done by distros. Replaced compile time flag checks with run-time function selection. Signed-off-by: Elza Mathew <elza.mathew@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2017-11-06 18:04:02 +00:00
/* Default hash function */
#if defined(RTE_ARCH_X86)
default_hash_func = (rte_hash_function)rte_hash_crc;
#elif defined(RTE_ARCH_ARM64)
if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_CRC32))
default_hash_func = (rte_hash_function)rte_hash_crc;
#endif
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
/* Setup hash context */
snprintf(h->name, sizeof(h->name), "%s", params->name);
h->entries = params->entries;
h->key_len = params->key_len;
h->key_entry_size = key_entry_size;
h->hash_func_init_val = params->hash_func_init_val;
h->num_buckets = num_buckets;
h->bucket_bitmask = h->num_buckets - 1;
h->buckets = buckets;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
h->buckets_ext = buckets_ext;
h->free_ext_bkts = r_ext;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
h->hash_func = (params->hash_func == NULL) ?
hash: select cuckoo function at run-time Compile-time function selection can potentially lead to lower performance on generic builds done by distros. Replaced compile time flag checks with run-time function selection. Signed-off-by: Elza Mathew <elza.mathew@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2017-11-06 18:04:02 +00:00
default_hash_func : params->hash_func;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
h->key_store = k;
h->free_slots = r;
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
h->tbl_chng_cnt = tbl_chng_cnt;
*h->tbl_chng_cnt = 0;
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
h->hw_trans_mem_support = hw_trans_mem_support;
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
h->use_local_cache = use_local_cache;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
h->readwrite_concur_support = readwrite_concur_support;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
h->ext_table_support = ext_table_support;
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
h->writer_takes_lock = writer_takes_lock;
hash: support no free on delete rte_hash_lookup_xxx APIs return the index of slot in the key store. Application(reader) can use that index to reference other data structures in its scope. Because of this, the index should not be freed till the application completes using the index. RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is introduced to support this. When this flag is enabled rte_hash_del_xxx APIs do not free the key-store index/internal memory associated with the deleted entry. The new API rte_hash_free_key_with_position should be called to free the key-store index/internal memory after calling rte_hash_del_xxx APIs. Suggested-by: Yipeng Wang <yipeng1.wang@intel.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:30 +00:00
h->no_free_on_del = no_free_on_del;
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
h->readwrite_concur_lf_support = readwrite_concur_lf_support;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: add vectorized comparison In lookup bulk function, the signatures of all entries are compared against the signature of the key that is being looked up. Now that all the signatures are together, they can be compared with vector instructions (SSE, AVX2), achieving higher lookup performance. Also, entries per bucket are increased to 8 when using processors with AVX2, as 256 bits can be compared at once, which is the size of 8x32-bit signatures. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:14 +00:00
#if defined(RTE_ARCH_X86)
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_SSE2))
hash: add vectorized comparison In lookup bulk function, the signatures of all entries are compared against the signature of the key that is being looked up. Now that all the signatures are together, they can be compared with vector instructions (SSE, AVX2), achieving higher lookup performance. Also, entries per bucket are increased to 8 when using processors with AVX2, as 256 bits can be compared at once, which is the size of 8x32-bit signatures. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:14 +00:00
h->sig_cmp_fn = RTE_HASH_COMPARE_SSE;
else
#endif
h->sig_cmp_fn = RTE_HASH_COMPARE_SCALAR;
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
/* Writer threads need to take the lock when:
* 1) RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY is enabled OR
* 2) RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD is enabled
hash: add scalable multi-writer insertion with Intel TSX This patch introduced scalable multi-writer Cuckoo Hash insertion based on a split Cuckoo Search and Move operation using Intel TSX. It can do scalable hash insertion with 22 cores with little performance loss and negligible TSX abortion rate. * Added an extra rte_hash flag definition to switch default single writer Cuckoo Hash behavior to multiwriter. - If HTM is available, it would use hardware feature for concurrency. - If HTM is not available, it would fall back to spinlock. * Created a rte_cuckoo_hash_x86.h file to hold all x86-arch related cuckoo_hash functions. And rte_cuckoo_hash.c uses compile time flag to select x86 file or other platform-specific implementations. While HTM check is still done at runtime (same idea with RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT) * Moved rte_hash private struct definitions to rte_cuckoo_hash.h, to allow rte_cuckoo_hash_x86.h or future platform dependent functions to include. * Following new functions are created for consistent names when new platform TM support are added. - rte_hash_cuckoo_move_insert_mw_tm: do insertion with bucket movement. - rte_hash_cuckoo_insert_mw_tm: do insertion without bucket movement. * One extra multi-writer test case is added. Signed-off-by: Wei Shen <wei1.shen@intel.com> Signed-off-by: Sameh Gobriel <sameh.gobriel@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-06-16 22:14:14 +00:00
*/
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (h->writer_takes_lock) {
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
h->readwrite_lock = rte_malloc(NULL, sizeof(rte_rwlock_t),
RTE_CACHE_LINE_SIZE);
if (h->readwrite_lock == NULL)
goto err_unlock;
rte_rwlock_init(h->readwrite_lock);
}
hash: add scalable multi-writer insertion with Intel TSX This patch introduced scalable multi-writer Cuckoo Hash insertion based on a split Cuckoo Search and Move operation using Intel TSX. It can do scalable hash insertion with 22 cores with little performance loss and negligible TSX abortion rate. * Added an extra rte_hash flag definition to switch default single writer Cuckoo Hash behavior to multiwriter. - If HTM is available, it would use hardware feature for concurrency. - If HTM is not available, it would fall back to spinlock. * Created a rte_cuckoo_hash_x86.h file to hold all x86-arch related cuckoo_hash functions. And rte_cuckoo_hash.c uses compile time flag to select x86 file or other platform-specific implementations. While HTM check is still done at runtime (same idea with RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT) * Moved rte_hash private struct definitions to rte_cuckoo_hash.h, to allow rte_cuckoo_hash_x86.h or future platform dependent functions to include. * Following new functions are created for consistent names when new platform TM support are added. - rte_hash_cuckoo_move_insert_mw_tm: do insertion with bucket movement. - rte_hash_cuckoo_insert_mw_tm: do insertion without bucket movement. * One extra multi-writer test case is added. Signed-off-by: Wei Shen <wei1.shen@intel.com> Signed-off-by: Sameh Gobriel <sameh.gobriel@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-06-16 22:14:14 +00:00
/* Populate free slots ring. Entry zero is reserved for key misses. */
hash: fix key slot size accuracy This commit calculates the needed key slot size more accurately. The previous local cache fix requires the free slot ring to be larger than actually needed. The calculation of the value is inaccurate. Fixes: 5915699153d7 ("hash: fix scaling by reducing contention") Cc: stable@dpdk.org Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:56 +00:00
for (i = 1; i < num_key_slots; i++)
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
rte_ring_sp_enqueue(r, (void *)((uintptr_t) i));
te->data = (void *) h;
TAILQ_INSERT_TAIL(hash_list, te, next);
rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
return h;
hash: fix race condition at creation To avoid a race condition while creating a new hash object, the list has to be locked before the lookup, and released only once the new object is added in the list. As the lock is held by the rte_ring_create(), move its creation at the beginning of the function and only take the lock after the ring is created to avoid a deadlock. Fixes: 48a3991196 ("hash: replace with cuckoo hash implementation") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:28:00 +00:00
err_unlock:
rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
err:
hash: fix race condition at creation To avoid a race condition while creating a new hash object, the list has to be locked before the lookup, and released only once the new object is added in the list. As the lock is held by the rte_ring_create(), move its creation at the beginning of the function and only take the lock after the ring is created to avoid a deadlock. Fixes: 48a3991196 ("hash: replace with cuckoo hash implementation") Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-04-06 13:28:00 +00:00
rte_ring_free(r);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
rte_ring_free(r_ext);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
rte_free(te);
rte_free(h);
rte_free(buckets);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
rte_free(buckets_ext);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
rte_free(k);
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
rte_free(tbl_chng_cnt);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
return NULL;
}
void
rte_hash_free(struct rte_hash *h)
{
struct rte_tailq_entry *te;
struct rte_hash_list *hash_list;
if (h == NULL)
return;
hash_list = RTE_TAILQ_CAST(rte_hash_tailq.head, rte_hash_list);
rte_rwlock_write_lock(RTE_EAL_TAILQ_RWLOCK);
/* find out tailq entry */
TAILQ_FOREACH(te, hash_list, next) {
if (te->data == (void *) h)
break;
}
if (te == NULL) {
rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
return;
}
TAILQ_REMOVE(hash_list, te, next);
rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (h->use_local_cache)
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
rte_free(h->local_free_slots);
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (h->writer_takes_lock)
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
rte_free(h->readwrite_lock);
hash: free internal ring when freeing hash Since freeing a ring is now possible, then when freeing a hash table, its internal ring can be freed as well. Therefore when a new table, with the same name as a previously freed table, is created, there is no need to look up the already allocated ring. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-02 16:07:13 +00:00
rte_ring_free(h->free_slots);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
rte_ring_free(h->free_ext_bkts);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
rte_free(h->key_store);
rte_free(h->buckets);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
rte_free(h->buckets_ext);
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
rte_free(h->tbl_chng_cnt);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
rte_free(h);
rte_free(te);
}
hash_sig_t
rte_hash_hash(const struct rte_hash *h, const void *key)
{
/* calc hash result by key */
return h->hash_func(key, h->key_len, h->hash_func_init_val);
}
hash: add API to query the key count Add a new function, rte_hash_count, to return the number of keys that are currently stored in the hash table. Corresponding test functions are added into hash_test and hash_multiwriter test. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 17:00:01 +00:00
int32_t
rte_hash_count(const struct rte_hash *h)
{
uint32_t tot_ring_cnt, cached_cnt = 0;
uint32_t i, ret;
if (h == NULL)
return -EINVAL;
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (h->use_local_cache) {
hash: add API to query the key count Add a new function, rte_hash_count, to return the number of keys that are currently stored in the hash table. Corresponding test functions are added into hash_test and hash_multiwriter test. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 17:00:01 +00:00
tot_ring_cnt = h->entries + (RTE_MAX_LCORE - 1) *
(LCORE_CACHE_SIZE - 1);
for (i = 0; i < RTE_MAX_LCORE; i++)
cached_cnt += h->local_free_slots[i].len;
ret = tot_ring_cnt - rte_ring_count(h->free_slots) -
cached_cnt;
} else {
tot_ring_cnt = h->entries;
ret = tot_ring_cnt - rte_ring_count(h->free_slots);
}
return ret;
}
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
/* Read write locks implemented using rte_rwlock */
static inline void
__hash_rw_writer_lock(const struct rte_hash *h)
{
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (h->writer_takes_lock && h->hw_trans_mem_support)
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
rte_rwlock_write_lock_tm(h->readwrite_lock);
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
else if (h->writer_takes_lock)
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
rte_rwlock_write_lock(h->readwrite_lock);
}
static inline void
__hash_rw_reader_lock(const struct rte_hash *h)
{
if (h->readwrite_concur_support && h->hw_trans_mem_support)
rte_rwlock_read_lock_tm(h->readwrite_lock);
else if (h->readwrite_concur_support)
rte_rwlock_read_lock(h->readwrite_lock);
}
static inline void
__hash_rw_writer_unlock(const struct rte_hash *h)
{
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (h->writer_takes_lock && h->hw_trans_mem_support)
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
rte_rwlock_write_unlock_tm(h->readwrite_lock);
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
else if (h->writer_takes_lock)
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
rte_rwlock_write_unlock(h->readwrite_lock);
}
static inline void
__hash_rw_reader_unlock(const struct rte_hash *h)
{
if (h->readwrite_concur_support && h->hw_trans_mem_support)
rte_rwlock_read_unlock_tm(h->readwrite_lock);
else if (h->readwrite_concur_support)
rte_rwlock_read_unlock(h->readwrite_lock);
}
hash: add reset function Added reset function to be able to empty the table, without having to destroy and create it again. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:51 +00:00
void
rte_hash_reset(struct rte_hash *h)
{
void *ptr;
hash: fix key slot size accuracy This commit calculates the needed key slot size more accurately. The previous local cache fix requires the free slot ring to be larger than actually needed. The calculation of the value is inaccurate. Fixes: 5915699153d7 ("hash: fix scaling by reducing contention") Cc: stable@dpdk.org Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:56 +00:00
uint32_t tot_ring_cnt, i;
hash: add reset function Added reset function to be able to empty the table, without having to destroy and create it again. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:51 +00:00
if (h == NULL)
return;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
__hash_rw_writer_lock(h);
hash: add reset function Added reset function to be able to empty the table, without having to destroy and create it again. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:51 +00:00
memset(h->buckets, 0, h->num_buckets * sizeof(struct rte_hash_bucket));
memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
*h->tbl_chng_cnt = 0;
hash: add reset function Added reset function to be able to empty the table, without having to destroy and create it again. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:51 +00:00
/* clear the free ring */
while (rte_ring_dequeue(h->free_slots, &ptr) == 0)
hash: remove unnecessary pause There is a rte_pause in hash table reset function. Since the loop is not a polling loop on shared data structure, the rte_pause is not needed. Fixes: b26473ff8f4a ("hash: add reset function") Cc: stable@dpdk.org Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 09:53:41 +00:00
continue;
hash: add reset function Added reset function to be able to empty the table, without having to destroy and create it again. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:51 +00:00
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
/* clear free extendable bucket ring and memory */
if (h->ext_table_support) {
memset(h->buckets_ext, 0, h->num_buckets *
sizeof(struct rte_hash_bucket));
while (rte_ring_dequeue(h->free_ext_bkts, &ptr) == 0)
hash: remove unnecessary pause There is a rte_pause in hash table reset function. Since the loop is not a polling loop on shared data structure, the rte_pause is not needed. Fixes: b26473ff8f4a ("hash: add reset function") Cc: stable@dpdk.org Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 09:53:41 +00:00
continue;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
}
hash: add reset function Added reset function to be able to empty the table, without having to destroy and create it again. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:51 +00:00
/* Repopulate the free slots ring. Entry zero is reserved for key misses */
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (h->use_local_cache)
hash: fix key slot size accuracy This commit calculates the needed key slot size more accurately. The previous local cache fix requires the free slot ring to be larger than actually needed. The calculation of the value is inaccurate. Fixes: 5915699153d7 ("hash: fix scaling by reducing contention") Cc: stable@dpdk.org Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:56 +00:00
tot_ring_cnt = h->entries + (RTE_MAX_LCORE - 1) *
(LCORE_CACHE_SIZE - 1);
else
tot_ring_cnt = h->entries;
for (i = 1; i < tot_ring_cnt + 1; i++)
hash: add reset function Added reset function to be able to empty the table, without having to destroy and create it again. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:51 +00:00
rte_ring_sp_enqueue(h->free_slots, (void *)((uintptr_t) i));
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
/* Repopulate the free ext bkt ring. */
if (h->ext_table_support) {
for (i = 1; i <= h->num_buckets; i++)
rte_ring_sp_enqueue(h->free_ext_bkts,
(void *)((uintptr_t) i));
}
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (h->use_local_cache) {
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
/* Reset local caches per lcore */
for (i = 0; i < RTE_MAX_LCORE; i++)
h->local_free_slots[i].len = 0;
}
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
__hash_rw_writer_unlock(h);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
/*
* Function called to enqueue back an index in the cache/ring,
* as slot has not being used and it can be used in the
* next addition attempt.
*/
static inline void
enqueue_slot_back(const struct rte_hash *h,
struct lcore_cache *cached_free_slots,
void *slot_id)
{
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (h->use_local_cache) {
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
cached_free_slots->objs[cached_free_slots->len] = slot_id;
cached_free_slots->len++;
} else
rte_ring_sp_enqueue(h->free_slots, slot_id);
}
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
/* Search a key from bucket and update its data.
* Writer holds the lock before calling this.
*/
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
static inline int32_t
search_and_update(const struct rte_hash *h, void *data, const void *key,
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
struct rte_hash_bucket *bkt, uint16_t sig)
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
{
int i;
struct rte_hash_key *k, *keys = h->key_store;
for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) {
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
if (bkt->sig_current[i] == sig) {
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
k = (struct rte_hash_key *) ((char *)keys +
bkt->key_idx[i] * h->key_entry_size);
if (rte_hash_cmp_eq(key, k->key, h) == 0) {
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
/* 'pdata' acts as the synchronization point
* when an existing hash entry is updated.
* Key is not updated in this case.
*/
__atomic_store_n(&k->pdata,
data,
__ATOMIC_RELEASE);
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
/*
* Return index where key is stored,
* subtracting the first dummy index
*/
return bkt->key_idx[i] - 1;
}
}
}
return -1;
}
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
/* Only tries to insert at one bucket (@prim_bkt) without trying to push
* buckets around.
* return 1 if matching existing key, return 0 if succeeds, return -1 for no
* empty entry.
*/
static inline int32_t
rte_hash_cuckoo_insert_mw(const struct rte_hash *h,
struct rte_hash_bucket *prim_bkt,
struct rte_hash_bucket *sec_bkt,
const struct rte_hash_key *key, void *data,
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
uint16_t sig, uint32_t new_idx,
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
int32_t *ret_val)
{
unsigned int i;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
struct rte_hash_bucket *cur_bkt;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
int32_t ret;
__hash_rw_writer_lock(h);
/* Check if key was inserted after last check but before this
* protected region in case of inserting duplicated keys.
*/
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
ret = search_and_update(h, data, key, prim_bkt, sig);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
if (ret != -1) {
__hash_rw_writer_unlock(h);
*ret_val = ret;
return 1;
}
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
FOR_EACH_BUCKET(cur_bkt, sec_bkt) {
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
ret = search_and_update(h, data, key, cur_bkt, sig);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
if (ret != -1) {
__hash_rw_writer_unlock(h);
*ret_val = ret;
return 1;
}
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
}
/* Insert new entry if there is room in the primary
* bucket.
*/
for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) {
/* Check if slot is available */
if (likely(prim_bkt->key_idx[i] == EMPTY_SLOT)) {
prim_bkt->sig_current[i] = sig;
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
/* Key can be of arbitrary length, so it is
* not possible to store it atomically.
* Hence the new key element's memory stores
* (key as well as data) should be complete
* before it is referenced.
*/
__atomic_store_n(&prim_bkt->key_idx[i],
new_idx,
__ATOMIC_RELEASE);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
break;
}
}
__hash_rw_writer_unlock(h);
if (i != RTE_HASH_BUCKET_ENTRIES)
return 0;
/* no empty entry */
return -1;
}
/* Shift buckets along provided cuckoo_path (@leaf and @leaf_slot) and fill
* the path head with new entry (sig, alt_hash, new_idx)
* return 1 if matched key found, return -1 if cuckoo path invalided and fail,
* return 0 if succeeds.
*/
static inline int
rte_hash_cuckoo_move_insert_mw(const struct rte_hash *h,
struct rte_hash_bucket *bkt,
struct rte_hash_bucket *alt_bkt,
const struct rte_hash_key *key, void *data,
struct queue_node *leaf, uint32_t leaf_slot,
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
uint16_t sig, uint32_t new_idx,
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
int32_t *ret_val)
{
uint32_t prev_alt_bkt_idx;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
struct rte_hash_bucket *cur_bkt;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
struct queue_node *prev_node, *curr_node = leaf;
struct rte_hash_bucket *prev_bkt, *curr_bkt = leaf->bkt;
uint32_t prev_slot, curr_slot = leaf_slot;
int32_t ret;
__hash_rw_writer_lock(h);
/* In case empty slot was gone before entering protected region */
if (curr_bkt->key_idx[curr_slot] != EMPTY_SLOT) {
__hash_rw_writer_unlock(h);
return -1;
}
/* Check if key was inserted after last check but before this
* protected region.
*/
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
ret = search_and_update(h, data, key, bkt, sig);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
if (ret != -1) {
__hash_rw_writer_unlock(h);
*ret_val = ret;
return 1;
}
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
FOR_EACH_BUCKET(cur_bkt, alt_bkt) {
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
ret = search_and_update(h, data, key, cur_bkt, sig);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
if (ret != -1) {
__hash_rw_writer_unlock(h);
*ret_val = ret;
return 1;
}
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
}
while (likely(curr_node->prev != NULL)) {
prev_node = curr_node->prev;
prev_bkt = prev_node->bkt;
prev_slot = curr_node->prev_slot;
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
prev_alt_bkt_idx = get_alt_bucket_index(h,
prev_node->cur_bkt_idx,
prev_bkt->sig_current[prev_slot]);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
if (unlikely(&h->buckets[prev_alt_bkt_idx]
!= curr_bkt)) {
/* revert it to empty, otherwise duplicated keys */
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
__atomic_store_n(&curr_bkt->key_idx[curr_slot],
EMPTY_SLOT,
__ATOMIC_RELEASE);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
__hash_rw_writer_unlock(h);
return -1;
}
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
if (h->readwrite_concur_lf_support) {
/* Inform the previous move. The current move need
* not be informed now as the current bucket entry
* is present in both primary and secondary.
* Since there is one writer, load acquires on
* tbl_chng_cnt are not required.
*/
__atomic_store_n(h->tbl_chng_cnt,
*h->tbl_chng_cnt + 1,
__ATOMIC_RELEASE);
/* The stores to sig_alt and sig_current should not
* move above the store to tbl_chng_cnt.
*/
__atomic_thread_fence(__ATOMIC_RELEASE);
}
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
/* Need to swap current/alt sig to allow later
* Cuckoo insert to move elements back to its
* primary bucket if available
*/
curr_bkt->sig_current[curr_slot] =
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
prev_bkt->sig_current[prev_slot];
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
/* Release the updated bucket entry */
__atomic_store_n(&curr_bkt->key_idx[curr_slot],
prev_bkt->key_idx[prev_slot],
__ATOMIC_RELEASE);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
curr_slot = prev_slot;
curr_node = prev_node;
curr_bkt = curr_node->bkt;
}
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
if (h->readwrite_concur_lf_support) {
/* Inform the previous move. The current move need
* not be informed now as the current bucket entry
* is present in both primary and secondary.
* Since there is one writer, load acquires on
* tbl_chng_cnt are not required.
*/
__atomic_store_n(h->tbl_chng_cnt,
*h->tbl_chng_cnt + 1,
__ATOMIC_RELEASE);
/* The stores to sig_alt and sig_current should not
* move above the store to tbl_chng_cnt.
*/
__atomic_thread_fence(__ATOMIC_RELEASE);
}
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
curr_bkt->sig_current[curr_slot] = sig;
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
/* Release the new bucket entry */
__atomic_store_n(&curr_bkt->key_idx[curr_slot],
new_idx,
__ATOMIC_RELEASE);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
__hash_rw_writer_unlock(h);
return 0;
}
/*
* Make space for new key, using bfs Cuckoo Search and Multi-Writer safe
* Cuckoo
*/
static inline int
rte_hash_cuckoo_make_space_mw(const struct rte_hash *h,
struct rte_hash_bucket *bkt,
struct rte_hash_bucket *sec_bkt,
const struct rte_hash_key *key, void *data,
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
uint16_t sig, uint32_t bucket_idx,
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
uint32_t new_idx, int32_t *ret_val)
{
unsigned int i;
struct queue_node queue[RTE_HASH_BFS_QUEUE_MAX_LEN];
struct queue_node *tail, *head;
struct rte_hash_bucket *curr_bkt, *alt_bkt;
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
uint32_t cur_idx, alt_idx;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
tail = queue;
head = queue + 1;
tail->bkt = bkt;
tail->prev = NULL;
tail->prev_slot = -1;
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
tail->cur_bkt_idx = bucket_idx;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
/* Cuckoo bfs Search */
while (likely(tail != head && head <
queue + RTE_HASH_BFS_QUEUE_MAX_LEN -
RTE_HASH_BUCKET_ENTRIES)) {
curr_bkt = tail->bkt;
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
cur_idx = tail->cur_bkt_idx;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) {
if (curr_bkt->key_idx[i] == EMPTY_SLOT) {
int32_t ret = rte_hash_cuckoo_move_insert_mw(h,
bkt, sec_bkt, key, data,
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
tail, i, sig,
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
new_idx, ret_val);
if (likely(ret != -1))
return ret;
}
/* Enqueue new node and keep prev node info */
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
alt_idx = get_alt_bucket_index(h, cur_idx,
curr_bkt->sig_current[i]);
alt_bkt = &(h->buckets[alt_idx]);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
head->bkt = alt_bkt;
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
head->cur_bkt_idx = alt_idx;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
head->prev = tail;
head->prev_slot = i;
head++;
}
tail++;
}
return -ENOSPC;
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
static inline int32_t
__rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
hash_sig_t sig, void *data)
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
{
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
uint16_t short_sig;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
uint32_t prim_bucket_idx, sec_bucket_idx;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
struct rte_hash_bucket *prim_bkt, *sec_bkt, *cur_bkt;
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
struct rte_hash_key *new_k, *keys = h->key_store;
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
void *slot_id = NULL;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
void *ext_bkt_id = NULL;
uint32_t new_idx, bkt_id;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
int ret;
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
unsigned n_slots;
unsigned lcore_id;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
unsigned int i;
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
struct lcore_cache *cached_free_slots = NULL;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
int32_t ret_val;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
struct rte_hash_bucket *last;
hash: add scalable multi-writer insertion with Intel TSX This patch introduced scalable multi-writer Cuckoo Hash insertion based on a split Cuckoo Search and Move operation using Intel TSX. It can do scalable hash insertion with 22 cores with little performance loss and negligible TSX abortion rate. * Added an extra rte_hash flag definition to switch default single writer Cuckoo Hash behavior to multiwriter. - If HTM is available, it would use hardware feature for concurrency. - If HTM is not available, it would fall back to spinlock. * Created a rte_cuckoo_hash_x86.h file to hold all x86-arch related cuckoo_hash functions. And rte_cuckoo_hash.c uses compile time flag to select x86 file or other platform-specific implementations. While HTM check is still done at runtime (same idea with RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT) * Moved rte_hash private struct definitions to rte_cuckoo_hash.h, to allow rte_cuckoo_hash_x86.h or future platform dependent functions to include. * Following new functions are created for consistent names when new platform TM support are added. - rte_hash_cuckoo_move_insert_mw_tm: do insertion with bucket movement. - rte_hash_cuckoo_insert_mw_tm: do insertion without bucket movement. * One extra multi-writer test case is added. Signed-off-by: Wei Shen <wei1.shen@intel.com> Signed-off-by: Sameh Gobriel <sameh.gobriel@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-06-16 22:14:14 +00:00
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
short_sig = get_short_sig(sig);
prim_bucket_idx = get_prim_bucket_index(h, sig);
sec_bucket_idx = get_alt_bucket_index(h, prim_bucket_idx, short_sig);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
prim_bkt = &h->buckets[prim_bucket_idx];
sec_bkt = &h->buckets[sec_bucket_idx];
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
rte_prefetch0(prim_bkt);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
rte_prefetch0(sec_bkt);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
/* Check if key is already inserted in primary location */
__hash_rw_writer_lock(h);
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
ret = search_and_update(h, data, key, prim_bkt, short_sig);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
if (ret != -1) {
__hash_rw_writer_unlock(h);
return ret;
}
/* Check if key is already inserted in secondary location */
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
FOR_EACH_BUCKET(cur_bkt, sec_bkt) {
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
ret = search_and_update(h, data, key, cur_bkt, short_sig);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
if (ret != -1) {
__hash_rw_writer_unlock(h);
return ret;
}
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
}
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
__hash_rw_writer_unlock(h);
/* Did not find a match, so get a new slot for storing the new key */
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (h->use_local_cache) {
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
lcore_id = rte_lcore_id();
cached_free_slots = &h->local_free_slots[lcore_id];
/* Try to get a free slot from the local cache */
if (cached_free_slots->len == 0) {
/* Need to get another burst of free slots from global ring */
n_slots = rte_ring_mc_dequeue_burst(h->free_slots,
ring: return remaining entry count when dequeuing Add an extra parameter to the ring dequeue burst/bulk functions so that those functions can optionally return the amount of remaining objs in the ring. This information can be used by applications in a number of ways, for instance, with single-consumer queues, it provides a max dequeue size which is guaranteed to work. Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Yuanhan Liu <yuanhan.liu@linux.intel.com> Acked-by: Olivier Matz <olivier.matz@6wind.com>
2017-03-29 15:21:25 +00:00
cached_free_slots->objs,
LCORE_CACHE_SIZE, NULL);
hash: fix lock release on add When adding items to a hash table with multiple threads, there is an spinlock used to prevent data corruption (unless Transactional Memory is supported). If there is a failure, the spinlock should be released, but there were cases where that was not happening. Fixes: be856325cba3 ("hash: add scalable multi-writer insertion with Intel TSX") Cc: stable@dpdk.org Signed-off-by: Mike Stolarchuk <mike.stolarchuk@bigswitch.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2017-07-07 05:54:25 +00:00
if (n_slots == 0) {
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
return -ENOSPC;
hash: fix lock release on add When adding items to a hash table with multiple threads, there is an spinlock used to prevent data corruption (unless Transactional Memory is supported). If there is a failure, the spinlock should be released, but there were cases where that was not happening. Fixes: be856325cba3 ("hash: add scalable multi-writer insertion with Intel TSX") Cc: stable@dpdk.org Signed-off-by: Mike Stolarchuk <mike.stolarchuk@bigswitch.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2017-07-07 05:54:25 +00:00
}
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
cached_free_slots->len += n_slots;
}
/* Get a free slot from the local cache */
cached_free_slots->len--;
slot_id = cached_free_slots->objs[cached_free_slots->len];
} else {
hash: fix lock release on add When adding items to a hash table with multiple threads, there is an spinlock used to prevent data corruption (unless Transactional Memory is supported). If there is a failure, the spinlock should be released, but there were cases where that was not happening. Fixes: be856325cba3 ("hash: add scalable multi-writer insertion with Intel TSX") Cc: stable@dpdk.org Signed-off-by: Mike Stolarchuk <mike.stolarchuk@bigswitch.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2017-07-07 05:54:25 +00:00
if (rte_ring_sc_dequeue(h->free_slots, &slot_id) != 0) {
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
return -ENOSPC;
hash: fix lock release on add When adding items to a hash table with multiple threads, there is an spinlock used to prevent data corruption (unless Transactional Memory is supported). If there is a failure, the spinlock should be released, but there were cases where that was not happening. Fixes: be856325cba3 ("hash: add scalable multi-writer insertion with Intel TSX") Cc: stable@dpdk.org Signed-off-by: Mike Stolarchuk <mike.stolarchuk@bigswitch.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2017-07-07 05:54:25 +00:00
}
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
new_k = RTE_PTR_ADD(keys, (uintptr_t)slot_id * h->key_entry_size);
new_idx = (uint32_t)((uintptr_t) slot_id);
/* Copy key */
hash: fix TSX aborts with newer gcc gcc 7 and 8 with O3 will generate vzeroupper from rte_memcpy into TSX region which may abort the TSX transaction. This fix changes rte_memcpy to memcpy which will not insert extra vzeroupper into the library. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
2018-11-12 10:47:16 +00:00
memcpy(new_k->key, key, h->key_len);
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
/* Key can be of arbitrary length, so it is not possible to store
* it atomically. Hence the new key element's memory stores
* (key as well as data) should be complete before it is referenced.
* 'pdata' acts as the synchronization point when an existing hash
* entry is updated.
*/
__atomic_store_n(&new_k->pdata,
data,
__ATOMIC_RELEASE);
hash: add scalable multi-writer insertion with Intel TSX This patch introduced scalable multi-writer Cuckoo Hash insertion based on a split Cuckoo Search and Move operation using Intel TSX. It can do scalable hash insertion with 22 cores with little performance loss and negligible TSX abortion rate. * Added an extra rte_hash flag definition to switch default single writer Cuckoo Hash behavior to multiwriter. - If HTM is available, it would use hardware feature for concurrency. - If HTM is not available, it would fall back to spinlock. * Created a rte_cuckoo_hash_x86.h file to hold all x86-arch related cuckoo_hash functions. And rte_cuckoo_hash.c uses compile time flag to select x86 file or other platform-specific implementations. While HTM check is still done at runtime (same idea with RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT) * Moved rte_hash private struct definitions to rte_cuckoo_hash.h, to allow rte_cuckoo_hash_x86.h or future platform dependent functions to include. * Following new functions are created for consistent names when new platform TM support are added. - rte_hash_cuckoo_move_insert_mw_tm: do insertion with bucket movement. - rte_hash_cuckoo_insert_mw_tm: do insertion without bucket movement. * One extra multi-writer test case is added. Signed-off-by: Wei Shen <wei1.shen@intel.com> Signed-off-by: Sameh Gobriel <sameh.gobriel@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-06-16 22:14:14 +00:00
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
/* Find an empty slot and insert */
ret = rte_hash_cuckoo_insert_mw(h, prim_bkt, sec_bkt, key, data,
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
short_sig, new_idx, &ret_val);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
if (ret == 0)
return new_idx - 1;
else if (ret == 1) {
enqueue_slot_back(h, cached_free_slots, slot_id);
return ret_val;
}
hash: add scalable multi-writer insertion with Intel TSX This patch introduced scalable multi-writer Cuckoo Hash insertion based on a split Cuckoo Search and Move operation using Intel TSX. It can do scalable hash insertion with 22 cores with little performance loss and negligible TSX abortion rate. * Added an extra rte_hash flag definition to switch default single writer Cuckoo Hash behavior to multiwriter. - If HTM is available, it would use hardware feature for concurrency. - If HTM is not available, it would fall back to spinlock. * Created a rte_cuckoo_hash_x86.h file to hold all x86-arch related cuckoo_hash functions. And rte_cuckoo_hash.c uses compile time flag to select x86 file or other platform-specific implementations. While HTM check is still done at runtime (same idea with RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT) * Moved rte_hash private struct definitions to rte_cuckoo_hash.h, to allow rte_cuckoo_hash_x86.h or future platform dependent functions to include. * Following new functions are created for consistent names when new platform TM support are added. - rte_hash_cuckoo_move_insert_mw_tm: do insertion with bucket movement. - rte_hash_cuckoo_insert_mw_tm: do insertion without bucket movement. * One extra multi-writer test case is added. Signed-off-by: Wei Shen <wei1.shen@intel.com> Signed-off-by: Sameh Gobriel <sameh.gobriel@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-06-16 22:14:14 +00:00
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
/* Primary bucket full, need to make space for new entry */
ret = rte_hash_cuckoo_make_space_mw(h, prim_bkt, sec_bkt, key, data,
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
short_sig, prim_bucket_idx, new_idx, &ret_val);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
if (ret == 0)
return new_idx - 1;
else if (ret == 1) {
enqueue_slot_back(h, cached_free_slots, slot_id);
return ret_val;
}
hash: add scalable multi-writer insertion with Intel TSX This patch introduced scalable multi-writer Cuckoo Hash insertion based on a split Cuckoo Search and Move operation using Intel TSX. It can do scalable hash insertion with 22 cores with little performance loss and negligible TSX abortion rate. * Added an extra rte_hash flag definition to switch default single writer Cuckoo Hash behavior to multiwriter. - If HTM is available, it would use hardware feature for concurrency. - If HTM is not available, it would fall back to spinlock. * Created a rte_cuckoo_hash_x86.h file to hold all x86-arch related cuckoo_hash functions. And rte_cuckoo_hash.c uses compile time flag to select x86 file or other platform-specific implementations. While HTM check is still done at runtime (same idea with RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT) * Moved rte_hash private struct definitions to rte_cuckoo_hash.h, to allow rte_cuckoo_hash_x86.h or future platform dependent functions to include. * Following new functions are created for consistent names when new platform TM support are added. - rte_hash_cuckoo_move_insert_mw_tm: do insertion with bucket movement. - rte_hash_cuckoo_insert_mw_tm: do insertion without bucket movement. * One extra multi-writer test case is added. Signed-off-by: Wei Shen <wei1.shen@intel.com> Signed-off-by: Sameh Gobriel <sameh.gobriel@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-06-16 22:14:14 +00:00
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
/* Also search secondary bucket to get better occupancy */
ret = rte_hash_cuckoo_make_space_mw(h, sec_bkt, prim_bkt, key, data,
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
short_sig, sec_bucket_idx, new_idx, &ret_val);
hash: add scalable multi-writer insertion with Intel TSX This patch introduced scalable multi-writer Cuckoo Hash insertion based on a split Cuckoo Search and Move operation using Intel TSX. It can do scalable hash insertion with 22 cores with little performance loss and negligible TSX abortion rate. * Added an extra rte_hash flag definition to switch default single writer Cuckoo Hash behavior to multiwriter. - If HTM is available, it would use hardware feature for concurrency. - If HTM is not available, it would fall back to spinlock. * Created a rte_cuckoo_hash_x86.h file to hold all x86-arch related cuckoo_hash functions. And rte_cuckoo_hash.c uses compile time flag to select x86 file or other platform-specific implementations. While HTM check is still done at runtime (same idea with RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT) * Moved rte_hash private struct definitions to rte_cuckoo_hash.h, to allow rte_cuckoo_hash_x86.h or future platform dependent functions to include. * Following new functions are created for consistent names when new platform TM support are added. - rte_hash_cuckoo_move_insert_mw_tm: do insertion with bucket movement. - rte_hash_cuckoo_insert_mw_tm: do insertion without bucket movement. * One extra multi-writer test case is added. Signed-off-by: Wei Shen <wei1.shen@intel.com> Signed-off-by: Sameh Gobriel <sameh.gobriel@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-06-16 22:14:14 +00:00
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
if (ret == 0)
return new_idx - 1;
else if (ret == 1) {
enqueue_slot_back(h, cached_free_slots, slot_id);
return ret_val;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
}
/* if ext table not enabled, we failed the insertion */
if (!h->ext_table_support) {
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
enqueue_slot_back(h, cached_free_slots, slot_id);
return ret;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
/* Now we need to go through the extendable bucket. Protection is needed
* to protect all extendable bucket processes.
*/
__hash_rw_writer_lock(h);
/* We check for duplicates again since could be inserted before the lock */
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
ret = search_and_update(h, data, key, prim_bkt, short_sig);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
if (ret != -1) {
enqueue_slot_back(h, cached_free_slots, slot_id);
goto failure;
}
FOR_EACH_BUCKET(cur_bkt, sec_bkt) {
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
ret = search_and_update(h, data, key, cur_bkt, short_sig);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
if (ret != -1) {
enqueue_slot_back(h, cached_free_slots, slot_id);
goto failure;
}
}
/* Search sec and ext buckets to find an empty entry to insert. */
FOR_EACH_BUCKET(cur_bkt, sec_bkt) {
for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) {
/* Check if slot is available */
if (likely(cur_bkt->key_idx[i] == EMPTY_SLOT)) {
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
cur_bkt->sig_current[i] = short_sig;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
cur_bkt->key_idx[i] = new_idx;
__hash_rw_writer_unlock(h);
return new_idx - 1;
}
}
}
/* Failed to get an empty entry from extendable buckets. Link a new
* extendable bucket. We first get a free bucket from ring.
*/
if (rte_ring_sc_dequeue(h->free_ext_bkts, &ext_bkt_id) != 0) {
ret = -ENOSPC;
goto failure;
}
bkt_id = (uint32_t)((uintptr_t)ext_bkt_id) - 1;
/* Use the first location of the new bucket */
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
(h->buckets_ext[bkt_id]).sig_current[0] = short_sig;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
(h->buckets_ext[bkt_id]).key_idx[0] = new_idx;
/* Link the new bucket to sec bucket linked list */
last = rte_hash_get_last_bkt(sec_bkt);
last->next = &h->buckets_ext[bkt_id];
__hash_rw_writer_unlock(h);
return new_idx - 1;
failure:
__hash_rw_writer_unlock(h);
return ret;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
int32_t
rte_hash_add_key_with_hash(const struct rte_hash *h,
const void *key, hash_sig_t sig)
{
RETURN_IF_TRUE(((h == NULL) || (key == NULL)), -EINVAL);
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
return __rte_hash_add_key_with_hash(h, key, sig, 0);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
int32_t
rte_hash_add_key(const struct rte_hash *h, const void *key)
{
RETURN_IF_TRUE(((h == NULL) || (key == NULL)), -EINVAL);
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
return __rte_hash_add_key_with_hash(h, key, rte_hash_hash(h, key), 0);
}
int
rte_hash_add_key_with_hash_data(const struct rte_hash *h,
const void *key, hash_sig_t sig, void *data)
{
int ret;
RETURN_IF_TRUE(((h == NULL) || (key == NULL)), -EINVAL);
ret = __rte_hash_add_key_with_hash(h, key, sig, data);
if (ret >= 0)
return 0;
else
return ret;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
int
rte_hash_add_key_data(const struct rte_hash *h, const void *key, void *data)
{
int ret;
RETURN_IF_TRUE(((h == NULL) || (key == NULL)), -EINVAL);
ret = __rte_hash_add_key_with_hash(h, key, rte_hash_hash(h, key), data);
if (ret >= 0)
return 0;
else
return ret;
}
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
/* Search one bucket to find the match key - uses rw lock */
static inline int32_t
search_one_bucket_l(const struct rte_hash *h, const void *key,
uint16_t sig, void **data,
const struct rte_hash_bucket *bkt)
{
int i;
struct rte_hash_key *k, *keys = h->key_store;
for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) {
if (bkt->sig_current[i] == sig &&
bkt->key_idx[i] != EMPTY_SLOT) {
k = (struct rte_hash_key *) ((char *)keys +
bkt->key_idx[i] * h->key_entry_size);
if (rte_hash_cmp_eq(key, k->key, h) == 0) {
if (data != NULL)
*data = k->pdata;
/*
* Return index where key is stored,
* subtracting the first dummy index
*/
return bkt->key_idx[i] - 1;
}
}
}
return -1;
}
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
/* Search one bucket to find the match key */
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
static inline int32_t
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
search_one_bucket_lf(const struct rte_hash *h, const void *key, uint16_t sig,
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
void **data, const struct rte_hash_bucket *bkt)
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
{
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
int i;
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
uint32_t key_idx;
void *pdata;
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
struct rte_hash_key *k, *keys = h->key_store;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) {
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
key_idx = __atomic_load_n(&bkt->key_idx[i],
__ATOMIC_ACQUIRE);
if (bkt->sig_current[i] == sig && key_idx != EMPTY_SLOT) {
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
k = (struct rte_hash_key *) ((char *)keys +
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
key_idx * h->key_entry_size);
pdata = __atomic_load_n(&k->pdata,
__ATOMIC_ACQUIRE);
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
if (rte_hash_cmp_eq(key, k->key, h) == 0) {
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
if (data != NULL)
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
*data = pdata;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
/*
* Return index where key is stored,
fix typos using codespell utility Fixing typos across dpdk source code using codespell utility. Skipped the ethdev driver's base code fixes to keep the base code intact. Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Acked-by: John McNamara <john.mcnamara@intel.com>
2017-06-07 05:05:06 +00:00
* subtracting the first dummy index
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
*/
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
return key_idx - 1;
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
}
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
return -1;
}
static inline int32_t
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
__rte_hash_lookup_with_hash_l(const struct rte_hash *h, const void *key,
hash_sig_t sig, void **data)
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
{
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
uint32_t prim_bucket_idx, sec_bucket_idx;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
struct rte_hash_bucket *bkt, *cur_bkt;
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
int ret;
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
uint16_t short_sig;
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
short_sig = get_short_sig(sig);
prim_bucket_idx = get_prim_bucket_index(h, sig);
sec_bucket_idx = get_alt_bucket_index(h, prim_bucket_idx, short_sig);
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
bkt = &h->buckets[prim_bucket_idx];
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
__hash_rw_reader_lock(h);
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
/* Check if key is in primary location */
ret = search_one_bucket_l(h, key, short_sig, data, bkt);
if (ret != -1) {
__hash_rw_reader_unlock(h);
return ret;
}
/* Calculate secondary hash */
bkt = &h->buckets[sec_bucket_idx];
/* Check if key is in secondary location */
FOR_EACH_BUCKET(cur_bkt, bkt) {
ret = search_one_bucket_l(h, key, short_sig,
data, cur_bkt);
if (ret != -1) {
__hash_rw_reader_unlock(h);
return ret;
}
}
__hash_rw_reader_unlock(h);
return -ENOENT;
}
static inline int32_t
__rte_hash_lookup_with_hash_lf(const struct rte_hash *h, const void *key,
hash_sig_t sig, void **data)
{
uint32_t prim_bucket_idx, sec_bucket_idx;
struct rte_hash_bucket *bkt, *cur_bkt;
uint32_t cnt_b, cnt_a;
int ret;
uint16_t short_sig;
short_sig = get_short_sig(sig);
prim_bucket_idx = get_prim_bucket_index(h, sig);
sec_bucket_idx = get_alt_bucket_index(h, prim_bucket_idx, short_sig);
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
do {
/* Load the table change counter before the lookup
* starts. Acquire semantics will make sure that
* loads in search_one_bucket are not hoisted.
*/
cnt_b = __atomic_load_n(h->tbl_chng_cnt,
__ATOMIC_ACQUIRE);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
/* Check if key is in primary location */
bkt = &h->buckets[prim_bucket_idx];
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
ret = search_one_bucket_lf(h, key, short_sig, data, bkt);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
if (ret != -1) {
__hash_rw_reader_unlock(h);
return ret;
}
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
/* Calculate secondary hash */
bkt = &h->buckets[sec_bucket_idx];
/* Check if key is in secondary location */
FOR_EACH_BUCKET(cur_bkt, bkt) {
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
ret = search_one_bucket_lf(h, key, short_sig,
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
data, cur_bkt);
if (ret != -1) {
__hash_rw_reader_unlock(h);
return ret;
}
}
/* The loads of sig_current in search_one_bucket
* should not move below the load from tbl_chng_cnt.
*/
__atomic_thread_fence(__ATOMIC_ACQUIRE);
/* Re-read the table change counter to check if the
* table has changed during search. If yes, re-do
* the search.
* This load should not get hoisted. The load
* acquires on cnt_b, key index in primary bucket
* and key index in secondary bucket will make sure
* that it does not get hoisted.
*/
cnt_a = __atomic_load_n(h->tbl_chng_cnt,
__ATOMIC_ACQUIRE);
} while (cnt_b != cnt_a);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
return -ENOENT;
}
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
static inline int32_t
__rte_hash_lookup_with_hash(const struct rte_hash *h, const void *key,
hash_sig_t sig, void **data)
{
if (h->readwrite_concur_lf_support)
return __rte_hash_lookup_with_hash_lf(h, key, sig, data);
else
return __rte_hash_lookup_with_hash_l(h, key, sig, data);
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
int32_t
rte_hash_lookup_with_hash(const struct rte_hash *h,
const void *key, hash_sig_t sig)
{
RETURN_IF_TRUE(((h == NULL) || (key == NULL)), -EINVAL);
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
return __rte_hash_lookup_with_hash(h, key, sig, NULL);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
int32_t
rte_hash_lookup(const struct rte_hash *h, const void *key)
{
RETURN_IF_TRUE(((h == NULL) || (key == NULL)), -EINVAL);
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key), NULL);
}
int
rte_hash_lookup_with_hash_data(const struct rte_hash *h,
const void *key, hash_sig_t sig, void **data)
{
RETURN_IF_TRUE(((h == NULL) || (key == NULL)), -EINVAL);
return __rte_hash_lookup_with_hash(h, key, sig, data);
}
int
rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)
{
RETURN_IF_TRUE(((h == NULL) || (key == NULL)), -EINVAL);
return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key), data);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
static inline void
remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
{
unsigned lcore_id, n_slots;
struct lcore_cache *cached_free_slots;
hash: separate multi-writer from r/w concurrency RW concurrency is required with single writer and multiple reader usecase as well. Hence, multi-writer should not be enabled by default when RW concurrency is enabled. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:29 +00:00
if (h->use_local_cache) {
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
lcore_id = rte_lcore_id();
cached_free_slots = &h->local_free_slots[lcore_id];
/* Cache full, need to free it. */
if (cached_free_slots->len == LCORE_CACHE_SIZE) {
/* Need to enqueue the free slots in global ring. */
n_slots = rte_ring_mp_enqueue_burst(h->free_slots,
cached_free_slots->objs,
ring: return free space when enqueuing Add an extra parameter to the ring enqueue burst/bulk functions so that those functions can optionally return the amount of free space in the ring. This information can be used by applications in a number of ways, for instance, with single-producer queues, it provides a max enqueue size which is guaranteed to work. It can also be used to implement watermark functionality in apps, replacing the older functionality with a more flexible version, which enables apps to implement multiple watermark thresholds, rather than just one. Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Yuanhan Liu <yuanhan.liu@linux.intel.com> Acked-by: Olivier Matz <olivier.matz@6wind.com>
2017-03-29 15:21:24 +00:00
LCORE_CACHE_SIZE, NULL);
hash: fix out-of-bound write while freeing key slot Add a debug check for out-of-bound write while freeing the key slot. Coverity issue: 325733 Fixes: e605a1d36ca7 ("hash: add lock-free r/w concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-11-22 02:51:56 +00:00
ERR_IF_TRUE((n_slots == 0),
"%s: could not enqueue free slots in global ring\n",
__func__);
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
cached_free_slots->len -= n_slots;
}
/* Put index of new free slot in cache. */
cached_free_slots->objs[cached_free_slots->len] =
(void *)((uintptr_t)bkt->key_idx[i]);
cached_free_slots->len++;
} else {
rte_ring_sp_enqueue(h->free_slots,
(void *)((uintptr_t)bkt->key_idx[i]));
}
}
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
/* Compact the linked list by moving key from last entry in linked list to the
* empty slot.
*/
static inline void
__rte_hash_compact_ll(struct rte_hash_bucket *cur_bkt, int pos) {
int i;
struct rte_hash_bucket *last_bkt;
if (!cur_bkt->next)
return;
last_bkt = rte_hash_get_last_bkt(cur_bkt);
for (i = RTE_HASH_BUCKET_ENTRIES - 1; i >= 0; i--) {
if (last_bkt->key_idx[i] != EMPTY_SLOT) {
cur_bkt->key_idx[pos] = last_bkt->key_idx[i];
cur_bkt->sig_current[pos] = last_bkt->sig_current[i];
last_bkt->sig_current[i] = NULL_SIGNATURE;
last_bkt->key_idx[i] = EMPTY_SLOT;
return;
}
}
}
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
/* Search one bucket and remove the matched key.
* Writer is expected to hold the lock while calling this
* function.
*/
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
static inline int32_t
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
search_and_remove(const struct rte_hash *h, const void *key,
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
struct rte_hash_bucket *bkt, uint16_t sig, int *pos)
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
{
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
struct rte_hash_key *k, *keys = h->key_store;
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
unsigned int i;
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
uint32_t key_idx;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
/* Check if key is in bucket */
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) {
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
key_idx = __atomic_load_n(&bkt->key_idx[i],
__ATOMIC_ACQUIRE);
if (bkt->sig_current[i] == sig && key_idx != EMPTY_SLOT) {
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
k = (struct rte_hash_key *) ((char *)keys +
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
key_idx * h->key_entry_size);
hash: fix multi-process support Hash library used a function pointer to choose a different key compare function, depending on the key size. As a result, multiple processes could not use the same hash table, as the function addresses vary from one process to another. Instead, a jump table is used, so each process has its own function addresses, accessing this table with an index stored in the hash table (note that using a custom key compare function is not supported in multi-process mode). Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2016-04-01 15:03:49 +00:00
if (rte_hash_cmp_eq(key, k->key, h) == 0) {
hash: support no free on delete rte_hash_lookup_xxx APIs return the index of slot in the key store. Application(reader) can use that index to reference other data structures in its scope. Because of this, the index should not be freed till the application completes using the index. RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is introduced to support this. When this flag is enabled rte_hash_del_xxx APIs do not free the key-store index/internal memory associated with the deleted entry. The new API rte_hash_free_key_with_position should be called to free the key-store index/internal memory after calling rte_hash_del_xxx APIs. Suggested-by: Yipeng Wang <yipeng1.wang@intel.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:30 +00:00
bkt->sig_current[i] = NULL_SIGNATURE;
/* Free the key store index if
* no_free_on_del is disabled.
*/
if (!h->no_free_on_del)
remove_entry(h, bkt, i);
hash: fix scaling by reducing contention If using multiple cores on a system with hardware transactional memory support, thread scaling does not work, as there was a single point in the hash library which is a bottleneck for all threads, which is the "free_slots" ring, which stores all the indices of the free slots in the table. This patch fixes the problem, by creating a local cache per logical core, which stores locally indices of free slots, so most times, writer threads will not interfere each other. Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation") Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-10-30 14:37:28 +00:00
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
__atomic_store_n(&bkt->key_idx[i],
EMPTY_SLOT,
__ATOMIC_RELEASE);
*pos = i;
/*
* Return index where key is stored,
fix typos using codespell utility Fixing typos across dpdk source code using codespell utility. Skipped the ethdev driver's base code fixes to keep the base code intact. Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Acked-by: John McNamara <john.mcnamara@intel.com>
2017-06-07 05:05:06 +00:00
* subtracting the first dummy index
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
*/
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
return key_idx - 1;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
}
}
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
return -1;
}
static inline int32_t
__rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
hash_sig_t sig)
{
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
uint32_t prim_bucket_idx, sec_bucket_idx;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
struct rte_hash_bucket *prim_bkt, *sec_bkt, *prev_bkt, *last_bkt;
struct rte_hash_bucket *cur_bkt;
int pos;
int32_t ret, i;
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
uint16_t short_sig;
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
short_sig = get_short_sig(sig);
prim_bucket_idx = get_prim_bucket_index(h, sig);
sec_bucket_idx = get_alt_bucket_index(h, prim_bucket_idx, short_sig);
prim_bkt = &h->buckets[prim_bucket_idx];
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
__hash_rw_writer_lock(h);
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
/* look for key in primary bucket */
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
ret = search_and_remove(h, key, prim_bkt, short_sig, &pos);
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
if (ret != -1) {
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
__rte_hash_compact_ll(prim_bkt, pos);
last_bkt = prim_bkt->next;
prev_bkt = prim_bkt;
goto return_bkt;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
/* Calculate secondary hash */
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
sec_bkt = &h->buckets[sec_bucket_idx];
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
FOR_EACH_BUCKET(cur_bkt, sec_bkt) {
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
ret = search_and_remove(h, key, cur_bkt, short_sig, &pos);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
if (ret != -1) {
__rte_hash_compact_ll(cur_bkt, pos);
last_bkt = sec_bkt->next;
prev_bkt = sec_bkt;
goto return_bkt;
}
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
__hash_rw_writer_unlock(h);
return -ENOENT;
/* Search last bucket to see if empty to be recycled */
return_bkt:
if (!last_bkt) {
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
__hash_rw_writer_unlock(h);
hash: move duplicated code into functions This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:57 +00:00
return ret;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
}
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
while (last_bkt->next) {
prev_bkt = last_bkt;
last_bkt = last_bkt->next;
}
for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) {
if (last_bkt->key_idx[i] != EMPTY_SLOT)
break;
}
/* found empty bucket and recycle */
if (i == RTE_HASH_BUCKET_ENTRIES) {
prev_bkt->next = last_bkt->next = NULL;
uint32_t index = last_bkt - h->buckets_ext + 1;
rte_ring_sp_enqueue(h->free_ext_bkts, (void *)(uintptr_t)index);
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
__hash_rw_writer_unlock(h);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
return ret;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
int32_t
rte_hash_del_key_with_hash(const struct rte_hash *h,
const void *key, hash_sig_t sig)
{
RETURN_IF_TRUE(((h == NULL) || (key == NULL)), -EINVAL);
return __rte_hash_del_key_with_hash(h, key, sig);
}
int32_t
rte_hash_del_key(const struct rte_hash *h, const void *key)
{
RETURN_IF_TRUE(((h == NULL) || (key == NULL)), -EINVAL);
return __rte_hash_del_key_with_hash(h, key, rte_hash_hash(h, key));
}
hash: retrieve a key given its position The function rte_hash_get_key_with_position is added in this patch. As the position returned when adding a key is frequently used as an offset into an array of user data, this function performs the operation of retrieving a key given this offset. A possible use case would be to delete a key from the hash table when its entry in the array of data has certain value. For instance, the key could be a flow 5-tuple, and the value stored in the array a time stamp. Signed-off-by: Juan Antonio Montesinos <juan.antonio.montesinos.delgado@ericsson.com> Signed-off-by: Yari Adan Petralanda <yari.adan.petralanda@ericsson.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2016-07-04 08:59:52 +00:00
int
rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position,
void **key)
{
RETURN_IF_TRUE(((h == NULL) || (key == NULL)), -EINVAL);
struct rte_hash_key *k, *keys = h->key_store;
k = (struct rte_hash_key *) ((char *) keys + (position + 1) *
h->key_entry_size);
*key = k->key;
if (position !=
__rte_hash_lookup_with_hash(h, *key, rte_hash_hash(h, *key),
NULL)) {
return -ENOENT;
}
return 0;
}
hash: support no free on delete rte_hash_lookup_xxx APIs return the index of slot in the key store. Application(reader) can use that index to reference other data structures in its scope. Because of this, the index should not be freed till the application completes using the index. RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is introduced to support this. When this flag is enabled rte_hash_del_xxx APIs do not free the key-store index/internal memory associated with the deleted entry. The new API rte_hash_free_key_with_position should be called to free the key-store index/internal memory after calling rte_hash_del_xxx APIs. Suggested-by: Yipeng Wang <yipeng1.wang@intel.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:30 +00:00
int __rte_experimental
rte_hash_free_key_with_position(const struct rte_hash *h,
const int32_t position)
{
RETURN_IF_TRUE(((h == NULL) || (position == EMPTY_SLOT)), -EINVAL);
unsigned int lcore_id, n_slots;
struct lcore_cache *cached_free_slots;
const int32_t total_entries = h->num_buckets * RTE_HASH_BUCKET_ENTRIES;
/* Out of bounds */
if (position >= total_entries)
return -EINVAL;
if (h->use_local_cache) {
lcore_id = rte_lcore_id();
cached_free_slots = &h->local_free_slots[lcore_id];
/* Cache full, need to free it. */
if (cached_free_slots->len == LCORE_CACHE_SIZE) {
/* Need to enqueue the free slots in global ring. */
n_slots = rte_ring_mp_enqueue_burst(h->free_slots,
cached_free_slots->objs,
LCORE_CACHE_SIZE, NULL);
hash: fix out-of-bound write while freeing key slot Add a debug check for out-of-bound write while freeing the key slot. Coverity issue: 325733 Fixes: e605a1d36ca7 ("hash: add lock-free r/w concurrency") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-11-22 02:51:56 +00:00
RETURN_IF_TRUE((n_slots == 0), -EFAULT);
hash: support no free on delete rte_hash_lookup_xxx APIs return the index of slot in the key store. Application(reader) can use that index to reference other data structures in its scope. Because of this, the index should not be freed till the application completes using the index. RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is introduced to support this. When this flag is enabled rte_hash_del_xxx APIs do not free the key-store index/internal memory associated with the deleted entry. The new API rte_hash_free_key_with_position should be called to free the key-store index/internal memory after calling rte_hash_del_xxx APIs. Suggested-by: Yipeng Wang <yipeng1.wang@intel.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:30 +00:00
cached_free_slots->len -= n_slots;
}
/* Put index of new free slot in cache. */
cached_free_slots->objs[cached_free_slots->len] =
(void *)((uintptr_t)position);
cached_free_slots->len++;
} else {
rte_ring_sp_enqueue(h->free_slots,
(void *)((uintptr_t)position));
}
return 0;
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
static inline void
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
compare_signatures(uint32_t *prim_hash_matches, uint32_t *sec_hash_matches,
hash: add vectorized comparison In lookup bulk function, the signatures of all entries are compared against the signature of the key that is being looked up. Now that all the signatures are together, they can be compared with vector instructions (SSE, AVX2), achieving higher lookup performance. Also, entries per bucket are increased to 8 when using processors with AVX2, as 256 bits can be compared at once, which is the size of 8x32-bit signatures. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:14 +00:00
const struct rte_hash_bucket *prim_bkt,
const struct rte_hash_bucket *sec_bkt,
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
uint16_t sig,
hash: add vectorized comparison In lookup bulk function, the signatures of all entries are compared against the signature of the key that is being looked up. Now that all the signatures are together, they can be compared with vector instructions (SSE, AVX2), achieving higher lookup performance. Also, entries per bucket are increased to 8 when using processors with AVX2, as 256 bits can be compared at once, which is the size of 8x32-bit signatures. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:14 +00:00
enum rte_hash_sig_compare_function sig_cmp_fn)
{
unsigned int i;
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
/* For match mask the first bit of every two bits indicates the match */
hash: add vectorized comparison In lookup bulk function, the signatures of all entries are compared against the signature of the key that is being looked up. Now that all the signatures are together, they can be compared with vector instructions (SSE, AVX2), achieving higher lookup performance. Also, entries per bucket are increased to 8 when using processors with AVX2, as 256 bits can be compared at once, which is the size of 8x32-bit signatures. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:14 +00:00
switch (sig_cmp_fn) {
#ifdef RTE_MACHINE_CPUFLAG_SSE2
case RTE_HASH_COMPARE_SSE:
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
/* Compare all signatures in the bucket */
*prim_hash_matches = _mm_movemask_epi8(_mm_cmpeq_epi16(
hash: add vectorized comparison In lookup bulk function, the signatures of all entries are compared against the signature of the key that is being looked up. Now that all the signatures are together, they can be compared with vector instructions (SSE, AVX2), achieving higher lookup performance. Also, entries per bucket are increased to 8 when using processors with AVX2, as 256 bits can be compared at once, which is the size of 8x32-bit signatures. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:14 +00:00
_mm_load_si128(
(__m128i const *)prim_bkt->sig_current),
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
_mm_set1_epi16(sig)));
/* Compare all signatures in the bucket */
*sec_hash_matches = _mm_movemask_epi8(_mm_cmpeq_epi16(
hash: add vectorized comparison In lookup bulk function, the signatures of all entries are compared against the signature of the key that is being looked up. Now that all the signatures are together, they can be compared with vector instructions (SSE, AVX2), achieving higher lookup performance. Also, entries per bucket are increased to 8 when using processors with AVX2, as 256 bits can be compared at once, which is the size of 8x32-bit signatures. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:14 +00:00
_mm_load_si128(
(__m128i const *)sec_bkt->sig_current),
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
_mm_set1_epi16(sig)));
hash: add vectorized comparison In lookup bulk function, the signatures of all entries are compared against the signature of the key that is being looked up. Now that all the signatures are together, they can be compared with vector instructions (SSE, AVX2), achieving higher lookup performance. Also, entries per bucket are increased to 8 when using processors with AVX2, as 256 bits can be compared at once, which is the size of 8x32-bit signatures. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:14 +00:00
break;
#endif
default:
for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) {
*prim_hash_matches |=
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
((sig == prim_bkt->sig_current[i]) << (i << 1));
hash: add vectorized comparison In lookup bulk function, the signatures of all entries are compared against the signature of the key that is being looked up. Now that all the signatures are together, they can be compared with vector instructions (SSE, AVX2), achieving higher lookup performance. Also, entries per bucket are increased to 8 when using processors with AVX2, as 256 bits can be compared at once, which is the size of 8x32-bit signatures. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:14 +00:00
*sec_hash_matches |=
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
((sig == sec_bkt->sig_current[i]) << (i << 1));
hash: add vectorized comparison In lookup bulk function, the signatures of all entries are compared against the signature of the key that is being looked up. Now that all the signatures are together, they can be compared with vector instructions (SSE, AVX2), achieving higher lookup performance. Also, entries per bucket are increased to 8 when using processors with AVX2, as 256 bits can be compared at once, which is the size of 8x32-bit signatures. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:14 +00:00
}
}
}
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
#define PREFETCH_OFFSET 4
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
static inline void
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
__rte_hash_lookup_bulk_l(const struct rte_hash *h, const void **keys,
int32_t num_keys, int32_t *positions,
uint64_t *hit_mask, void *data[])
{
uint64_t hits = 0;
int32_t i;
int32_t ret;
uint32_t prim_hash[RTE_HASH_LOOKUP_BULK_MAX];
uint32_t prim_index[RTE_HASH_LOOKUP_BULK_MAX];
uint32_t sec_index[RTE_HASH_LOOKUP_BULK_MAX];
uint16_t sig[RTE_HASH_LOOKUP_BULK_MAX];
const struct rte_hash_bucket *primary_bkt[RTE_HASH_LOOKUP_BULK_MAX];
const struct rte_hash_bucket *secondary_bkt[RTE_HASH_LOOKUP_BULK_MAX];
uint32_t prim_hitmask[RTE_HASH_LOOKUP_BULK_MAX] = {0};
uint32_t sec_hitmask[RTE_HASH_LOOKUP_BULK_MAX] = {0};
struct rte_hash_bucket *cur_bkt, *next_bkt;
/* Prefetch first keys */
for (i = 0; i < PREFETCH_OFFSET && i < num_keys; i++)
rte_prefetch0(keys[i]);
/*
* Prefetch rest of the keys, calculate primary and
* secondary bucket and prefetch them
*/
for (i = 0; i < (num_keys - PREFETCH_OFFSET); i++) {
rte_prefetch0(keys[i + PREFETCH_OFFSET]);
prim_hash[i] = rte_hash_hash(h, keys[i]);
sig[i] = get_short_sig(prim_hash[i]);
prim_index[i] = get_prim_bucket_index(h, prim_hash[i]);
sec_index[i] = get_alt_bucket_index(h, prim_index[i], sig[i]);
primary_bkt[i] = &h->buckets[prim_index[i]];
secondary_bkt[i] = &h->buckets[sec_index[i]];
rte_prefetch0(primary_bkt[i]);
rte_prefetch0(secondary_bkt[i]);
}
/* Calculate and prefetch rest of the buckets */
for (; i < num_keys; i++) {
prim_hash[i] = rte_hash_hash(h, keys[i]);
sig[i] = get_short_sig(prim_hash[i]);
prim_index[i] = get_prim_bucket_index(h, prim_hash[i]);
sec_index[i] = get_alt_bucket_index(h, prim_index[i], sig[i]);
primary_bkt[i] = &h->buckets[prim_index[i]];
secondary_bkt[i] = &h->buckets[sec_index[i]];
rte_prefetch0(primary_bkt[i]);
rte_prefetch0(secondary_bkt[i]);
}
__hash_rw_reader_lock(h);
/* Compare signatures and prefetch key slot of first hit */
for (i = 0; i < num_keys; i++) {
compare_signatures(&prim_hitmask[i], &sec_hitmask[i],
primary_bkt[i], secondary_bkt[i],
sig[i], h->sig_cmp_fn);
if (prim_hitmask[i]) {
uint32_t first_hit =
__builtin_ctzl(prim_hitmask[i])
>> 1;
uint32_t key_idx =
primary_bkt[i]->key_idx[first_hit];
const struct rte_hash_key *key_slot =
(const struct rte_hash_key *)(
(const char *)h->key_store +
key_idx * h->key_entry_size);
rte_prefetch0(key_slot);
continue;
}
if (sec_hitmask[i]) {
uint32_t first_hit =
__builtin_ctzl(sec_hitmask[i])
>> 1;
uint32_t key_idx =
secondary_bkt[i]->key_idx[first_hit];
const struct rte_hash_key *key_slot =
(const struct rte_hash_key *)(
(const char *)h->key_store +
key_idx * h->key_entry_size);
rte_prefetch0(key_slot);
}
}
/* Compare keys, first hits in primary first */
for (i = 0; i < num_keys; i++) {
positions[i] = -ENOENT;
while (prim_hitmask[i]) {
uint32_t hit_index =
__builtin_ctzl(prim_hitmask[i])
>> 1;
uint32_t key_idx =
primary_bkt[i]->key_idx[hit_index];
const struct rte_hash_key *key_slot =
(const struct rte_hash_key *)(
(const char *)h->key_store +
key_idx * h->key_entry_size);
/*
* If key index is 0, do not compare key,
* as it is checking the dummy slot
*/
if (!!key_idx &
!rte_hash_cmp_eq(
key_slot->key, keys[i], h)) {
if (data != NULL)
data[i] = key_slot->pdata;
hits |= 1ULL << i;
positions[i] = key_idx - 1;
goto next_key;
}
prim_hitmask[i] &= ~(3ULL << (hit_index << 1));
}
while (sec_hitmask[i]) {
uint32_t hit_index =
__builtin_ctzl(sec_hitmask[i])
>> 1;
uint32_t key_idx =
secondary_bkt[i]->key_idx[hit_index];
const struct rte_hash_key *key_slot =
(const struct rte_hash_key *)(
(const char *)h->key_store +
key_idx * h->key_entry_size);
/*
* If key index is 0, do not compare key,
* as it is checking the dummy slot
*/
if (!!key_idx &
!rte_hash_cmp_eq(
key_slot->key, keys[i], h)) {
if (data != NULL)
data[i] = key_slot->pdata;
hits |= 1ULL << i;
positions[i] = key_idx - 1;
goto next_key;
}
sec_hitmask[i] &= ~(3ULL << (hit_index << 1));
}
next_key:
continue;
}
/* all found, do not need to go through ext bkt */
if ((hits == ((1ULL << num_keys) - 1)) || !h->ext_table_support) {
if (hit_mask != NULL)
*hit_mask = hits;
__hash_rw_reader_unlock(h);
return;
}
/* need to check ext buckets for match */
for (i = 0; i < num_keys; i++) {
if ((hits & (1ULL << i)) != 0)
continue;
next_bkt = secondary_bkt[i]->next;
FOR_EACH_BUCKET(cur_bkt, next_bkt) {
if (data != NULL)
ret = search_one_bucket_l(h, keys[i],
sig[i], &data[i], cur_bkt);
else
ret = search_one_bucket_l(h, keys[i],
sig[i], NULL, cur_bkt);
if (ret != -1) {
positions[i] = ret;
hits |= 1ULL << i;
break;
}
}
}
__hash_rw_reader_unlock(h);
if (hit_mask != NULL)
*hit_mask = hits;
}
static inline void
__rte_hash_lookup_bulk_lf(const struct rte_hash *h, const void **keys,
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
int32_t num_keys, int32_t *positions,
uint64_t *hit_mask, void *data[])
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
{
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
uint64_t hits = 0;
int32_t i;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
int32_t ret;
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
uint32_t prim_hash[RTE_HASH_LOOKUP_BULK_MAX];
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
uint32_t prim_index[RTE_HASH_LOOKUP_BULK_MAX];
uint32_t sec_index[RTE_HASH_LOOKUP_BULK_MAX];
uint16_t sig[RTE_HASH_LOOKUP_BULK_MAX];
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
const struct rte_hash_bucket *primary_bkt[RTE_HASH_LOOKUP_BULK_MAX];
const struct rte_hash_bucket *secondary_bkt[RTE_HASH_LOOKUP_BULK_MAX];
uint32_t prim_hitmask[RTE_HASH_LOOKUP_BULK_MAX] = {0};
uint32_t sec_hitmask[RTE_HASH_LOOKUP_BULK_MAX] = {0};
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
struct rte_hash_bucket *cur_bkt, *next_bkt;
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
void *pdata[RTE_HASH_LOOKUP_BULK_MAX];
uint32_t cnt_b, cnt_a;
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
/* Prefetch first keys */
for (i = 0; i < PREFETCH_OFFSET && i < num_keys; i++)
rte_prefetch0(keys[i]);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
/*
* Prefetch rest of the keys, calculate primary and
* secondary bucket and prefetch them
*/
for (i = 0; i < (num_keys - PREFETCH_OFFSET); i++) {
rte_prefetch0(keys[i + PREFETCH_OFFSET]);
hash: add vectorized comparison In lookup bulk function, the signatures of all entries are compared against the signature of the key that is being looked up. Now that all the signatures are together, they can be compared with vector instructions (SSE, AVX2), achieving higher lookup performance. Also, entries per bucket are increased to 8 when using processors with AVX2, as 256 bits can be compared at once, which is the size of 8x32-bit signatures. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:14 +00:00
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
prim_hash[i] = rte_hash_hash(h, keys[i]);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
sig[i] = get_short_sig(prim_hash[i]);
prim_index[i] = get_prim_bucket_index(h, prim_hash[i]);
sec_index[i] = get_alt_bucket_index(h, prim_index[i], sig[i]);
primary_bkt[i] = &h->buckets[prim_index[i]];
secondary_bkt[i] = &h->buckets[sec_index[i]];
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
rte_prefetch0(primary_bkt[i]);
rte_prefetch0(secondary_bkt[i]);
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
/* Calculate and prefetch rest of the buckets */
for (; i < num_keys; i++) {
prim_hash[i] = rte_hash_hash(h, keys[i]);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
sig[i] = get_short_sig(prim_hash[i]);
prim_index[i] = get_prim_bucket_index(h, prim_hash[i]);
sec_index[i] = get_alt_bucket_index(h, prim_index[i], sig[i]);
primary_bkt[i] = &h->buckets[prim_index[i]];
secondary_bkt[i] = &h->buckets[sec_index[i]];
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
rte_prefetch0(primary_bkt[i]);
rte_prefetch0(secondary_bkt[i]);
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
do {
/* Load the table change counter before the lookup
* starts. Acquire semantics will make sure that
* loads in compare_signatures are not hoisted.
*/
cnt_b = __atomic_load_n(h->tbl_chng_cnt,
__ATOMIC_ACQUIRE);
/* Compare signatures and prefetch key slot of first hit */
for (i = 0; i < num_keys; i++) {
compare_signatures(&prim_hitmask[i], &sec_hitmask[i],
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
primary_bkt[i], secondary_bkt[i],
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
sig[i], h->sig_cmp_fn);
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
if (prim_hitmask[i]) {
uint32_t first_hit =
__builtin_ctzl(prim_hitmask[i])
>> 1;
uint32_t key_idx =
primary_bkt[i]->key_idx[first_hit];
const struct rte_hash_key *key_slot =
(const struct rte_hash_key *)(
(const char *)h->key_store +
key_idx * h->key_entry_size);
rte_prefetch0(key_slot);
continue;
}
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
if (sec_hitmask[i]) {
uint32_t first_hit =
__builtin_ctzl(sec_hitmask[i])
>> 1;
uint32_t key_idx =
secondary_bkt[i]->key_idx[first_hit];
const struct rte_hash_key *key_slot =
(const struct rte_hash_key *)(
(const char *)h->key_store +
key_idx * h->key_entry_size);
rte_prefetch0(key_slot);
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
}
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
/* Compare keys, first hits in primary first */
for (i = 0; i < num_keys; i++) {
positions[i] = -ENOENT;
while (prim_hitmask[i]) {
uint32_t hit_index =
__builtin_ctzl(prim_hitmask[i])
>> 1;
uint32_t key_idx =
__atomic_load_n(
&primary_bkt[i]->key_idx[hit_index],
__ATOMIC_ACQUIRE);
const struct rte_hash_key *key_slot =
(const struct rte_hash_key *)(
(const char *)h->key_store +
key_idx * h->key_entry_size);
if (key_idx != EMPTY_SLOT)
pdata[i] = __atomic_load_n(
&key_slot->pdata,
__ATOMIC_ACQUIRE);
/*
* If key index is 0, do not compare key,
* as it is checking the dummy slot
*/
if (!!key_idx &
!rte_hash_cmp_eq(
key_slot->key, keys[i], h)) {
if (data != NULL)
data[i] = pdata[i];
hits |= 1ULL << i;
positions[i] = key_idx - 1;
goto next_key;
}
prim_hitmask[i] &= ~(3ULL << (hit_index << 1));
}
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
while (sec_hitmask[i]) {
uint32_t hit_index =
__builtin_ctzl(sec_hitmask[i])
>> 1;
uint32_t key_idx =
__atomic_load_n(
&secondary_bkt[i]->key_idx[hit_index],
__ATOMIC_ACQUIRE);
const struct rte_hash_key *key_slot =
(const struct rte_hash_key *)(
(const char *)h->key_store +
key_idx * h->key_entry_size);
if (key_idx != EMPTY_SLOT)
pdata[i] = __atomic_load_n(
&key_slot->pdata,
__ATOMIC_ACQUIRE);
/*
* If key index is 0, do not compare key,
* as it is checking the dummy slot
*/
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
if (!!key_idx &
!rte_hash_cmp_eq(
key_slot->key, keys[i], h)) {
if (data != NULL)
data[i] = pdata[i];
hits |= 1ULL << i;
positions[i] = key_idx - 1;
goto next_key;
}
sec_hitmask[i] &= ~(3ULL << (hit_index << 1));
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
}
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
next_key:
continue;
hash: modify lookup bulk pipeline This patch replaces the pipelined rte_hash lookup mechanism with a loop-and-jump model, which performs significantly better, especially for smaller table sizes and smaller table occupancies. Signed-off-by: Byron Marohn <byron.marohn@intel.com> Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Sameh Gobriel <sameh.gobriel@intel.com>
2016-10-04 23:25:15 +00:00
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
/* The loads of sig_current in compare_signatures
* should not move below the load from tbl_chng_cnt.
*/
__atomic_thread_fence(__ATOMIC_ACQUIRE);
/* Re-read the table change counter to check if the
* table has changed during search. If yes, re-do
* the search.
* This load should not get hoisted. The load
* acquires on cnt_b, primary key index and secondary
* key index will make sure that it does not get
* hoisted.
*/
cnt_a = __atomic_load_n(h->tbl_chng_cnt,
__ATOMIC_ACQUIRE);
} while (cnt_b != cnt_a);
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
/* all found, do not need to go through ext bkt */
if ((hits == ((1ULL << num_keys) - 1)) || !h->ext_table_support) {
if (hit_mask != NULL)
*hit_mask = hits;
__hash_rw_reader_unlock(h);
return;
}
/* need to check ext buckets for match */
for (i = 0; i < num_keys; i++) {
if ((hits & (1ULL << i)) != 0)
continue;
next_bkt = secondary_bkt[i]->next;
FOR_EACH_BUCKET(cur_bkt, next_bkt) {
if (data != NULL)
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
ret = search_one_bucket_lf(h, keys[i],
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
sig[i], &data[i], cur_bkt);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
else
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
ret = search_one_bucket_lf(h, keys[i],
hash: use partial-key hashing This commit changes the hashing mechanism to "partial-key hashing" to calculate bucket index and signature of key. This is proposed in Bin Fan, et al's paper "MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing". Basically the idea is to use "xor" to derive alternative bucket from current bucket index and signature. With "partial-key hashing", it reduces the bucket memory requirement from two cache lines to one cache line, which improves the memory efficiency and thus the lookup speed. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:48 +00:00
sig[i], NULL, cur_bkt);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
if (ret != -1) {
positions[i] = ret;
hits |= 1ULL << i;
break;
}
}
}
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
if (hit_mask != NULL)
*hit_mask = hits;
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
}
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
static inline void
__rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
int32_t num_keys, int32_t *positions,
uint64_t *hit_mask, void *data[])
{
if (h->readwrite_concur_lf_support)
hash: fix return of bulk lookup The __rte_hash_lookup_bulk() function returns void, and therefore should not return with an expression. This commit fixes the following compiler warning when attempting to compile with "-pedantic -std=c11". warning: ISO C forbids ‘return’ with expression, in function returning void [-Wpedantic] Fixes: 9eca8bd7a61c ("hash: separate lock-free and r/w lock lookup") Cc: stable@dpdk.org Signed-off-by: Jeff Shaw <jeffrey.b.shaw@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-12-08 00:01:26 +00:00
__rte_hash_lookup_bulk_lf(h, keys, num_keys, positions,
hit_mask, data);
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
else
hash: fix return of bulk lookup The __rte_hash_lookup_bulk() function returns void, and therefore should not return with an expression. This commit fixes the following compiler warning when attempting to compile with "-pedantic -std=c11". warning: ISO C forbids ‘return’ with expression, in function returning void [-Wpedantic] Fixes: 9eca8bd7a61c ("hash: separate lock-free and r/w lock lookup") Cc: stable@dpdk.org Signed-off-by: Jeff Shaw <jeffrey.b.shaw@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-12-08 00:01:26 +00:00
__rte_hash_lookup_bulk_l(h, keys, num_keys, positions,
hit_mask, data);
hash: separate lock-free and r/w lock lookup The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Suggested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com> Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
2018-11-10 18:55:34 +00:00
}
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
int
rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
uint32_t num_keys, int32_t *positions)
{
RETURN_IF_TRUE(((h == NULL) || (keys == NULL) || (num_keys == 0) ||
(num_keys > RTE_HASH_LOOKUP_BULK_MAX) ||
(positions == NULL)), -EINVAL);
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
__rte_hash_lookup_bulk(h, keys, num_keys, positions, NULL, NULL);
hash: replace with cuckoo hash implementation This patch replaces the existing hash library with another approach, using the Cuckoo Hash method to resolve collisions (open addressing), which pushes items from a full bucket when a new entry tries to be added in it, storing the evicted entry in an alternative location, using a secondary hash function. This gives the user the ability to store more entries when a bucket is full, in comparison with the previous implementation. Therefore, the unit test has been updated, as some scenarios have changed (such as the previous removed restriction). Also note that the API has not been changed, although new fields have been added in the rte_hash structure (structure is internal now). The main change when creating a new table is that the number of entries per bucket is fixed now, so its parameter is ignored now (still there to maintain the same parameters structure). The hash unit test has been updated to reflect these changes. As a last note, the maximum burst size in lookup_burst function hash been increased to 64, to improve performance. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:50 +00:00
return 0;
}
hash: allow to store data in hash table Usually hash tables not only store keys, but also data associated to them. In order to maintain the existing API, the old functions will still return the index where the key was stored. The new functions will return the data associated to that key. In the case of the lookup_bulk function, it will return also the number of entries found and a bitmask of which entries were found. Unit tests have been updated to use these new functions. As a final point, a flag has been added in rte_hash_parameters to indicate if there are new parameters for future versions, so there is no need to maintain multiple versions of the existing functions in the future. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> [Thomas: fix || operator in a precondition check]
2015-07-11 00:18:52 +00:00
int
rte_hash_lookup_bulk_data(const struct rte_hash *h, const void **keys,
uint32_t num_keys, uint64_t *hit_mask, void *data[])
{
RETURN_IF_TRUE(((h == NULL) || (keys == NULL) || (num_keys == 0) ||
(num_keys > RTE_HASH_LOOKUP_BULK_MAX) ||
(hit_mask == NULL)), -EINVAL);
int32_t positions[num_keys];
__rte_hash_lookup_bulk(h, keys, num_keys, positions, hit_mask, data);
/* Return number of hits */
return __builtin_popcountl(*hit_mask);
}
hash: add iterate function Since now rte_hash structure is private, a new function has been added to let the user iterate through the hash table, returning next key and data associated on each iteration, plus the position where they were stored. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:53 +00:00
int32_t
rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next)
{
uint32_t bucket_idx, idx, position;
struct rte_hash_key *next_key;
RETURN_IF_TRUE(((h == NULL) || (next == NULL)), -EINVAL);
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
const uint32_t total_entries_main = h->num_buckets *
RTE_HASH_BUCKET_ENTRIES;
const uint32_t total_entries = total_entries_main << 1;
/* Out of bounds of all buckets (both main table and ext table) */
if (*next >= total_entries_main)
goto extend_table;
hash: add iterate function Since now rte_hash structure is private, a new function has been added to let the user iterate through the hash table, returning next key and data associated on each iteration, plus the position where they were stored. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:53 +00:00
/* Calculate bucket and index of current iterator */
bucket_idx = *next / RTE_HASH_BUCKET_ENTRIES;
idx = *next % RTE_HASH_BUCKET_ENTRIES;
/* If current position is empty, go to the next one */
hash: add lock-free r/w concurrency Add lock-free read-write concurrency. This is achieved by the following changes. 1) Add memory ordering to avoid race conditions. The only race condition that can occur is - using the key store element before the key write is completed. Hence, while inserting the element the release memory order is used. Any other race condition is caught by the key comparison. Memory orderings are added only where needed. For ex: reads in the writer's context do not need memory ordering as there is a single writer. key_idx in the bucket entry and pdata in the key store element are used for synchronisation. key_idx is used to release an inserted entry in the bucket to the reader. Use of pdata for synchronisation is required due to updation of an existing entry where-in only the pdata is updated without updating key_idx. 2) Reader-writer concurrency issue, caused by moving the keys to their alternative locations during key insert, is solved by introducing a global counter(tbl_chng_cnt) indicating a change in table. 3) Add the flag to enable reader-writer concurrency during run time. Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Reviewed-by: Gavin Hu <gavin.hu@arm.com> Reviewed-by: Ola Liljedahl <ola.liljedahl@arm.com> Reviewed-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-26 05:37:32 +00:00
while ((position = __atomic_load_n(&h->buckets[bucket_idx].key_idx[idx],
__ATOMIC_ACQUIRE)) == EMPTY_SLOT) {
hash: add iterate function Since now rte_hash structure is private, a new function has been added to let the user iterate through the hash table, returning next key and data associated on each iteration, plus the position where they were stored. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:53 +00:00
(*next)++;
/* End of table */
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
if (*next == total_entries_main)
goto extend_table;
hash: add iterate function Since now rte_hash structure is private, a new function has been added to let the user iterate through the hash table, returning next key and data associated on each iteration, plus the position where they were stored. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:53 +00:00
bucket_idx = *next / RTE_HASH_BUCKET_ENTRIES;
idx = *next % RTE_HASH_BUCKET_ENTRIES;
}
hash: fix race condition in iterate In rte_hash_iterate, the reader lock did not protect the while loop which checks empty entry. This created a race condition that the entry may become empty when enters the lock, then a wrong key data value would be read out. This commit reads out the position in the while condition, which makes sure that the position will not be changed to empty before entering the lock. Fixes: f2e3001b53ec ("hash: support read/write concurrency") Cc: stable@dpdk.org Reported-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:45 +00:00
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
__hash_rw_reader_lock(h);
hash: add iterate function Since now rte_hash structure is private, a new function has been added to let the user iterate through the hash table, returning next key and data associated on each iteration, plus the position where they were stored. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:53 +00:00
next_key = (struct rte_hash_key *) ((char *)h->key_store +
position * h->key_entry_size);
/* Return key and data */
*key = next_key->key;
*data = next_key->pdata;
hash: support read/write concurrency The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
2018-07-10 16:59:58 +00:00
__hash_rw_reader_unlock(h);
hash: add iterate function Since now rte_hash structure is private, a new function has been added to let the user iterate through the hash table, returning next key and data associated on each iteration, plus the position where they were stored. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:53 +00:00
/* Increment iterator */
(*next)++;
remove extra parentheses in return statement fix the error reported by checkpatch: "ERROR: return is not a function, parentheses are not required" remove parentheses in return like: "return (logical expressions)" remove parentheses in return a function like: "return (rte_mempool_lookup(...))" Fixes: 6307b909b8e0 ("lib: remove extra parenthesis after return") Signed-off-by: Huawei Xie <huawei.xie@intel.com>
2016-01-27 13:58:30 +00:00
return position - 1;
hash: add extendable bucket feature In use cases that hash table capacity needs to be guaranteed, the extendable bucket feature can be used to contain extra keys in linked lists when conflict happens. This is similar concept to the extendable bucket hash table in packet framework. This commit adds the extendable bucket feature. User can turn it on or off through the extra flag field during table creation time. Extendable bucket table composes of buckets that can be linked list to current main table. When extendable bucket is enabled, the hash table load can always achieve 100%. In other words, the table can always accommodate the same number of keys as the specified table size. This provides 100% table capacity guarantee. Although keys ending up in the ext buckets may have longer look up time, they should be rare due to the cuckoo algorithm. Signed-off-by: Yipeng Wang <yipeng1.wang@intel.com> Acked-by: Dharmik Thakkar <dharmik.thakkar@arm.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2018-10-22 18:39:46 +00:00
/* Begin to iterate extendable buckets */
extend_table:
/* Out of total bound or if ext bucket feature is not enabled */
if (*next >= total_entries || !h->ext_table_support)
return -ENOENT;
bucket_idx = (*next - total_entries_main) / RTE_HASH_BUCKET_ENTRIES;
idx = (*next - total_entries_main) % RTE_HASH_BUCKET_ENTRIES;
while ((position = h->buckets_ext[bucket_idx].key_idx[idx]) == EMPTY_SLOT) {
(*next)++;
if (*next == total_entries)
return -ENOENT;
bucket_idx = (*next - total_entries_main) /
RTE_HASH_BUCKET_ENTRIES;
idx = (*next - total_entries_main) % RTE_HASH_BUCKET_ENTRIES;
}
__hash_rw_reader_lock(h);
next_key = (struct rte_hash_key *) ((char *)h->key_store +
position * h->key_entry_size);
/* Return key and data */
*key = next_key->key;
*data = next_key->pdata;
__hash_rw_reader_unlock(h);
/* Increment iterator */
(*next)++;
return position - 1;
hash: add iterate function Since now rte_hash structure is private, a new function has been added to let the user iterate through the hash table, returning next key and data associated on each iteration, plus the position where they were stored. Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
2015-07-11 00:18:53 +00:00
}
Reference in New Issue Copy Permalink