Give user a chance to customize the hash key compare function.
The default rte_hash_cmp_eq function is set in the rte_hash_create
function, but these builtin ones may not good enough, so the user
may call this to override the default one.
Signed-off-by: Yu Nemo Wenbin <yuwb_bjy@ctbri.com.cn>
Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
rte_hash_create function was accidentally duplicated in
DPDK_2.1 in rte_hash_version.map.
Fixes: 473d1beb ("hash: allow to store data in hash table")
Reported-by: Ferruh Yigit <ferruh.yigit@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
select hash function for cuckoo, fbk as rte_hash_crc_4byte
if arm64-CRC extension available
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
armv8-a has optional CRC32 extension, march=armv8-a+crc enables code
generation for the ARMv8-A architecture together with
the optional CRC32 extensions.
added RTE_MACHINE_CPUFLAG_CRC32 to detect the availability of
CRC32 extension in compile time. At run-time, The RTE_CPUFLAG_CRC32
can be used to find the availability.
armv8-a+crc target support added in GCC 4.9,
Used inline assembly and emulated __ARM_FEATURE_CRC32 to work
with tool-chain < 4.9
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
The following measurements shows improvement over the default
libc memcmp function
Length(B) by X% over libc memcmp
16 149.57%
32 122.7%
48 104.96%
64 98.21%
80 93.75%
96 90.55%
112 110.48%
128 137.24%
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
If user has not added an all zero key in the hash table,
and tries to look it up, it results in an incorrect hit,
as dummy slot in the key table has all zero as well.
Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation")
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
If using multiple cores on a system with hardware transactional
memory support, thread scaling does not work, as there was a single
point in the hash library which is a bottleneck for all threads,
which is the "free_slots" ring, which stores all the indices of
the free slots in the table.
This patch fixes the problem, by creating a local cache per logical core,
which stores locally indices of free slots,
so most times, writer threads will not interfere each other.
Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation")
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Since freeing a ring is now possible, then when freeing
a hash table, its internal ring can be freed as well.
Therefore when a new table, with the same name as a previously
freed table, is created, there is no need to look up
the already allocated ring.
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
When calculating the size for the table which allocates
the keys, size was calculated wrongly from multiplying
two 32-bit variables, resulting on a 32-bit number,
before casting to 64-bit, so maximum size was 4G.
Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation")
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
The function rte_jhash2() was renamed rte_jhash_32b and
macros RTE_HASH_KEY_LENGTH_MAX and RTE_HASH_BUCKET_ENTRIES_MAX
were tagged as deprecated, so they can be removed in 2.2.
RTE_HASH_KEY_LENGTH is replaced in unit tests by an internal macro
for the memory allocation of all keys used.
The library version number is incremented.
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
Make thash library arch-independent.
Leave unaligned union rte_thash_tuple if no support for SSE3.
Makes 32bit compiler happy by adding ULL suffix.
Signed-off-by: Vladimir Medvedkin <medvedkinv@gmail.com>
When adding with cuckoo hash a key which was already inserted
a new slot is dequeued and then enqueued back, but the enqueue
operation was not done properly.
Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation")
Signed-off-by: Xavier Simonart <xavier.simonart@intel.com>
Acked-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
In order to keep the ABI consistent with the old hash library,
hash_func_init_val field has been moved, so it remains
at the same offset as previously, since hash_func and
hash_func_init_val are fields accessed by the public function
rte_hash_hash and must keep the same offset as older versions.
Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation")
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Hash library uses optimized compare functions that use
x86 intrinsics, therefore non-x86 systems could not build
the library. In that case, the compare function is set
to the generic memcmp.
Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation")
Reported-by: Zhigang Lu <zlu@ezchip.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Zhigang Lu <zlu@ezchip.com>
_mm_test_all_zeros is not available for CPUs with no SSE4.1,
therefore, DPDK would not build.
This patch adds an alternative for this, using _mm_cmpeq_epi32 and
_mm_movemask_epi8.
Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation")
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
gcc 4.4 and 4.5 throws following error:
rte_cuckoo_hash.c:145: error: flexible array member in otherwise empty struct.
This is due to empty length in flexible array, which has been changed to use
size 0 in the declaration of the array.
Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation")
Reported-by: Olga Shern <olgas@mellanox.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
When encountering a loop while adding a new entry,
element out of bounds of array was being unnecessarily resetted.
Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation")
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Move malloc inside eal and create a new section in MAINTAINERS file for
Memory Allocation in EAL.
Create a dummy malloc library to avoid breaking applications that have
librte_malloc in their DT_NEEDED entries.
This is the first step towards using malloc to allocate memory directly
from memsegs. Thus, memzones would allocate memory through malloc,
allowing to free memzones.
Signed-off-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
The cuckoo hash has a fixed number of entries per bucket, so the
configuration parameter for this is unused. We change this field in the
parameters struct to "reserved" to indicate that there is now no such
parameter value, while at the same time keeping ABI consistency.
Fixes: 48a399119619 ("hash: replace with cuckoo hash implementation")
Suggested-by: Thomas Monjalon <thomas.monjalon@6wind.com>
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
This is necessary because the required CPU flags may not be defined on
other architectures.
Signed-off-by: Cyril Chemparathy <cchemparathy@ezchip.com>
Signed-off-by: Zhigang Lu <zlu@ezchip.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Since now rte_hash structure is private, a new function
has been added to let the user iterate through the hash table,
returning next key and data associated on each iteration,
plus the position where they were stored.
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Usually hash tables not only store keys, but also data associated
to them. In order to maintain the existing API, the old functions
will still return the index where the key was stored.
The new functions will return the data associated to that key.
In the case of the lookup_bulk function, it will return also
the number of entries found and a bitmask of which entries
were found.
Unit tests have been updated to use these new functions.
As a final point, a flag has been added in rte_hash_parameters
to indicate if there are new parameters for future versions,
so there is no need to maintain multiple versions
of the existing functions in the future.
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
[Thomas: fix || operator in a precondition check]
Added reset function to be able to empty the table,
without having to destroy and create it again.
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
This patch replaces the existing hash library with another approach,
using the Cuckoo Hash method to resolve collisions (open addressing),
which pushes items from a full bucket when a new entry tries
to be added in it, storing the evicted entry in an alternative location,
using a secondary hash function.
This gives the user the ability to store more entries when a bucket
is full, in comparison with the previous implementation.
Therefore, the unit test has been updated, as some scenarios have changed
(such as the previous removed restriction).
Also note that the API has not been changed, although new fields
have been added in the rte_hash structure (structure is internal now).
The main change when creating a new table is that the number of entries
per bucket is fixed now, so its parameter is ignored now
(still there to maintain the same parameters structure).
The hash unit test has been updated to reflect these changes.
As a last note, the maximum burst size in lookup_burst function
hash been increased to 64, to improve performance.
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
rte_hash structure should not be a public structure,
and therefore it should be moved to the C file and be declared
as internal. rte_hash_hash implementation is also moved
to the C file, as it uses the structure.
This patch also removes part of a unit test that was checking
a field of the structure.
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
In the no-x86 branch, a parentheses is missing.
And need to include <limits.h> header to use CHAR_BIT.
Fixes: 49361c3f3cfa ("hash: remove duplicated code")
Signed-off-by: Bruce Liu <2103458176@qq.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Software implementation of the Toeplitz hash function used by RSS.
Can be used either for packet distribution on single queue NIC
or for simulating of RSS computation on specific NIC (for example
after GRE header decapsulating).
Signed-off-by: Vladimir Medvedkin <medvedkinv@gmail.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
rte_hash_hash is a public function but was not in
rte_hash_version.map
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
Since sig_tbl_bucket_size and key_tbl_key_size are explicitly aligned
at initialization, offset dereferences in the hash table code cannot
possibly be unaligned. However, the compiler is unaware of this fact
and complains on -Wcast-align. This patch modifies the code to use
RTE_PTR_ADD(), thereby silencing the compiler by casting through (void
*).
Signed-off-by: Cyril Chemparathy <cchemparathy@ezchip.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
Changed name to something more meaningful,
and mark rte_jhash2 as deprecated.
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
rte_jhash is basically like __rte_jhash_2hashes but
it returns only 1 hash, instead of 2.
In order to remove duplicated code, rte_jhash calls __rte_jhash_2hashes,
passing 0 as the second seed and returning just the first hash value.
(performance penalty is negligible)
The same is done with rte_jhash2. Also, rte_jhash2 is just an specific case
where keys are multiple of 32 bits, and where no key alignment check is required.
So,to avoid duplicated code, the function calls __rte_jhash_2hashes
with check_align = 0 (to use the optimal path)
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
With the jhash update, two new functions were introduced:
- rte_jhash_2hashes: Same as rte_jhash, but takes two seeds
and return two hashes (uint32_ts)
- rte_jhash2_2hashes: Same as rte_jhash2, but takes two seeds
and return two hashes (uint32_ts)
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
[Thomas: fix doxygen typos]
Jenkins hash function was developed originally in 1996,
and was integrated in first versions of DPDK.
The function has been improved in 2006,
achieving up to 35% better performance, compared to the original one.
This patch integrates that code into the rte_jhash library.
It also updates the precalculated hash values in the unit test,
as the code now returns different values (expected).
A final note has been added in release notes for stating
the changes made.
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Fix rte_hash_crc() function by making use of uintptr_t variable
to hold a pointer to data being hashed. In this way, casting uint64_t
pointer to uint32_t avoided.
Fixes: 614289298daf ("hash: slice CRC data into 8-byte pieces")
Signed-off-by: Yerden Zhumabekov <e_zhumabekov@sts.kz>
Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
When compile target i686 in platform x86_64, the stub function will
be called, and return zero.
Fixes: 12fa4a00788d ("hash: fix unsupported crc instruction in i686 platform")
Signed-off-by: Michael Qiu <michael.qiu@intel.com>
Acked-by: Thomas Monjalon <thomas.monjalon@6wind.com>
Error: unsupported instruction `crc32'
The root cause is that i686 platform does not support 'crc32q'
Need make it only available in x86_64 platform.
Signed-off-by: Michael Qiu <michael.qiu@intel.com>
Acked-by: Yerden Zhumabekov <yerden.zhumabekov@sts.kz>
Calculating hash for data of variable length is more efficient
when that data is sliced into 8-byte pieces. The rest part of data
is hashed using CRC32 functions with either 8 and 4 byte operands.
Signed-off-by: Yerden Zhumabekov <e_zhumabekov@sts.kz>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Initially, SSE4.2 support is detected via the constructor function.
Added rte_hash_crc_set_alg() function to detect and set CRC32
implementation if necessary. SSE4.2 is allowed by default.
rte_hash_crc_*byte() functions reworked so they choose available
CRC32 implementation in the runtime.
Signed-off-by: Yerden Zhumabekov <e_zhumabekov@sts.kz>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
SSE4.2 provides CRC32 intrinsic with 8-byte operand.
Signed-off-by: Yerden Zhumabekov <e_zhumabekov@sts.kz>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Give up using built-in intrinsics and use our own assembly
implementation. Remove #include entry as well.
Signed-off-by: Yerden Zhumabekov <e_zhumabekov@sts.kz>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Added:
- crc32c_sse42_u32() emits 'crc32l' asm instruction;
- crc32c_sse42_u64() emits 'crc32q' asm instruction;
- crc32c_sse42_u64_mimic(), wrapper in case of run on 32-bit platform.
Signed-off-by: Yerden Zhumabekov <e_zhumabekov@sts.kz>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Add lookup tables for CRC32 algorithm, crc32c_1word() and
crc32c_2words() functions returning hash of 32-bit and 64-bit
operand.
Signed-off-by: Yerden Zhumabekov <e_zhumabekov@sts.kz>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
To differentiate libraries that break ABI, we add a library version number
suffix to the library, which must be incremented when a given libraries ABI is
broken. This patch enforces that addition, sets the initial abi soname
extension to 1 for each library and creates a symlink to the base SONAME so that
the test applications will link properly.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Acked-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
Add linker version script files to each DPDK library to put a stake in the
ground from which we can start cleaning up API's
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Acked-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
CACHE_LINE_SIZE is a macro defined in machine/param.h in FreeBSD and
conflicts with DPDK macro version.
Adding RTE_ prefix to avoid conflicts.
CACHE_LINE_MASK and CACHE_LINE_ROUNDUP are also prefixed.
Signed-off-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
[Thomas: updated on HEAD, including PPC]
Since the data structures such as rings are shared in their entirety,
those TAILQ pointers are shared as well. Meaning that, after a
successful rte_ring creation, the tailq_next pointer of the last
ring in the TAILQ will be updated with a pointer to a ring which may
not be present in the address space of another process (i.e. a ring
that may be host-local or guest-local, and not shared over IVSHMEM).
Any successive ring create/lookup on the other side of IVSHMEM will
result in trying to dereference an invalid pointer.
This patchset fixes this problem by creating a default tailq entry
that may be used by any data structure that chooses to use TAILQs.
This default TAILQ entry will consist of a tailq_next/tailq_prev
pointers, and an opaque pointer to arbitrary data. All TAILQ
pointers from data structures themselves will be removed and
replaced by those generic TAILQ entries, thus fixing the problem
of potentially exposing local address space to shared structures.
Technically, only rte_ring structure require modification, because
IVSHMEM is only using memzones (which aren't in TAILQs) and rings,
but for consistency's sake other TAILQ-based data structures were
adapted as well.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
