d/numam-dpdk
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
99d7c45cf8
numam-dpdk/drivers/net/mlx5/mlx5_mr.c

549 lines
16 KiB
C
Raw Normal View History

net/mlx5: use SPDX tags in 6WIND copyrighted files Signed-off-by: Olivier Matz <olivier.matz@6wind.com> Acked-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Thomas Monjalon <thomas@monjalon.net>
2018-01-29 13:11:30 +00:00
/* SPDX-License-Identifier: BSD-3-Clause
* Copyright 2016 6WIND S.A.
align SPDX Mellanox copyrights Aligning Mellanox SPDX copyrights to a single format. In addition replace to SPDX licence files which were missed. Signed-off-by: Shahaf Shuler <shahafs@mellanox.com> Acked-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
2018-03-20 19:20:35 +00:00
* Copyright 2016 Mellanox Technologies, Ltd
net/mlx5: split memory registration function Except for the first time when memory registration occurs, the lkey is always cached. Since memory registration is slow and performs system calls, performance can be improved by moving that code to its own function outside of the data path so only the lookup code is left in the original inlined function. Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
2016-06-24 13:17:41 +00:00
*/
remove useless include of EAL memory config header Restrict this header inclusion to its real users. Fixes: 028669bc9f0d ("eal: hide shared memory config") Cc: stable@dpdk.org Signed-off-by: David Marchand <david.marchand@redhat.com> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
2019-08-12 06:53:33 +00:00
#include <rte_eal_memconfig.h>
net/mlx5: split memory registration function Except for the first time when memory registration occurs, the lkey is always cached. Since memory registration is slow and performs system calls, performance can be improved by moving that code to its own function outside of the data path so only the lookup code is left in the original inlined function. Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
2016-06-24 13:17:41 +00:00
#include <rte_mempool.h>
net/mlx5: add reference counter on memory region This patch introduce the Memory region as a shared object where users should get a reference to it by calling the priv_mr_get() or priv_mr_new() to create the memory region. This last one will register the memory pool in the kernel driver and retrieve the associated memory region. This should help to reduce the memory consumption cause by registering multiple times the same memory pool. Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Acked-by: Yongseok Koh <yskoh@mellanox.com>
2017-10-09 14:44:45 +00:00
#include <rte_malloc.h>
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
#include <rte_rwlock.h>
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
#include <rte_bus_pci.h>
net/mlx5: split memory registration function Except for the first time when memory registration occurs, the lkey is always cached. Since memory registration is slow and performs system calls, performance can be improved by moving that code to its own function outside of the data path so only the lookup code is left in the original inlined function. Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
2016-06-24 13:17:41 +00:00
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
#include <mlx5_common_mp.h>
#include <mlx5_common_mr.h>
common/mlx5: introduce common library A new Mellanox vdpa PMD will be added to support vdpa operations by Mellanox adapters. This vdpa PMD design includes mlx5_glue and mlx5_devx operations and large parts of them are shared with the net/mlx5 PMD. Create a new common library in drivers/common for mlx5 PMDs. Move mlx5_glue, mlx5_devx_cmds and their dependencies to the new mlx5 common library in drivers/common. The files mlx5_devx_cmds.c, mlx5_devx_cmds.h, mlx5_glue.c, mlx5_glue.h and mlx5_prm.h are moved as is from drivers/net/mlx5 to drivers/common/mlx5. Share the log mechanism macros. Separate also the log mechanism to allow different log level control to the common library. Build files and version files are adjusted accordingly. Include lines are adjusted accordingly. Signed-off-by: Matan Azrad <matan@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-01-29 12:38:27 +00:00
net/mlx5: split memory registration function Except for the first time when memory registration occurs, the lkey is always cached. Since memory registration is slow and performs system calls, performance can be improved by moving that code to its own function outside of the data path so only the lookup code is left in the original inlined function. Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
2016-06-24 13:17:41 +00:00
#include "mlx5.h"
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
#include "mlx5_mr.h"
net/mlx5: split memory registration function Except for the first time when memory registration occurs, the lkey is always cached. Since memory registration is slow and performs system calls, performance can be improved by moving that code to its own function outside of the data path so only the lookup code is left in the original inlined function. Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
2016-06-24 13:17:41 +00:00
#include "mlx5_rxtx.h"
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
struct mr_find_contig_memsegs_data {
uintptr_t addr;
uintptr_t start;
uintptr_t end;
const struct rte_memseg_list *msl;
};
struct mr_update_mp_data {
struct rte_eth_dev *dev;
struct mlx5_mr_ctrl *mr_ctrl;
int ret;
};
/**
* Callback for memory free event. Iterate freed memsegs and check whether it
* belongs to an existing MR. If found, clear the bit from bitmap of MR. As a
* result, the MR would be fragmented. If it becomes empty, the MR will be freed
* later by mlx5_mr_garbage_collect(). Even if this callback is called from a
* secondary process, the garbage collector will be called in primary process
* as the secondary process can't call mlx5_mr_create().
*
* The global cache must be rebuilt if there's any change and this event has to
* be propagated to dataplane threads to flush the local caches.
*
net/mlx5: update memory event callback for shared context Mellanox mlx5 PMD implements the list of devices to process the memory free events to reflect the actual memory state to Memory Regions. Because this list contains the devices and devices may share the same context the callback routine may be called multiple times with the same parameter, that is not optimal. This patch modifies the list to contain the device contexts instead of device objects and shared context is included in the list only once. Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com> Acked-by: Shahaf Shuler <shahafs@mellanox.com>
2019-04-27 04:32:57 +00:00
* @param sh
* Pointer to the Ethernet device shared context.
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
* @param addr
* Address of freed memory.
* @param len
* Size of freed memory.
*/
static void
net/mlx5: rename Verbs shared object Replace all 'mlx5_ibv_shared' appearances with 'mlx5_dev_ctx_shared'. Signed-off-by: Ophir Munk <ophirmu@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com>
2020-06-03 15:05:55 +00:00
mlx5_mr_mem_event_free_cb(struct mlx5_dev_ctx_shared *sh,
net/mlx5: update memory event callback for shared context Mellanox mlx5 PMD implements the list of devices to process the memory free events to reflect the actual memory state to Memory Regions. Because this list contains the devices and devices may share the same context the callback routine may be called multiple times with the same parameter, that is not optimal. This patch modifies the list to contain the device contexts instead of device objects and shared context is included in the list only once. Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com> Acked-by: Shahaf Shuler <shahafs@mellanox.com>
2019-04-27 04:32:57 +00:00
const void *addr, size_t len)
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
{
const struct rte_memseg_list *msl;
struct mlx5_mr *mr;
int ms_n;
int i;
int rebuild = 0;
net/mlx5: update memory event callback for shared context Mellanox mlx5 PMD implements the list of devices to process the memory free events to reflect the actual memory state to Memory Regions. Because this list contains the devices and devices may share the same context the callback routine may be called multiple times with the same parameter, that is not optimal. This patch modifies the list to contain the device contexts instead of device objects and shared context is included in the list only once. Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com> Acked-by: Shahaf Shuler <shahafs@mellanox.com>
2019-04-27 04:32:57 +00:00
DEBUG("device %s free callback: addr=%p, len=%zu",
sh->ibdev_name, addr, len);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
msl = rte_mem_virt2memseg_list(addr);
/* addr and len must be page-aligned. */
common/mlx5: improve assert control Use the MLX5_ASSERT macros instead of the standard assert clause. Depends on the RTE_LIBRTE_MLX5_DEBUG configuration option to define it. If RTE_LIBRTE_MLX5_DEBUG is enabled MLX5_ASSERT is equal to RTE_VERIFY to bypass the global CONFIG_RTE_ENABLE_ASSERT option. If RTE_LIBRTE_MLX5_DEBUG is disabled, the global CONFIG_RTE_ENABLE_ASSERT can still make this assert active by calling RTE_VERIFY inside RTE_ASSERT. Signed-off-by: Alexander Kozyrev <akozyrev@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-01-30 16:14:40 +00:00
MLX5_ASSERT((uintptr_t)addr ==
RTE_ALIGN((uintptr_t)addr, msl->page_sz));
MLX5_ASSERT(len == RTE_ALIGN(len, msl->page_sz));
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
ms_n = len / msl->page_sz;
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
rte_rwlock_write_lock(&sh->share_cache.rwlock);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
/* Clear bits of freed memsegs from MR. */
for (i = 0; i < ms_n; ++i) {
const struct rte_memseg *ms;
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
struct mr_cache_entry entry;
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
uintptr_t start;
int ms_idx;
uint32_t pos;
/* Find MR having this memseg. */
start = (uintptr_t)addr + i * msl->page_sz;
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
mr = mlx5_mr_lookup_list(&sh->share_cache, &entry, start);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
if (mr == NULL)
continue;
common/mlx5: improve assert control Use the MLX5_ASSERT macros instead of the standard assert clause. Depends on the RTE_LIBRTE_MLX5_DEBUG configuration option to define it. If RTE_LIBRTE_MLX5_DEBUG is enabled MLX5_ASSERT is equal to RTE_VERIFY to bypass the global CONFIG_RTE_ENABLE_ASSERT option. If RTE_LIBRTE_MLX5_DEBUG is disabled, the global CONFIG_RTE_ENABLE_ASSERT can still make this assert active by calling RTE_VERIFY inside RTE_ASSERT. Signed-off-by: Alexander Kozyrev <akozyrev@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-01-30 16:14:40 +00:00
MLX5_ASSERT(mr->msl); /* Can't be external memory. */
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
ms = rte_mem_virt2memseg((void *)start, msl);
common/mlx5: improve assert control Use the MLX5_ASSERT macros instead of the standard assert clause. Depends on the RTE_LIBRTE_MLX5_DEBUG configuration option to define it. If RTE_LIBRTE_MLX5_DEBUG is enabled MLX5_ASSERT is equal to RTE_VERIFY to bypass the global CONFIG_RTE_ENABLE_ASSERT option. If RTE_LIBRTE_MLX5_DEBUG is disabled, the global CONFIG_RTE_ENABLE_ASSERT can still make this assert active by calling RTE_VERIFY inside RTE_ASSERT. Signed-off-by: Alexander Kozyrev <akozyrev@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-01-30 16:14:40 +00:00
MLX5_ASSERT(ms != NULL);
MLX5_ASSERT(msl->page_sz == ms->hugepage_sz);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
ms_idx = rte_fbarray_find_idx(&msl->memseg_arr, ms);
pos = ms_idx - mr->ms_base_idx;
common/mlx5: improve assert control Use the MLX5_ASSERT macros instead of the standard assert clause. Depends on the RTE_LIBRTE_MLX5_DEBUG configuration option to define it. If RTE_LIBRTE_MLX5_DEBUG is enabled MLX5_ASSERT is equal to RTE_VERIFY to bypass the global CONFIG_RTE_ENABLE_ASSERT option. If RTE_LIBRTE_MLX5_DEBUG is disabled, the global CONFIG_RTE_ENABLE_ASSERT can still make this assert active by calling RTE_VERIFY inside RTE_ASSERT. Signed-off-by: Alexander Kozyrev <akozyrev@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-01-30 16:14:40 +00:00
MLX5_ASSERT(rte_bitmap_get(mr->ms_bmp, pos));
MLX5_ASSERT(pos < mr->ms_bmp_n);
net/mlx5: update memory event callback for shared context Mellanox mlx5 PMD implements the list of devices to process the memory free events to reflect the actual memory state to Memory Regions. Because this list contains the devices and devices may share the same context the callback routine may be called multiple times with the same parameter, that is not optimal. This patch modifies the list to contain the device contexts instead of device objects and shared context is included in the list only once. Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com> Acked-by: Shahaf Shuler <shahafs@mellanox.com>
2019-04-27 04:32:57 +00:00
DEBUG("device %s MR(%p): clear bitmap[%u] for addr %p",
sh->ibdev_name, (void *)mr, pos, (void *)start);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
rte_bitmap_clear(mr->ms_bmp, pos);
if (--mr->ms_n == 0) {
LIST_REMOVE(mr, mr);
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
LIST_INSERT_HEAD(&sh->share_cache.mr_free_list, mr, mr);
net/mlx5: update memory event callback for shared context Mellanox mlx5 PMD implements the list of devices to process the memory free events to reflect the actual memory state to Memory Regions. Because this list contains the devices and devices may share the same context the callback routine may be called multiple times with the same parameter, that is not optimal. This patch modifies the list to contain the device contexts instead of device objects and shared context is included in the list only once. Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com> Acked-by: Shahaf Shuler <shahafs@mellanox.com>
2019-04-27 04:32:57 +00:00
DEBUG("device %s remove MR(%p) from list",
sh->ibdev_name, (void *)mr);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
}
/*
* MR is fragmented or will be freed. the global cache must be
* rebuilt.
*/
rebuild = 1;
}
if (rebuild) {
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
mlx5_mr_rebuild_cache(&sh->share_cache);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
/*
* Flush local caches by propagating invalidation across cores.
* rte_smp_wmb() is enough to synchronize this event. If one of
* freed memsegs is seen by other core, that means the memseg
* has been allocated by allocator, which will come after this
* free call. Therefore, this store instruction (incrementing
* generation below) will be guaranteed to be seen by other core
* before the core sees the newly allocated memory.
*/
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
++sh->share_cache.dev_gen;
net/mlx5: clean-up developer logs Split maintainers logs from user logs. A lot of debug logs are present providing internal information on how the PMD works to users. Such logs should not be available for them and thus should remain available only when the PMD is compiled in debug mode. This commits removes some useless debug logs, move the Maintainers ones under DEBUG and also move dump into debug mode only. Cc: stable@dpdk.org Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
2018-06-05 08:45:22 +00:00
DEBUG("broadcasting local cache flush, gen=%d",
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
sh->share_cache.dev_gen);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
rte_smp_wmb();
}
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
rte_rwlock_write_unlock(&sh->share_cache.rwlock);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
}
/**
* Callback for memory event. This can be called from both primary and secondary
* process.
*
* @param event_type
* Memory event type.
* @param addr
* Address of memory.
* @param len
* Size of memory.
*/
void
mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
size_t len, void *arg __rte_unused)
{
net/mlx5: rename Verbs shared object Replace all 'mlx5_ibv_shared' appearances with 'mlx5_dev_ctx_shared'. Signed-off-by: Ophir Munk <ophirmu@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com>
2020-06-03 15:05:55 +00:00
struct mlx5_dev_ctx_shared *sh;
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
struct mlx5_dev_list *dev_list = &mlx5_shared_data->mem_event_cb_list;
net/mlx5: fix memory event on secondary process As the memory event is propagated to secondary processes, the event is processed redundantly. This should be processed once because the data structure used for MR and the event is global across the processes. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com> Acked-by: Shahaf Shuler <shahafs@mellanox.com>
2019-04-01 21:12:53 +00:00
/* Must be called from the primary process. */
common/mlx5: improve assert control Use the MLX5_ASSERT macros instead of the standard assert clause. Depends on the RTE_LIBRTE_MLX5_DEBUG configuration option to define it. If RTE_LIBRTE_MLX5_DEBUG is enabled MLX5_ASSERT is equal to RTE_VERIFY to bypass the global CONFIG_RTE_ENABLE_ASSERT option. If RTE_LIBRTE_MLX5_DEBUG is disabled, the global CONFIG_RTE_ENABLE_ASSERT can still make this assert active by calling RTE_VERIFY inside RTE_ASSERT. Signed-off-by: Alexander Kozyrev <akozyrev@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-01-30 16:14:40 +00:00
MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
switch (event_type) {
case RTE_MEM_EVENT_FREE:
rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock);
/* Iterate all the existing mlx5 devices. */
net/mlx5: update memory event callback for shared context Mellanox mlx5 PMD implements the list of devices to process the memory free events to reflect the actual memory state to Memory Regions. Because this list contains the devices and devices may share the same context the callback routine may be called multiple times with the same parameter, that is not optimal. This patch modifies the list to contain the device contexts instead of device objects and shared context is included in the list only once. Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com> Acked-by: Shahaf Shuler <shahafs@mellanox.com>
2019-04-27 04:32:57 +00:00
LIST_FOREACH(sh, dev_list, mem_event_cb)
mlx5_mr_mem_event_free_cb(sh, addr, len);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock);
break;
case RTE_MEM_EVENT_ALLOC:
default:
break;
}
}
/**
* Bottom-half of LKey search on Rx.
*
* @param rxq
* Pointer to Rx queue structure.
* @param addr
* Search key.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
*/
uint32_t
mlx5_rx_addr2mr_bh(struct mlx5_rxq_data *rxq, uintptr_t addr)
{
struct mlx5_rxq_ctrl *rxq_ctrl =
container_of(rxq, struct mlx5_rxq_ctrl, rxq);
struct mlx5_mr_ctrl *mr_ctrl = &rxq->mr_ctrl;
net/mlx: prefix private structure The private structure stored in rte_eth_dev->data->dev_private was named "struct priv". In order to ease code browsing, the structure is renamed "struct mlx[45]_priv". Cc: stable@dpdk.org Signed-off-by: Thomas Monjalon <thomas@monjalon.net> Acked-by: Yongseok Koh <yskoh@mellanox.com>
2019-02-21 09:29:14 +00:00
struct mlx5_priv *priv = rxq_ctrl->priv;
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
return mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id,
&priv->sh->share_cache, mr_ctrl, addr,
priv->config.mr_ext_memseg_en);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
}
/**
* Bottom-half of LKey search on Tx.
*
* @param txq
* Pointer to Tx queue structure.
* @param addr
* Search key.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
*/
net/mlx5: optimize Tx external memory registration There's some performance drop due to extra condition checks on the datapath. Checking for external memory registration should be consolidated to the existing bottom-half. Fixes: 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com> Acked-by: Shahaf Shuler <shahafs@mellanox.com>
2018-11-15 10:29:17 +00:00
static uint32_t
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
{
struct mlx5_txq_ctrl *txq_ctrl =
container_of(txq, struct mlx5_txq_ctrl, txq);
struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
net/mlx: prefix private structure The private structure stored in rte_eth_dev->data->dev_private was named "struct priv". In order to ease code browsing, the structure is renamed "struct mlx[45]_priv". Cc: stable@dpdk.org Signed-off-by: Thomas Monjalon <thomas@monjalon.net> Acked-by: Yongseok Koh <yskoh@mellanox.com>
2019-02-21 09:29:14 +00:00
struct mlx5_priv *priv = txq_ctrl->priv;
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
return mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id,
&priv->sh->share_cache, mr_ctrl, addr,
priv->config.mr_ext_memseg_en);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
}
net/mlx5: optimize Tx external memory registration There's some performance drop due to extra condition checks on the datapath. Checking for external memory registration should be consolidated to the existing bottom-half. Fixes: 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com> Acked-by: Shahaf Shuler <shahafs@mellanox.com>
2018-11-15 10:29:17 +00:00
/**
* Bottom-half of LKey search on Tx. If it can't be searched in the memseg
* list, register the mempool of the mbuf as externally allocated memory.
*
* @param txq
* Pointer to Tx queue structure.
* @param mb
* Pointer to mbuf.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
*/
uint32_t
mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
{
uintptr_t addr = (uintptr_t)mb->buf_addr;
uint32_t lkey;
lkey = mlx5_tx_addr2mr_bh(txq, addr);
if (lkey == UINT32_MAX && rte_errno == ENXIO) {
/* Mempool may have externally allocated memory. */
return mlx5_tx_update_ext_mp(txq, addr, mlx5_mb2mp(mb));
}
return lkey;
}
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
/**
* Called during rte_mempool_mem_iter() by mlx5_mr_update_ext_mp().
*
* Externally allocated chunk is registered and a MR is created for the chunk.
* The MR object is added to the global list. If memseg list of a MR object
* (mr->msl) is null, the MR object can be regarded as externally allocated
* memory.
*
* Once external memory is registered, it should be static. If the memory is
* freed and the virtual address range has different physical memory mapped
* again, it may cause crash on device due to the wrong translation entry. PMD
* can't track the free event of the external memory for now.
*/
static void
mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
struct rte_mempool_memhdr *memhdr,
unsigned mem_idx __rte_unused)
{
struct mr_update_mp_data *data = opaque;
struct rte_eth_dev *dev = data->dev;
net/mlx: prefix private structure The private structure stored in rte_eth_dev->data->dev_private was named "struct priv". In order to ease code browsing, the structure is renamed "struct mlx[45]_priv". Cc: stable@dpdk.org Signed-off-by: Thomas Monjalon <thomas@monjalon.net> Acked-by: Yongseok Koh <yskoh@mellanox.com>
2019-02-21 09:29:14 +00:00
struct mlx5_priv *priv = dev->data->dev_private;
net/mlx5: rename Verbs shared object Replace all 'mlx5_ibv_shared' appearances with 'mlx5_dev_ctx_shared'. Signed-off-by: Ophir Munk <ophirmu@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com>
2020-06-03 15:05:55 +00:00
struct mlx5_dev_ctx_shared *sh = priv->sh;
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
struct mlx5_mr_ctrl *mr_ctrl = data->mr_ctrl;
struct mlx5_mr *mr = NULL;
uintptr_t addr = (uintptr_t)memhdr->addr;
size_t len = memhdr->len;
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
struct mr_cache_entry entry;
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
uint32_t lkey;
common/mlx5: improve assert control Use the MLX5_ASSERT macros instead of the standard assert clause. Depends on the RTE_LIBRTE_MLX5_DEBUG configuration option to define it. If RTE_LIBRTE_MLX5_DEBUG is enabled MLX5_ASSERT is equal to RTE_VERIFY to bypass the global CONFIG_RTE_ENABLE_ASSERT option. If RTE_LIBRTE_MLX5_DEBUG is disabled, the global CONFIG_RTE_ENABLE_ASSERT can still make this assert active by calling RTE_VERIFY inside RTE_ASSERT. Signed-off-by: Alexander Kozyrev <akozyrev@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-01-30 16:14:40 +00:00
MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
/* If already registered, it should return. */
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
rte_rwlock_read_lock(&sh->share_cache.rwlock);
lkey = mlx5_mr_lookup_cache(&sh->share_cache, &entry, addr);
rte_rwlock_read_unlock(&sh->share_cache.rwlock);
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
if (lkey != UINT32_MAX)
return;
DRV_LOG(DEBUG, "port %u register MR for chunk #%d of mempool (%s)",
dev->data->port_id, mem_idx, mp->name);
net/mlx5: add memory region callbacks in per-device cache Prior to this commit MR operations were verbs based and hard coded under common/mlx5/linux directory. This commit enables upper layers (e.g. net/mlx5) to determine which MR operations to use. For example the net layer could set devx based MR operations in non-Linux environments. The reg_mr and dereg_mr callbacks are added to the global per-device MR cache 'struct mlx5_mr_share_cache'. Signed-off-by: Ophir Munk <ophirmu@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com>
2020-06-16 09:44:45 +00:00
mr = mlx5_create_mr_ext(sh->pd, addr, len, mp->socket_id,
sh->share_cache.reg_mr_cb);
net/mlx5: refactor external memory registration Move the memory region creation to a separate function to prepare the ground for the reuse of it on the PCI driver map and unmap functions. Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:01 +00:00
if (!mr) {
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
DRV_LOG(WARNING,
net/mlx5: refactor external memory registration Move the memory region creation to a separate function to prepare the ground for the reuse of it on the PCI driver map and unmap functions. Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:01 +00:00
"port %u unable to allocate a new MR of"
" mempool (%s).",
dev->data->port_id, mp->name);
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
data->ret = -1;
return;
}
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
rte_rwlock_write_lock(&sh->share_cache.rwlock);
LIST_INSERT_HEAD(&sh->share_cache.mr_list, mr, mr);
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
/* Insert to the global cache table. */
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
mlx5_mr_insert_cache(&sh->share_cache, mr);
rte_rwlock_write_unlock(&sh->share_cache.rwlock);
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
/* Insert to the local cache table */
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
mlx5_mr_addr2mr_bh(sh->pd, &priv->mp_id, &sh->share_cache,
mr_ctrl, addr, priv->config.mr_ext_memseg_en);
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
}
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
/**
* Finds the first ethdev that match the pci device.
* The existence of multiple ethdev per pci device is only with representors.
* On such case, it is enough to get only one of the ports as they all share
* the same ibv context.
*
* @param pdev
* Pointer to the PCI device.
*
* @return
* Pointer to the ethdev if found, NULL otherwise.
*/
static struct rte_eth_dev *
pci_dev_to_eth_dev(struct rte_pci_device *pdev)
{
net/mlx5: replace a port lookup with ethdev API Looking for an ethdev port is better (and more efficient) with an ethdev API than an EAL one. Signed-off-by: Thomas Monjalon <thomas@monjalon.net> Acked-by: Shahaf Shuler <shahafs@mellanox.com>
2019-04-18 00:15:29 +00:00
uint16_t port_id;
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
net/mlx5: use direct API to find port by device Using RTE_ETH_FOREACH_DEV_OF loop is not necessary when the driver wants to find only the first match. Use rte_eth_find_next_of to find it. Signed-off-by: Michael Baum <michaelba@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com>
2020-06-24 13:23:57 +00:00
port_id = rte_eth_find_next_of(0, &pdev->device);
if (port_id == RTE_MAX_ETHPORTS)
return NULL;
return &rte_eth_devices[port_id];
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
}
/**
* DPDK callback to DMA map external memory to a PCI device.
*
* @param pdev
* Pointer to the PCI device.
* @param addr
* Starting virtual address of memory to be mapped.
* @param iova
* Starting IOVA address of memory to be mapped.
* @param len
* Length of memory segment being mapped.
*
* @return
* 0 on success, negative value on error.
*/
int
mlx5_dma_map(struct rte_pci_device *pdev, void *addr,
uint64_t iova __rte_unused, size_t len)
{
struct rte_eth_dev *dev;
struct mlx5_mr *mr;
struct mlx5_priv *priv;
net/mlx5: rename Verbs shared object Replace all 'mlx5_ibv_shared' appearances with 'mlx5_dev_ctx_shared'. Signed-off-by: Ophir Munk <ophirmu@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com>
2020-06-03 15:05:55 +00:00
struct mlx5_dev_ctx_shared *sh;
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
dev = pci_dev_to_eth_dev(pdev);
if (!dev) {
DRV_LOG(WARNING, "unable to find matching ethdev "
"to PCI device %p", (void *)pdev);
rte_errno = ENODEV;
return -1;
}
priv = dev->data->dev_private;
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
sh = priv->sh;
net/mlx5: add memory region callbacks in per-device cache Prior to this commit MR operations were verbs based and hard coded under common/mlx5/linux directory. This commit enables upper layers (e.g. net/mlx5) to determine which MR operations to use. For example the net layer could set devx based MR operations in non-Linux environments. The reg_mr and dereg_mr callbacks are added to the global per-device MR cache 'struct mlx5_mr_share_cache'. Signed-off-by: Ophir Munk <ophirmu@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com>
2020-06-16 09:44:45 +00:00
mr = mlx5_create_mr_ext(sh->pd, (uintptr_t)addr, len, SOCKET_ID_ANY,
sh->share_cache.reg_mr_cb);
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
if (!mr) {
DRV_LOG(WARNING,
"port %u unable to dma map", dev->data->port_id);
rte_errno = EINVAL;
return -1;
}
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
rte_rwlock_write_lock(&sh->share_cache.rwlock);
LIST_INSERT_HEAD(&sh->share_cache.mr_list, mr, mr);
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
/* Insert to the global cache table. */
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
mlx5_mr_insert_cache(&sh->share_cache, mr);
rte_rwlock_write_unlock(&sh->share_cache.rwlock);
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
return 0;
}
/**
* DPDK callback to DMA unmap external memory to a PCI device.
*
* @param pdev
* Pointer to the PCI device.
* @param addr
* Starting virtual address of memory to be unmapped.
* @param iova
* Starting IOVA address of memory to be unmapped.
* @param len
* Length of memory segment being unmapped.
*
* @return
* 0 on success, negative value on error.
*/
int
mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr,
uint64_t iova __rte_unused, size_t len __rte_unused)
{
struct rte_eth_dev *dev;
struct mlx5_priv *priv;
net/mlx5: rename Verbs shared object Replace all 'mlx5_ibv_shared' appearances with 'mlx5_dev_ctx_shared'. Signed-off-by: Ophir Munk <ophirmu@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com>
2020-06-03 15:05:55 +00:00
struct mlx5_dev_ctx_shared *sh;
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
struct mlx5_mr *mr;
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
struct mr_cache_entry entry;
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
dev = pci_dev_to_eth_dev(pdev);
if (!dev) {
DRV_LOG(WARNING, "unable to find matching ethdev "
"to PCI device %p", (void *)pdev);
rte_errno = ENODEV;
return -1;
}
priv = dev->data->dev_private;
net/mlx5: share Memory Regions for multiport device The multiport Infiniband device support was introduced [1]. All active ports, belonging to the same Infiniband device use the single shared Infiniband context of that device and share the resources: - QPs are created within shared context - Verbs flows are also created with specifying port index - DV/DR resources - Protection Domain - Event Handlers This patchset adds support for Memory Regions sharing between ports, created on the base of multiport Infiniband device. The datapath of mlx5 uses the layered cache subsystem for allocating/releasing Memory Regions, only the lowest layer L3 is subject to share due to performance issues. [1] http://patches.dpdk.org/cover/51800/ Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com> Acked-by: Yongseok Koh <yskoh@mellanox.com>
2019-04-27 04:32:56 +00:00
sh = priv->sh;
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
rte_rwlock_read_lock(&sh->share_cache.rwlock);
mr = mlx5_mr_lookup_list(&sh->share_cache, &entry, (uintptr_t)addr);
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
if (!mr) {
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
rte_rwlock_read_unlock(&sh->share_cache.rwlock);
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
DRV_LOG(WARNING, "address 0x%" PRIxPTR " wasn't registered "
"to PCI device %p", (uintptr_t)addr,
(void *)pdev);
rte_errno = EINVAL;
return -1;
}
LIST_REMOVE(mr, mr);
common/mlx5: free MR resource on device DMA unmap mlx5 PMD created the MR (Memory Region) resource on the mlx5_dma_map call to make the memory available for DMA operations. On the mlx5_dma_unmap call the MR resource was not freed but inserted to MR Free list for further garbage collection. Actual MR resource destroying happened on device stop call. That caused the runtime out of memory in case of application performed multiple DMA map/unmap calls. The fix immediately frees the MR resource on mlx5_dma_unmap call not engaging the list. The export for mlx5_mr_free function from common PMD part is added as well. Fixes: 989e999d9305 ("net/mlx5: support PCI device DMA map and unmap") Cc: stable@dpdk.org Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2020-11-02 02:22:28 +00:00
mlx5_mr_free(mr, sh->share_cache.dereg_mr_cb);
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
DEBUG("port %u remove MR(%p) from list", dev->data->port_id,
(void *)mr);
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
mlx5_mr_rebuild_cache(&sh->share_cache);
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
/*
* Flush local caches by propagating invalidation across cores.
* rte_smp_wmb() is enough to synchronize this event. If one of
* freed memsegs is seen by other core, that means the memseg
* has been allocated by allocator, which will come after this
* free call. Therefore, this store instruction (incrementing
* generation below) will be guaranteed to be seen by other core
* before the core sees the newly allocated memory.
*/
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
++sh->share_cache.dev_gen;
DEBUG("broadcasting local cache flush, gen=%d",
sh->share_cache.dev_gen);
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
rte_smp_wmb();
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
rte_rwlock_read_unlock(&sh->share_cache.rwlock);
net/mlx5: support PCI device DMA map and unmap The implementation reuses the external memory registration work done by commit[1]. Note about representors: The current representor design will not work with those map and unmap functions. The reason is that for representors we have multiple IB devices share the same PCI function, so mapping will happen only on one of the representors and not all of them. While it is possible to implement such support, the IB representor design is going to be changed during DPDK19.05. The new design will have a single IB device for all representors, hence sharing of a single memory region between all representors will be possible. [1] commit 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
2019-03-10 08:28:02 +00:00
return 0;
}
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
/**
* Register MR for entire memory chunks in a Mempool having externally allocated
* memory and fill in local cache.
*
* @param dev
* Pointer to Ethernet device.
* @param mr_ctrl
* Pointer to per-queue MR control structure.
* @param mp
* Pointer to registering Mempool.
*
* @return
* 0 on success, -1 on failure.
*/
static uint32_t
mlx5_mr_update_ext_mp(struct rte_eth_dev *dev, struct mlx5_mr_ctrl *mr_ctrl,
struct rte_mempool *mp)
{
struct mr_update_mp_data data = {
.dev = dev,
.mr_ctrl = mr_ctrl,
.ret = 0,
};
rte_mempool_mem_iter(mp, mlx5_mr_update_ext_mp_cb, &data);
return data.ret;
}
/**
* Register MR entire memory chunks in a Mempool having externally allocated
* memory and search LKey of the address to return.
*
* @param dev
* Pointer to Ethernet device.
* @param addr
* Search key.
* @param mp
* Pointer to registering Mempool where addr belongs.
*
* @return
* LKey for address on success, UINT32_MAX on failure.
*/
uint32_t
mlx5_tx_update_ext_mp(struct mlx5_txq_data *txq, uintptr_t addr,
struct rte_mempool *mp)
{
struct mlx5_txq_ctrl *txq_ctrl =
container_of(txq, struct mlx5_txq_ctrl, txq);
struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
net/mlx: prefix private structure The private structure stored in rte_eth_dev->data->dev_private was named "struct priv". In order to ease code browsing, the structure is renamed "struct mlx[45]_priv". Cc: stable@dpdk.org Signed-off-by: Thomas Monjalon <thomas@monjalon.net> Acked-by: Yongseok Koh <yskoh@mellanox.com>
2019-02-21 09:29:14 +00:00
struct mlx5_priv *priv = txq_ctrl->priv;
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
net/mlx5: fix external memory registration Secondary process is not allowed to register MR due to a restriction of library and kernel driver. Fixes: 7e43a32ee060 ("net/mlx5: support externally allocated static memory") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com> Acked-by: Shahaf Shuler <shahafs@mellanox.com>
2019-04-01 21:17:53 +00:00
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
DRV_LOG(WARNING,
"port %u using address (%p) from unregistered mempool"
" having externally allocated memory"
" in secondary process, please create mempool"
" prior to rte_eth_dev_start()",
PORT_ID(priv), (void *)addr);
return UINT32_MAX;
}
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
mlx5_mr_update_ext_mp(ETH_DEV(priv), mr_ctrl, mp);
return mlx5_tx_addr2mr_bh(txq, addr);
}
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
/* Called during rte_mempool_mem_iter() by mlx5_mr_update_mp(). */
static void
mlx5_mr_update_mp_cb(struct rte_mempool *mp __rte_unused, void *opaque,
struct rte_mempool_memhdr *memhdr,
unsigned mem_idx __rte_unused)
{
struct mr_update_mp_data *data = opaque;
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
struct rte_eth_dev *dev = data->dev;
struct mlx5_priv *priv = dev->data->dev_private;
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
uint32_t lkey;
/* Stop iteration if failed in the previous walk. */
if (data->ret < 0)
return;
/* Register address of the chunk and update local caches. */
common/mlx5: refactor memory management Refactor common memory btree and cache management to common driver. Replace some input parameters of MR APIs to more common data structure like PD, port_id, share_cache,... so that multiple PMD drivers can use those MR APIs. Modify mlx5 net pmd driver to use MR management APIs from common driver. Signed-off-by: Vu Pham <vuhuong@mellanox.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
2020-04-13 21:17:48 +00:00
lkey = mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id,
&priv->sh->share_cache, data->mr_ctrl,
(uintptr_t)memhdr->addr,
priv->config.mr_ext_memseg_en);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
if (lkey == UINT32_MAX)
data->ret = -1;
}
/**
* Register entire memory chunks in a Mempool.
*
* @param dev
* Pointer to Ethernet device.
* @param mr_ctrl
* Pointer to per-queue MR control structure.
* @param mp
* Pointer to registering Mempool.
*
* @return
* 0 on success, -1 on failure.
*/
int
mlx5_mr_update_mp(struct rte_eth_dev *dev, struct mlx5_mr_ctrl *mr_ctrl,
struct rte_mempool *mp)
{
struct mr_update_mp_data data = {
.dev = dev,
.mr_ctrl = mr_ctrl,
.ret = 0,
};
net/mlx5: register multiple pool for Rx queue The split feature for receiving packets was added to the mlx5 PMD, now Rx queue can receive the data to the buffers belonging to the different pools and the memory of all the involved pool must be registered for DMA operations in order to allow hardware to store the data. Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> Acked-by: Matan Azrad <matan@nvidia.com>
2020-10-26 11:55:02 +00:00
DRV_LOG(DEBUG, "Port %u Rx queue registering mp %s "
"having %u chunks.", dev->data->port_id,
mp->name, mp->nb_mem_chunks);
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
rte_mempool_mem_iter(mp, mlx5_mr_update_mp_cb, &data);
net/mlx5: support externally allocated static memory When MLX PMD registers memory for DMA, it accesses the global memseg list of DPDK to maximize the range of registration so that LKey search can be more efficient. Granularity of MR registration is per page. Externally allocated memory shouldn't be used for DMA because it can't be searched in the memseg list and free event can't be tracked by DPDK. If it is used, the following error will occur: net_mlx5: port 0 unable to find virtually contiguous chunk for address (0x5600017587c0). rte_memseg_contig_walk() failed. There's a pending patchset [1] which enables externally allocated memory. Once it is merged, users can register their own memory out of EAL then that will resolve this issue. Meanwhile, if the external memory is static (allocated on startup and never freed), such memory can also be registered by little tweak in the code. [1] http://patches.dpdk.org/project/dpdk/list/?series=1415 This patch is not a bug fix but needs to be included in stable versions. Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-09-24 18:36:47 +00:00
if (data.ret < 0 && rte_errno == ENXIO) {
/* Mempool may have externally allocated memory. */
return mlx5_mr_update_ext_mp(dev, mr_ctrl, mp);
}
net/mlx5: add new memory region support This is the new design of Memory Region (MR) for mlx PMD, in order to: - Accommodate the new memory hotplug model. - Support non-contiguous Mempool. There are multiple layers for MR search. L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized array by linear search. L0/L1 is in an inline function - mlx5_mr_lookup_cache(). If L1 misses, the bottom-half function is called to look up the address from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh() and it is not an inline function. Data structure for L2 is the Binary Tree. If L2 misses, the search falls into the slowest path which takes locks in order to access global device cache (priv->mr.cache) which is also a B-tree and caches the original MR list (priv->mr.mr_list) of the device. Unless the global cache is overflowed, it is all-inclusive of the MR list. This is L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and can't be expanded on the fly due to deadlock. Refer to the comments in the code for the details - mr_lookup_dev(). If L3 is overflowed, the list will have to be searched directly bypassing the cache although it is slower. If L3 misses, a new MR for the address should be created - mlx5_mr_create(). When it creates a new MR, it tries to register adjacent memsegs as much as possible which are virtually contiguous around the address. This must take two locks - memory_hotplug_lock and priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any allocation/free of memory inside. In the free callback of the memory hotplug event, freed space is searched from the MR list and corresponding bits are cleared from the bitmap of MRs. This can fragment a MR and the MR will have multiple search entries in the caches. Once there's a change by the event, the global cache must be rebuilt and all the per-queue caches will be flushed as well. If memory is frequently freed in run-time, that may cause jitter on dataplane processing in the worst case by incurring MR cache flush and rebuild. But, it would be the least probable scenario. To guarantee the most optimal performance, it is highly recommended to use an EAL option - '--socket-mem'. Then, the reserved memory will be pinned and won't be freed dynamically. And it is also recommended to configure per-lcore cache of Mempool. Even though there're many MRs for a device or MRs are highly fragmented, the cache of Mempool will be much helpful to reduce misses on per-queue caches anyway. '--legacy-mem' is also supported. Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
2018-05-09 11:09:04 +00:00
return data.ret;
}
Reference in New Issue Copy Permalink