dpdkbuild: build dpdk with meson+ninja

Makefile support in DPDK was deprecated and will be removed soon,
so switch to the officially supported way of building DPDK -
with meson and ninja. Two new tools. Basically, our Makefiles
will invoke meson+ninja for DPDK, no other SPDK components are
affected.

Apparently DPDK wanted to move away from an octopus-like config
system and the ideology behind meson configuration is simple now:
build everything by default. Some PMDs can be explicitly disabled
with meson command line, but all libraries (both static and shared
versions) and test apps are built unconditionally.

How long does it take to build minimal DPDK with meson? Too much.
On my machine half of the total build time is spent on libraries
we don't need at all. (I have some hacks up my sleeve to disable
building those libraries - see the subsequent patch.) As for the
official way of building a minimal DPDK, there was a patch [1]
on dpdk mailing list to introduce more specific configuration,
but it was rejected:

> We talked about this a few times in the past, and it was actually one
> of the design goals to _avoid_ replicating the octopus-like config
> system of the makefiles. That's because it makes the test matrix
> insanely complicated, not to mention the harm to user friendliness,
> among other things.
>
> If someone doesn't want to use a PMD, they can just avoid installing it
> - it's simple enough.
>
> Sorry, but from me it's a very strong NACK.

Let's not follow that direction, hack the DPDK build system instead.

As for advantages of meson+ninja over Makefiles? I can't find any.
It's another build system that does a lot for you with some functions,
magic options, and a built-in dependency system. It seems nice if you know
the syntax, but it's another component that you need to learn, debug,
and possibly find bugs in (there's a lot of github issues open for meson).
I would compare it to CMake.

As for changes in this patch: rather that explicitly disabling
PMDs we don't need, specify a list of PMDs we do need and disable
everything else found in ./dpdk/drivers/*. This way we won't have
to disable the new PMDs as they're added to DPDK.

Meson configuration also sets RTE_EAL_PMD_PATH #define to a valid directory
with built PMD shared libs. When it's set, DPDK dynamically loads all shared
libraries inside. The drivers there depend on DPDK shared libs and fail to
load in static SPDK builds, so we disable them altogether by unsetting
RTE_EAL_PMD_PATH in the meson-generated config file - just like
DPDK Makefiles did. EAL checks for RTE_EAL_PMD_PATH being empty and skips
loading any external PMDs then. We do it for both static and shared libs.
We specify all PMDs at build time for now, so there's just no need to load
them dynamically.

We have three more hacks in our submodule:
 * disable building dpdk apps by commenting-out a line in dpdk/meson.build
 * disable building unnecessary libs (build everything that spdk *may*
   need)
 * build isa-l compress pmd with `-L[...] -lisal`. DPDK expects to find
   libisal with pkg-config. We don't want to prepare a pkg-config file,
   so comment-out a failing check in another meson.build file and provide
   isa-l through CFLAGS and LDFLAGS.

We also need to make some changes to our test/external_code. First of
all, -ldpdk is no more. Meson build generates a pkg-config file with all
libs, but we'll switch to it in a separate patch - for now just specify
all -lrte_ libs one by one. -Wl,--no-as-needed has to be added to some
test cases, otherwise rte_mempool_ring isn't loaded. We don't use any
APIs from this library, it only has a static constructor that provides
a few callbacks used by rte_mempool_create(). Also, since DPDK now builds
both static and shared libraries, we need to add -Wl,-Bstatic to force
using static libswhere required. It's only needed for DPDK libs, but we
use it for SPDK libs as well since there's no harm.

As for performance:
$ ./configure --enable-debug --with-crypto --with-reduce
$ time make -j40 -C dpdkbuild all
with meson:
real    0m8.287s
user    1m7.983s
sys     0m10.548s

before, with the old DPDK makefiles:
real    0m20.232s
user    0m55.921s
sys     0m16.491s

The subsequent builds are much faster too:
$ time make -j40 -C dpdkbuild all
meson:
real    0m0.876s
user    0m0.663s
sys     0m0.217s

makefiles:
real    0m10.150s
user    0m11.740s
sys     0m6.772s

[1] http://inbox.dpdk.org/dev/1a07d1cd59d84dce84e56c10fdabf5e5504560a6.camel@debian.org/

Change-Id: Ic65db563014100bafb12e61ee0530cc2ae64401d
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1440
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This commit is contained in:
Darek Stojaczyk 2020-03-23 17:55:08 +01:00 committed by Tomasz Zawadzki
parent c8c899f492
commit 6b41a08654
4 changed files with 102 additions and 89 deletions

View File

@ -49,6 +49,9 @@ DIRS-$(CONFIG_ISAL) += isalbuild
cc_version cxx_version .libs_only_other .ldflags ldflags install \
uninstall
# Workaround for ninja. See dpdkbuild/Makefile
export MAKE_PID := $(shell echo $$PPID)
ifeq ($(SPDK_ROOT_DIR)/lib/env_dpdk,$(CONFIG_ENV))
ifeq ($(CURDIR)/dpdk/build,$(CONFIG_DPDK_DIR))
ifneq ($(SKIP_DPDK_BUILD),1)

View File

@ -36,87 +36,39 @@ include $(SPDK_ROOT_DIR)/mk/spdk.common.mk
.PHONY: all clean install uninstall
DPDK_FRAMEWORK = n
DPDK_OPTS =
DPDK_OPTS = -Denable_docs=false
DPDK_CFLAGS =
DPDK_OPTS += CONFIG_RTE_BUILD_SHARED_LIB=n
DPDK_OPTS += CONFIG_RTE_LIBRTE_PMD_AESNI_MB=n
DPDK_OPTS += CONFIG_RTE_LIBRTE_REORDER=n
DPDK_OPTS += CONFIG_RTE_LIBRTE_ETHER=n
DPDK_OPTS += CONFIG_RTE_LIBRTE_CMDLINE=n
DPDK_OPTS += CONFIG_RTE_LIBRTE_METER=n
DPDK_OPTS += CONFIG_RTE_LIBRTE_HASH=n
DPDK_OPTS += CONFIG_RTE_LIBRTE_VHOST=n
DPDK_OPTS += CONFIG_RTE_EAL_IGB_UIO=n
DPDK_OPTS += CONFIG_RTE_LIBRTE_PMD_QAT=n
DPDK_OPTS += CONFIG_RTE_LIBRTE_PMD_QAT_SYM=n
DPDK_OPTS += CONFIG_RTE_LIBRTE_PMD_ISAL=n
ifeq ($(CONFIG_SHARED),y)
DPDK_OPTS += CONFIG_RTE_BUILD_SHARED_LIB=y
DPDK_LDFLAGS+= -rpath $(SPDK_ROOT_DIR)/dpdk/build/lib
DPDK_KMODS = false
ifeq ($(CONFIG_IGB_UIO_DRIVER),y)
DPDK_KMODS = true
endif
ifeq ($(OS),FreeBSD)
DPDK_KMODS = true
endif
DPDK_OPTS += -Denable_kmods=$(DPDK_KMODS)
# the drivers we use
DPDK_DRIVERS = bus bus/pci bus/vdev mempool/ring
ifeq ($(CONFIG_CRYPTO),y)
DPDK_FRAMEWORK = y
DPDK_OPTS += CONFIG_RTE_LIBRTE_PMD_AESNI_MB=y
DPDK_OPTS += CONFIG_RTE_LIBRTE_REORDER=y
# crypto/qat is just a stub, the compress/qat pmd is used instead
DPDK_DRIVERS += crypto crypto/aesni_mb crypto/qat compress/qat common/qat
DPDK_CFLAGS += -I$(IPSEC_MB_DIR)
DPDK_LDFLAGS += -L$(IPSEC_MB_DIR)
endif
ifeq ($(CONFIG_REDUCE),y)
DPDK_FRAMEWORK = y
DPDK_OPTS += CONFIG_RTE_LIBRTE_PMD_ISAL=y
DPDK_DRIVERS += compress compress/isal compress/qat common/qat
DPDK_CFLAGS += -I$(ISAL_DIR)
DPDK_LDFLAGS += -L$(ISAL_DIR)/.libs
DPDK_LDFLAGS += -L$(ISAL_DIR)/.libs -lisal
endif
ifeq ($(CONFIG_VHOST),y)
DPDK_OPTS += CONFIG_RTE_LIBRTE_ETHER=y
DPDK_OPTS += CONFIG_RTE_LIBRTE_CMDLINE=y
DPDK_OPTS += CONFIG_RTE_LIBRTE_METER=y
DPDK_OPTS += CONFIG_RTE_LIBRTE_HASH=y
DPDK_OPTS += CONFIG_RTE_LIBRTE_VHOST=y
endif
ifeq ($(CONFIG_IGB_UIO_DRIVER),y)
DPDK_OPTS += CONFIG_RTE_EAL_IGB_UIO=y
endif
ifeq ($(CONFIG_RAID5),y)
DPDK_OPTS += CONFIG_RTE_LIBRTE_HASH=y
endif
ifeq ($(DPDK_FRAMEWORK),y)
DPDK_OPTS += CONFIG_RTE_LIBRTE_PMD_QAT=y
DPDK_OPTS += CONFIG_RTE_LIBRTE_PMD_QAT_SYM=y
endif
ifeq ($(TARGET_MACHINE),aarch64)
DPDK_CONFIG := arm64-armv8a
else
DPDK_CONFIG := $(TARGET_MACHINE)-native
endif
DPDK_OPTS += -Dmachine=$(TARGET_ARCHITECTURE)
ifneq ($(CONFIG_CROSS_PREFIX),)
DPDK_OPTS += CROSS=$(CONFIG_CROSS_PREFIX)-
endif
ifeq ($(OS),Linux)
DPDK_CONFIG := $(DPDK_CONFIG)-linuxapp
NPROC := $(shell nproc)
else
ifeq ($(OS),FreeBSD)
DPDK_CONFIG := $(DPDK_CONFIG)-bsdapp
NPROC := $(shell sysctl hw.ncpu | awk '{print $$NF}')
endif
endif
ifeq ($(CC_TYPE),clang)
DPDK_CONFIG := $(DPDK_CONFIG)-clang
else
DPDK_CONFIG := $(DPDK_CONFIG)-gcc
$(error Automatic DPDK cross build is not supported. Please compile DPDK manually \
with e.g. `meson build --cross-file config/arm/arm64_armv8_linux_gcc`)
endif
DPDK_CFLAGS += -fPIC
@ -154,15 +106,58 @@ endif
# Force-disable scan-build
SUB_CC = $(patsubst %ccc-analyzer,$(DEFAULT_CC),$(CC))
$(SPDK_ROOT_DIR)/dpdk/build: $(SPDK_ROOT_DIR)/mk/cc.mk $(SPDK_ROOT_DIR)/include/spdk/config.h
$(Q)rm -rf $(SPDK_ROOT_DIR)/dpdk/build
$(Q)$(MAKE) -C $(SPDK_ROOT_DIR)/dpdk config T=$(DPDK_CONFIG) $(DPDK_OPTS)
DPDK_ALL_DRIVER_DIRS = $(shell find $(SPDK_ROOT_DIR)/dpdk/drivers -mindepth 1 -type d)
DPDK_ALL_DRIVERS = $(DPDK_ALL_DRIVER_DIRS:$(SPDK_ROOT_DIR)/dpdk/drivers/%=%)
DPDK_DISABLED_DRVERS = $(filter-out $(DPDK_DRIVERS),$(DPDK_ALL_DRIVERS))
all: $(SPDK_ROOT_DIR)/dpdk/build
$(Q)$(MAKE) -C $(SPDK_ROOT_DIR)/dpdk/build EXTRA_CFLAGS="$(DPDK_CFLAGS)" EXTRA_LDFLAGS="$(DPDK_LDFLAGS)" CC=$(SUB_CC) T="$(DPDK_CONFIG)" $(DPDK_OPTS)
ifeq ($(OS),Linux)
SED_INPLACE_FLAG = "-i"
MESON_PREFIX = $(SPDK_ROOT_DIR)/dpdk/build
else
SED_INPLACE_FLAG = "-i ''"
MESON_PREFIX = "/"
endif
# Some ninja versions come with a (broken?) jobserver which defaults to use
# only 1 thread for the build. We workaround this by specifying -j to ninja
# with the same value as top-makefile. This is OK as long as DPDK is not built
# in parralel with anything else, which is the case for now.
ifeq ($(MAKE_PID),)
MAKE_PID := $(shell echo $$PPID)
endif
MAKE_NUMJOBS := $(shell ps T | sed -nE 's/\s*$(MAKE_PID)\s.* (-j|--jobs=)( *[0-9]+).*/\1\2/p')
all: $(SPDK_ROOT_DIR)/dpdk/build-tmp
$(Q)# DPDK doesn't handle nested make calls, so unset MAKEFLAGS
$(Q)env -u MAKEFLAGS ninja -C $(SPDK_ROOT_DIR)/dpdk/build-tmp $(MAKE_NUMJOBS)
$(Q) \
# Meson on FreeBSD sometimes appends --prefix value to the default DESTDIR (which is e.g. \
# /usr/local) instead of replacing it. --prefix needs to be an absolute path, so we set \
# it to / and then set DESTDIR directly, so libs and headers are copied to "DESTDIR//". \
# DPDK kernel modules are set to install in $DESTDIR/boot/modules, but we move them \
# to DESTDIR/kmod to be consistent with the makefile build.
$(Q)if [ "$(OS)" = "FreeBSD" ]; then \
env -u MAKEFLAGS DESTDIR=$(SPDK_ROOT_DIR)/dpdk/build ninja -C $(SPDK_ROOT_DIR)/dpdk/build-tmp $(MAKE_NUMJOBS) install > /dev/null && \
mv $(SPDK_ROOT_DIR)/dpdk/build/boot/modules $(SPDK_ROOT_DIR)/dpdk/build/kmod; \
else \
env -u MAKEFLAGS ninja -C $(SPDK_ROOT_DIR)/dpdk/build-tmp $(MAKE_NUMJOBS) install > /dev/null; \
fi
$(SPDK_ROOT_DIR)/dpdk/build-tmp: $(SPDK_ROOT_DIR)/mk/cc.mk $(SPDK_ROOT_DIR)/include/spdk/config.h
$(Q)rm -rf $(SPDK_ROOT_DIR)/dpdk/build $(SPDK_ROOT_DIR)/dpdk/build-tmp
$(Q)cd "$(SPDK_ROOT_DIR)/dpdk"; CC="$(SUB_CC)" meson --prefix="$(MESON_PREFIX)" --libdir lib -Dc_args="$(DPDK_CFLAGS)" -Dc_link_args="$(DPDK_LDFLAGS)" $(DPDK_OPTS) -Ddisable_drivers="$(shell echo $(DPDK_DISABLED_DRVERS) | sed -E "s/ +/,/g")" build-tmp
$(Q)sed $(SED_INPLACE_FLAG) 's/#define RTE_EAL_PMD_PATH .*/#define RTE_EAL_PMD_PATH ""/g' $(SPDK_ROOT_DIR)/dpdk/build-tmp/rte_build_config.h
$(Q) \
# TODO Meson build adds libbsd dependency when it's available. This means any app will be \
# forced to link with -lbsd, but only if it's available on the system. The clean way to \
# handle this would be to rely on DPDK's pkg-config file which will contain the -lbsd when \
# required. For now just remove the libbsd dependency. DPDK will fallback to its internal \
# functions.
$(Q)sed $(SED_INPLACE_FLAG) 's/#define RTE_USE_LIBBSD .*//g' $(SPDK_ROOT_DIR)/dpdk/build-tmp/rte_build_config.h
clean:
$(Q)rm -rf $(SPDK_ROOT_DIR)/dpdk/build
$(Q)rm -rf $(SPDK_ROOT_DIR)/dpdk/build $(SPDK_ROOT_DIR)/dpdk/build-tmp
install:
@:

View File

@ -48,10 +48,10 @@ DPDK_INC_DIR := $(DPDK_ABS_DIR)/include/dpdk
endif
DPDK_INC := -I$(DPDK_INC_DIR)
ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_eal.a))
DPDK_LIB_EXT = .a
else
ifeq ($(CONFIG_SHARED),y)
DPDK_LIB_EXT = .so
else
DPDK_LIB_EXT = .a
endif
DPDK_LIB_LIST = rte_eal rte_mempool rte_ring rte_mbuf
@ -94,7 +94,7 @@ endif
ifeq ($(CONFIG_REDUCE),y)
DPDK_FRAMEWORK=y
DPDK_LIB_LIST += rte_pmd_isal_comp
DPDK_LIB_LIST += rte_pmd_isal
endif
ifeq ($(DPDK_FRAMEWORK),y)
@ -144,7 +144,8 @@ else
ENV_DPDK_FILE = $(call spdk_lib_list_to_static_libs,env_dpdk)
endif
ENV_LIBS = $(ENV_DPDK_FILE) $(DPDK_LIB)
ENV_LINKER_ARGS = $(call dpdk_env_linker_args,$(DPDK_LIB_LIST))
ENV_LINKER_ARGS = -Wl,-rpath-link $(DPDK_ABS_DIR)/lib
ENV_LINKER_ARGS += $(call dpdk_env_linker_args,$(DPDK_LIB_LIST))
ifeq ($(CONFIG_IPSEC_MB),y)
ENV_LINKER_ARGS += -lIPSec_MB -L$(IPSEC_MB_DIR)
@ -160,6 +161,13 @@ ENV_LINKER_ARGS += -lnuma
endif
endif
# DPDK built with meson puts those defines elsewhere
ifneq (,$(wildcard $(DPDK_INC_DIR)/rte_build_config.h))
ifneq (,$(shell grep -e "define RTE_LIBRTE_VHOST_NUMA 1" -e "define RTE_EAL_NUMA_AWARE_HUGEPAGES 1" $(DPDK_INC_DIR)/rte_build_config.h))
ENV_LINKER_ARGS += -lnuma
endif
endif
ifeq ($(OS),Linux)
ENV_LINKER_ARGS += -ldl
endif

View File

@ -31,36 +31,43 @@
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# Shows how to compile both an external bdev and an external application against the SPDK combined shared object and dpdk shared object.
# Shows how to compile both an external bdev and an external application against the SPDK combined shared object and dpdk shared objects.
bdev_shared_combo:
$(CC) $(COMMON_CFLAGS) -L../passthru -o hello_bdev ./hello_bdev.c -lpassthru_external -lspdk -lspdk_env_dpdk -ldpdk
$(CC) $(COMMON_CFLAGS) -L../passthru -Wl,-rpath=$(SPDK_LIB_DIR),--no-as-needed -o hello_bdev ./hello_bdev.c -lpassthru_external \
-lspdk -lspdk_env_dpdk -lrte_eal -lrte_mempool -lrte_ring -lrte_mbuf -lrte_mempool_ring -lrte_pci -lrte_bus_pci -lrte_kvargs \
-lrte_vhost -lrte_net -lrte_hash -lrte_cryptodev -Wl,--no-whole-archive
# Shows how to compile both an external bdev and an external application against the SPDK individual shared objects and dpdk shared object.
# Shows how to compile both an external bdev and an external application against the SPDK individual shared objects and dpdk shared objects.
bdev_shared_iso:
$(CC) $(COMMON_CFLAGS) -L../passthru -Wl,-rpath=$(SPDK_LIB_DIR),--no-as-needed -o hello_bdev ./hello_bdev.c \
-lpassthru_external -lspdk_event_bdev -lspdk_bdev -lspdk_bdev_malloc -lspdk_log -lspdk_thread -lspdk_util -lspdk_event -lspdk_env_dpdk -ldpdk
-lpassthru_external -lspdk_event_bdev -lspdk_event_accel -lspdk_event_vmd -lspdk_bdev -lspdk_bdev_malloc -lspdk_log -lspdk_thread -lspdk_util -lspdk_event \
-lspdk_env_dpdk -lrte_eal -lrte_mempool -lrte_ring -lrte_mbuf -lrte_mempool_ring -lrte_pci -lrte_bus_pci -lrte_kvargs \
-lrte_vhost -lrte_net -lrte_hash -lrte_cryptodev -Wl,--no-whole-archive -lnuma
# Shows how to compile an external application against the SPDK combined shared object and dpdk shared object.
# Shows how to compile an external application against the SPDK combined shared object and dpdk shared objects.
alone_shared_combo:
$(CC) $(COMMON_CFLAGS) -o hello_bdev ./hello_bdev.c -lspdk -lspdk_env_dpdk -ldpdk
$(CC) $(COMMON_CFLAGS) -Wl,-rpath=$(SPDK_LIB_DIR),--no-as-needed -o hello_bdev ./hello_bdev.c -lspdk -lspdk_env_dpdk -lrte_eal \
-lrte_mempool -lrte_ring -lrte_mbuf -lrte_mempool_ring -lrte_pci -lrte_bus_pci -lrte_kvargs -lrte_vhost -lrte_net -lrte_hash -lrte_cryptodev
# Shows how to compile an external application against the SPDK individual shared objects and dpdk shared object.
# Shows how to compile an external application against the SPDK individual shared objects and dpdk shared objects.
alone_shared_iso:
$(CC) $(COMMON_CFLAGS) -Wl,-rpath=$(SPDK_LIB_DIR),--no-as-needed -o hello_bdev ./hello_bdev.c -lspdk_event_bdev \
-lspdk_bdev -lspdk_bdev_malloc -lspdk_log -lspdk_thread -lspdk_util -lspdk_event -lspdk_env_dpdk -ldpdk
-lspdk_event_accel -lspdk_event_vmd -lspdk_bdev -lspdk_bdev_malloc -lspdk_log -lspdk_thread -lspdk_util -lspdk_event \
-lspdk_env_dpdk -lrte_eal -lrte_mempool -lrte_ring -lrte_mbuf -lrte_mempool_ring -lrte_pci -lrte_bus_pci -lrte_kvargs \
-lrte_vhost -lrte_net -lrte_hash -lrte_cryptodev
# Shows how to compile an external application against the SPDK archives.
alone_static:
$(CC) $(COMMON_CFLAGS) -o hello_bdev ./hello_bdev.c -Wl,--whole-archive -lspdk_bdev_malloc -lspdk_event_bdev -lspdk_event_accel -lspdk_event_vmd \
$(CC) $(COMMON_CFLAGS) -o hello_bdev ./hello_bdev.c -Wl,--whole-archive,-Bstatic -lspdk_bdev_malloc -lspdk_event_bdev -lspdk_event_accel -lspdk_event_vmd \
-lspdk_event_sock -lspdk_bdev -lspdk_accel -lspdk_event -lspdk_thread -lspdk_util -lspdk_conf -lspdk_trace -lspdk_log -lspdk_json \
-lspdk_jsonrpc -lspdk_rpc -lspdk_sock -lspdk_notify -lspdk_vmd -lspdk_env_dpdk -lrte_eal -lrte_mempool -lrte_ring \
-lrte_mbuf -lrte_mempool_ring -lrte_pci -lrte_bus_pci -lrte_kvargs -lrte_vhost -lrte_net -lrte_hash -lrte_telemetry \
-lrte_cryptodev -Wl,--no-whole-archive -lnuma -luuid -lpthread -ldl -lrt
-lrte_cryptodev -Wl,--no-whole-archive,-Bdynamic -lnuma -luuid -lpthread -ldl -lrt
# Shows how to compile and external bdev and application sgainst the SPDK archives.
bdev_static:
$(CC) $(COMMON_CFLAGS) -L../passthru -o hello_bdev ./hello_bdev.c -Wl,--whole-archive -lpassthru_external -lspdk_bdev_malloc -lspdk_event_bdev \
$(CC) $(COMMON_CFLAGS) -L../passthru -o hello_bdev ./hello_bdev.c -Wl,--whole-archive,-Bstatic -lpassthru_external -lspdk_bdev_malloc -lspdk_event_bdev \
-lspdk_event_accel -lspdk_event_vmd -lspdk_event_sock -lspdk_bdev -lspdk_accel -lspdk_event -lspdk_thread -lspdk_util -lspdk_conf -lspdk_trace \
-lspdk_log -lspdk_json -lspdk_jsonrpc -lspdk_rpc -lspdk_sock -lspdk_notify -lspdk_vmd -lspdk_env_dpdk -lrte_eal -lrte_mempool \
-lrte_ring -lrte_mbuf -lrte_mempool_ring -lrte_pci -lrte_bus_pci -lrte_kvargs -lrte_vhost -lrte_net -lrte_hash -lrte_telemetry -lrte_cryptodev \
-Wl,--no-whole-archive -lnuma -luuid -lpthread -ldl -lrt
-Wl,--no-whole-archive,-Bdynamic -lnuma -luuid -lpthread -ldl -lrt