d/numam-dpdk
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
44a87aa9c2
numam-dpdk/lib/regexdev/meson.build

8 lines
244 B
Meson
Raw Normal View History

regexdev: introduce API As RegEx usage become more used by DPDK applications, for example: * Next Generation Firewalls (NGFW) * Deep Packet and Flow Inspection (DPI) * Intrusion Prevention Systems (IPS) * DDoS Mitigation * Network Monitoring * Data Loss Prevention (DLP) * Smart NICs * Grammar based content processing * URL, spam and adware filtering * Advanced auditing and policing of user/application security policies * Financial data mining - parsing of streamed financial feeds * Application recognition. * Dmemory introspection. * Natural Language Processing (NLP) * Sentiment Analysis. * Big data database acceleration. * Computational storage. Number of PMD providers started to work on HW implementation, along side with SW implementations. This lib adds the support for those kind of devices. The RegEx Device API is composed of two parts: - The application-oriented RegEx API that includes functions to setup a RegEx device (configure it, setup its queue pairs and start it), update the rule database and so on. - The driver-oriented RegEx API that exports a function allowing a RegEx poll Mode Driver (PMD) to simultaneously register itself as a RegEx device driver. RegEx device components and definitions: +-----------------+ | | | o---------+ rte_regexdev_[en|de]queue_burst() | PCRE based o------+ | | | RegEx pattern | | | +--------+ | | matching engine o------+--+--o | | +------+ | | | | | queue |<==o===>|Core 0| | o----+ | | | pair 0 | | | | | | | | +--------+ +------+ +-----------------+ | | | ^ | | | +--------+ | | | | | | +------+ | | +--+--o queue |<======>|Core 1| Rule|Database | | | pair 1 | | | +------+----------+ | | +--------+ +------+ | Group 0 | | | | +-------------+ | | | +--------+ +------+ | | Rules 0..n | | | | | | |Core 2| | +-------------+ | | +--o queue |<======>| | | Group 1 | | | pair 2 | +------+ | +-------------+ | | +--------+ | | Rules 0..n | | | | +-------------+ | | +--------+ | Group 2 | | | | +------+ | +-------------+ | | | queue |<======>|Core n| | | Rules 0..n | | +-------o pair n | | | | +-------------+ | +--------+ +------+ | Group n | | +-------------+ |<-------rte_regexdev_rule_db_update() | | | |<-------rte_regexdev_rule_db_compile_activate() | | Rules 0..n | |<-------rte_regexdev_rule_db_import() | +-------------+ |------->rte_regexdev_rule_db_export() +-----------------+ RegEx: A regular expression is a concise and flexible means for matching strings of text, such as particular characters, words, or patterns of characters. A common abbreviation for this is â~@~\RegExâ~@~]. RegEx device: A hardware or software-based implementation of RegEx device API for PCRE based pattern matching syntax and semantics. PCRE RegEx syntax and semantics specification: http://regexkit.sourceforge.net/Documentation/pcre/pcrepattern.html RegEx queue pair: Each RegEx device should have one or more queue pair to transmit a burst of pattern matching request and receive a burst of receive the pattern matching response. The pattern matching request/response embedded in *rte_regex_ops* structure. Rule: A pattern matching rule expressed in PCRE RegEx syntax along with Match ID and Group ID to identify the rule upon the match. Rule database: The RegEx device accepts regular expressions and converts them into a compiled rule database that can then be used to scan data. Compilation allows the device to analyze the given pattern(s) and pre-determine how to scan for these patterns in an optimized fashion that would be far too expensive to compute at run-time. A rule database contains a set of rules that compiled in device specific binary form. Match ID or Rule ID: A unique identifier provided at the time of rule creation for the application to identify the rule upon match. Group ID: Group of rules can be grouped under one group ID to enable rule isolation and effective pattern matching. A unique group identifier provided at the time of rule creation for the application to identify the rule upon match. Scan: A pattern matching request through *enqueue* API. It may possible that a given RegEx device may not support all the features of PCRE. The application may probe unsupported features through struct rte_regexdev_info::pcre_unsup_flags By default, all the functions of the RegEx Device API exported by a PMD are lock-free functions which assume to not be invoked in parallel on different logical cores to work on the same target object. For instance, the dequeue function of a PMD cannot be invoked in parallel on two logical cores to operates on same RegEx queue pair. Of course, this function can be invoked in parallel by different logical core on different queue pair. It is the responsibility of the upper level application to enforce this rule. In all functions of the RegEx API, the RegEx device is designated by an integer >= 0 named the device identifier *dev_id* At the RegEx driver level, RegEx devices are represented by a generic data structure of type *rte_regexdev*. RegEx devices are dynamically registered during the PCI/SoC device probing phase performed at EAL initialization time. When a RegEx device is being probed, a *rte_regexdev* structure and a new device identifier are allocated for that device. Then, the regexdev_init() function supplied by the RegEx driver matching the probed device is invoked to properly initialize the device. The role of the device init function consists of resetting the hardware or software RegEx driver implementations. If the device init operation is successful, the correspondence between the device identifier assigned to the new device and its associated *rte_regexdev* structure is effectively registered. Otherwise, both the *rte_regexdev* structure and the device identifier are freed. The functions exported by the application RegEx API to setup a device designated by its device identifier must be invoked in the following order: - rte_regexdev_configure() - rte_regexdev_queue_pair_setup() - rte_regexdev_start() Then, the application can invoke, in any order, the functions exported by the RegEx API to enqueue pattern matching job, dequeue pattern matching response, get the stats, update the rule database, get/set device attributes and so on If the application wants to change the configuration (i.e. call rte_regexdev_configure() or rte_regexdev_queue_pair_setup()), it must call rte_regexdev_stop() first to stop the device and then do the reconfiguration before calling rte_regexdev_start() again. The enqueue and dequeue functions should not be invoked when the device is stopped. Finally, an application can close a RegEx device by invoking the rte_regexdev_close() function. Each function of the application RegEx API invokes a specific function of the PMD that controls the target device designated by its device identifier. For this purpose, all device-specific functions of a RegEx driver are supplied through a set of pointers contained in a generic structure of type *regexdev_ops*. The address of the *regexdev_ops* structure is stored in the *rte_regexdev* structure by the device init function of the RegEx driver, which is invoked during the PCI/SoC device probing phase, as explained earlier. In other words, each function of the RegEx API simply retrieves the *rte_regexdev* structure associated with the device identifier and performs an indirect invocation of the corresponding driver function supplied in the *regexdev_ops* structure of the *rte_regexdev* structure. For performance reasons, the address of the fast-path functions of the RegEx driver is not contained in the *regexdev_ops* structure. Instead, they are directly stored at the beginning of the *rte_regexdev* structure to avoid an extra indirect memory access during their invocation. RTE RegEx device drivers do not use interrupts for enqueue or dequeue operation. Instead, RegEx drivers export Poll-Mode enqueue and dequeue functions to applications. The *enqueue* operation submits a burst of RegEx pattern matching request to the RegEx device and the *dequeue* operation gets a burst of pattern matching response for the ones submitted through *enqueue* operation. Typical application utilisation of the RegEx device API will follow the following programming flow. - rte_regexdev_configure() - rte_regexdev_queue_pair_setup() - rte_regexdev_rule_db_update() Needs to invoke if precompiled rule database not provided in rte_regexdev_config::rule_db for rte_regexdev_configure() and/or application needs to update rule database. - rte_regexdev_rule_db_compile_activate() Needs to invoke if rte_regexdev_rule_db_update function was used. - Create or reuse exiting mempool for *rte_regex_ops* objects. - rte_regexdev_start() - rte_regexdev_enqueue_burst() - rte_regexdev_dequeue_burst() Signed-off-by: Jerin Jacob <jerinj@marvell.com> Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com> Signed-off-by: Ori Kam <orika@mellanox.com>
2020-07-06 17:36:46 +00:00
# SPDX-License-Identifier: BSD-3-Clause
# Copyright 2020 Mellanox Technologies, Ltd
sources = files('rte_regexdev.c')
lib: clean up build files Switch from using tabs to 4 spaces for meson.build indentation. Perform other formatting cleanups such as ensure that long lists of files are one per line, and terminating with a final comma before the closing brace to make addition/removals easier. In some cases, reorder lists of items where they were not in alphabetical order. Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
2021-04-20 10:22:24 +00:00
headers = files('rte_regexdev.h', 'rte_regexdev_driver.h')
build: add header includes check To verify that all DPDK headers are ok for inclusion directly in a C file, and are not missing any other pre-requisite headers, we can auto-generate for each header an empty C file that includes that header. Compiling these files will throw errors if any header has unmet dependencies. For some libraries, there may be some header files which are not for direct inclusion, but rather are to be included via other header files. To allow later checking of these files for missing includes, we separate out the indirect include files from the direct ones. To ensure ongoing compliance, we enable this build test as part of the default x86 build in "test-meson-builds.sh". Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
2021-01-29 16:48:21 +00:00
indirect_headers += files('rte_regexdev_core.h')
regexdev: introduce API As RegEx usage become more used by DPDK applications, for example: * Next Generation Firewalls (NGFW) * Deep Packet and Flow Inspection (DPI) * Intrusion Prevention Systems (IPS) * DDoS Mitigation * Network Monitoring * Data Loss Prevention (DLP) * Smart NICs * Grammar based content processing * URL, spam and adware filtering * Advanced auditing and policing of user/application security policies * Financial data mining - parsing of streamed financial feeds * Application recognition. * Dmemory introspection. * Natural Language Processing (NLP) * Sentiment Analysis. * Big data database acceleration. * Computational storage. Number of PMD providers started to work on HW implementation, along side with SW implementations. This lib adds the support for those kind of devices. The RegEx Device API is composed of two parts: - The application-oriented RegEx API that includes functions to setup a RegEx device (configure it, setup its queue pairs and start it), update the rule database and so on. - The driver-oriented RegEx API that exports a function allowing a RegEx poll Mode Driver (PMD) to simultaneously register itself as a RegEx device driver. RegEx device components and definitions: +-----------------+ | | | o---------+ rte_regexdev_[en|de]queue_burst() | PCRE based o------+ | | | RegEx pattern | | | +--------+ | | matching engine o------+--+--o | | +------+ | | | | | queue |<==o===>|Core 0| | o----+ | | | pair 0 | | | | | | | | +--------+ +------+ +-----------------+ | | | ^ | | | +--------+ | | | | | | +------+ | | +--+--o queue |<======>|Core 1| Rule|Database | | | pair 1 | | | +------+----------+ | | +--------+ +------+ | Group 0 | | | | +-------------+ | | | +--------+ +------+ | | Rules 0..n | | | | | | |Core 2| | +-------------+ | | +--o queue |<======>| | | Group 1 | | | pair 2 | +------+ | +-------------+ | | +--------+ | | Rules 0..n | | | | +-------------+ | | +--------+ | Group 2 | | | | +------+ | +-------------+ | | | queue |<======>|Core n| | | Rules 0..n | | +-------o pair n | | | | +-------------+ | +--------+ +------+ | Group n | | +-------------+ |<-------rte_regexdev_rule_db_update() | | | |<-------rte_regexdev_rule_db_compile_activate() | | Rules 0..n | |<-------rte_regexdev_rule_db_import() | +-------------+ |------->rte_regexdev_rule_db_export() +-----------------+ RegEx: A regular expression is a concise and flexible means for matching strings of text, such as particular characters, words, or patterns of characters. A common abbreviation for this is â~@~\RegExâ~@~]. RegEx device: A hardware or software-based implementation of RegEx device API for PCRE based pattern matching syntax and semantics. PCRE RegEx syntax and semantics specification: http://regexkit.sourceforge.net/Documentation/pcre/pcrepattern.html RegEx queue pair: Each RegEx device should have one or more queue pair to transmit a burst of pattern matching request and receive a burst of receive the pattern matching response. The pattern matching request/response embedded in *rte_regex_ops* structure. Rule: A pattern matching rule expressed in PCRE RegEx syntax along with Match ID and Group ID to identify the rule upon the match. Rule database: The RegEx device accepts regular expressions and converts them into a compiled rule database that can then be used to scan data. Compilation allows the device to analyze the given pattern(s) and pre-determine how to scan for these patterns in an optimized fashion that would be far too expensive to compute at run-time. A rule database contains a set of rules that compiled in device specific binary form. Match ID or Rule ID: A unique identifier provided at the time of rule creation for the application to identify the rule upon match. Group ID: Group of rules can be grouped under one group ID to enable rule isolation and effective pattern matching. A unique group identifier provided at the time of rule creation for the application to identify the rule upon match. Scan: A pattern matching request through *enqueue* API. It may possible that a given RegEx device may not support all the features of PCRE. The application may probe unsupported features through struct rte_regexdev_info::pcre_unsup_flags By default, all the functions of the RegEx Device API exported by a PMD are lock-free functions which assume to not be invoked in parallel on different logical cores to work on the same target object. For instance, the dequeue function of a PMD cannot be invoked in parallel on two logical cores to operates on same RegEx queue pair. Of course, this function can be invoked in parallel by different logical core on different queue pair. It is the responsibility of the upper level application to enforce this rule. In all functions of the RegEx API, the RegEx device is designated by an integer >= 0 named the device identifier *dev_id* At the RegEx driver level, RegEx devices are represented by a generic data structure of type *rte_regexdev*. RegEx devices are dynamically registered during the PCI/SoC device probing phase performed at EAL initialization time. When a RegEx device is being probed, a *rte_regexdev* structure and a new device identifier are allocated for that device. Then, the regexdev_init() function supplied by the RegEx driver matching the probed device is invoked to properly initialize the device. The role of the device init function consists of resetting the hardware or software RegEx driver implementations. If the device init operation is successful, the correspondence between the device identifier assigned to the new device and its associated *rte_regexdev* structure is effectively registered. Otherwise, both the *rte_regexdev* structure and the device identifier are freed. The functions exported by the application RegEx API to setup a device designated by its device identifier must be invoked in the following order: - rte_regexdev_configure() - rte_regexdev_queue_pair_setup() - rte_regexdev_start() Then, the application can invoke, in any order, the functions exported by the RegEx API to enqueue pattern matching job, dequeue pattern matching response, get the stats, update the rule database, get/set device attributes and so on If the application wants to change the configuration (i.e. call rte_regexdev_configure() or rte_regexdev_queue_pair_setup()), it must call rte_regexdev_stop() first to stop the device and then do the reconfiguration before calling rte_regexdev_start() again. The enqueue and dequeue functions should not be invoked when the device is stopped. Finally, an application can close a RegEx device by invoking the rte_regexdev_close() function. Each function of the application RegEx API invokes a specific function of the PMD that controls the target device designated by its device identifier. For this purpose, all device-specific functions of a RegEx driver are supplied through a set of pointers contained in a generic structure of type *regexdev_ops*. The address of the *regexdev_ops* structure is stored in the *rte_regexdev* structure by the device init function of the RegEx driver, which is invoked during the PCI/SoC device probing phase, as explained earlier. In other words, each function of the RegEx API simply retrieves the *rte_regexdev* structure associated with the device identifier and performs an indirect invocation of the corresponding driver function supplied in the *regexdev_ops* structure of the *rte_regexdev* structure. For performance reasons, the address of the fast-path functions of the RegEx driver is not contained in the *regexdev_ops* structure. Instead, they are directly stored at the beginning of the *rte_regexdev* structure to avoid an extra indirect memory access during their invocation. RTE RegEx device drivers do not use interrupts for enqueue or dequeue operation. Instead, RegEx drivers export Poll-Mode enqueue and dequeue functions to applications. The *enqueue* operation submits a burst of RegEx pattern matching request to the RegEx device and the *dequeue* operation gets a burst of pattern matching response for the ones submitted through *enqueue* operation. Typical application utilisation of the RegEx device API will follow the following programming flow. - rte_regexdev_configure() - rte_regexdev_queue_pair_setup() - rte_regexdev_rule_db_update() Needs to invoke if precompiled rule database not provided in rte_regexdev_config::rule_db for rte_regexdev_configure() and/or application needs to update rule database. - rte_regexdev_rule_db_compile_activate() Needs to invoke if rte_regexdev_rule_db_update function was used. - Create or reuse exiting mempool for *rte_regex_ops* objects. - rte_regexdev_start() - rte_regexdev_enqueue_burst() - rte_regexdev_dequeue_burst() Signed-off-by: Jerin Jacob <jerinj@marvell.com> Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com> Signed-off-by: Ori Kam <orika@mellanox.com>
2020-07-06 17:36:46 +00:00
deps += ['mbuf']
Reference in New Issue Copy Permalink