Unlike split ring, packed ring does not mandate the ring size
to be a power of 2. So we have to use a modulo operation when
wrapping ring index.
Fixes: 873e8dad6f ("vhost: support packed ring in async datapath")
Cc: stable@dpdk.org
Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Acked-by: Cheng Jiang <cheng1.jiang@intel.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
The front-end should refill the descriptor with the mbuf indicated by
the buff_id rather then the index of used descriptor. Back-end may
return buffers out of order if async copy mode is enabled.
When initializing rxq, refill the descriptors in order as buff_id is
not available at that time.
Fixes: a76290c8f1 ("net/virtio: implement Rx path for packed queues")
Cc: stable@dpdk.org
Signed-off-by: Cheng Jiang <cheng1.jiang@intel.com>
Signed-off-by: Marvin Liu <yong.liu@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
This patch allows to check the amount of in-flight packets
for the vhost queue using async acceleration.
Signed-off-by: Jiayu Hu <jiayu.hu@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
We assume that in the sync path, if there is no buffer wrap in the
avail descriptors fetched in a batch, there is no buffer wrap in the
used descriptors which need to be written back in this batch, but
this assumption is wrong in the async path since there are inflight
descriptors which are processed by the DMA device.
This patch refactors the batch copy code and adds used ring buffer
wrap check as a batch copy condition to fix this issue.
Fixes: 873e8dad6f ("vhost: support packed ring in async datapath")
Cc: stable@dpdk.org
Signed-off-by: Cheng Jiang <cheng1.jiang@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
We introduced some new indexes in packed ring of async vhost. They
will eventually overflow and lead to errors if the ring size is not
a power of 2. This patch is to check and keep these indexes within a
reasonable range.
Fixes: 873e8dad6f ("vhost: support packed ring in async datapath")
Cc: stable@dpdk.org
Signed-off-by: Cheng Jiang <cheng1.jiang@intel.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
When parsing the virtio net header and packet header for dequeue offload,
we need to perform sanity check on the packet header to ensure:
- No out-of-boundary memory access.
- The packet header and virtio_net header are valid and aligned.
Fixes: d0cf91303d ("vhost: add Tx offload capabilities")
Cc: stable@dpdk.org
Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
This patch adds checking if RAW API is supported at the start
of the test command "cryptodev_qat_raw_api_autotest".
Signed-off-by: Adam Dybkowski <adamx.dybkowski@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
This patch provides the support for IPsec protocol
offload to the hardware.
Following security operations are added:
- session_create
- session_destroy
- capabilities_get
Signed-off-by: Michael Shamis <michaelsh@marvell.com>
Reviewed-by: Liron Himi <lironh@marvell.com>
Tested-by: Liron Himi <lironh@marvell.com>
In order to test the new mlx5 crypto PMD, the driver is added to the
crypto test application.
Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
The AES-XTS algorithm can supports wrapped key and data-unit.
The encryption/decryption can be done out of place and using multi
segments.
Add multi segment and out of place tests to the recently added AES-XTS
vectors, which support using data-unit and a wrapped key.
Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
The AES-XTS algorithm supports using a wrapped key.
In AES-XTS the data-unit defines the data block size to be
encrypted\decrypted.
Add AES-XTS vectors with a wrapped key.
Add a variable stating whether the key is wrapped or not.
Add the AES-XTS data-unit.
Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
The crypto operations are done with the WQE set which contains
one UMR WQE and one rdma write WQE. Most segments of the WQE
set are initialized properly during queue setup, only limited
segments are initialized according to the crypto detail in the
datapath process.
This commit adds the enqueue and dequeue operations and updates
the WQE set segments accordingly.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Currently, HW handles the WQEs much faster than the software,
Using the constant WQE set layout can initialize most of the WQE
segments in advanced, and software only needs to configure very
limited segments in datapath. This accelerates the software WQE
organize in datapath.
This commit initializes the fixed WQE set segments.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
The mlx5 HW crypto operations are done by attaching crypto property
to a memory region. Once done, every access to the memory via the
crypto-enabled memory region will result with in-line encryption or
decryption of the data.
As a result, the design choice is to provide two types of WQEs. One
is UMR WQE which sets the crypto property and the other is rdma write
WQE which sends DMA command to copy data from local MR to remote MR.
The size of the WQEs will be defined by a new devarg called
max_segs_num.
This devarg also defines the maximum segments in mbuf chain that will be
supported for crypto operations.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
A keytag is a piece of data encrypted together with a DEK.
When a DEK is referenced by an MKEY.bsf through its index, the keytag is
also supplied in the BSF as plaintext. The HW will decrypt the DEK (and
the attached keytag) and will fail the operation if the keytags don't
match.
This commit adds the configuration of the keytag with devargs.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
To work with crypto engines that are marked with wrapped_import_method,
a login session is required.
A crypto login object needs to be created using DevX.
The crypto login object contains:
- The credential pointer.
- The import_KEK pointer to be used for all secured information
communicated in crypto commands (key fields), including the
provided credential in this command.
- The credential secret, wrapped by the import_KEK indicated in
this command. Size includes 8 bytes IV for wrapping.
Added devargs for the required login values:
- wcs_file - path to the file containing the credential.
- import_kek_id - the import KEK pointer.
- credential_id - the credential pointer.
Create the login DevX object in pci_probe function and destroy it in
pci_remove.
Destroying the crypto login object means logout.
Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Mellanox user space drivers don't deal with physical addresses as part
of a memory protection mechanism.
The device translates the given virtual address to a physical address
using the given memory key as an address space identifier.
That's why any mbuf virtual address is moved directly to the HW
descriptor(WQE).
The mapping between the virtual address to the physical address is saved
in MR configured by the kernel to the HW.
Each MR has a key that should also be moved to the WQE by the SW.
When the SW sees an unmapped address, it extends the address range and
creates a MR using a system call.
Add memory region cache management:
- 2 level cache per queue-pair - no locks.
- 1 shared cache between all the queues using a lock.
Using this way, the MR key search per data-path address is optimized.
Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Sessions are used in symmetric transformations in order to prepare
objects and data for packet processing stage.
A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct,
bsf_size, bsf_p_type, block size index, encryption_order and encryption
standard.
Implement the next session operations:
mlx5_crypto_sym_session_get_size- returns the size of the mlx5
session struct.
mlx5_crypto_sym_session_configure- prepares the DEK hash-list
and saves all the session data.
mlx5_crypto_sym_session_clear - destroys the DEK hash-list.
Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
The HW queue pairs are a pair of send queue and receive queue of
independent work queues packed together in one object for the purpose
of transferring data between nodes of a network.
Completion Queue is a FIFO queue of completed work requests.
In crypto driver we use one QP in loopback in order to encrypt and
decrypt data locally without sending it to the wire.
In the configured QP we only use the SQ to perform the encryption and
decryption operations.
Added implementation for the QP setup function which creates the CQ,
creates the QP and changes its state to RTS (ready to send).
Added implementation for the release QP function to release all the QP
resources.
Added the ops structure that contains any operation which is supported
by the cryptodev.
Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
The basic dev control operations are configure, close, start, stop and
get info.
Extended the existing support of configure and close:
-mlx5_crypto_dev_configure- function used to configure device.
-mlx5_crypto_dev_close- function used to close a configured
device.
-mlx5_crypto_dev_stop- function used to stop device.
-mlx5_crypto_dev_start- function used to start device.
-mlx5_crypto_dev_infos_get- function used to get info.
Added config struct to user private data with the fields socket id,
number of queue pairs and feature flags to be disabled.
Add the dev_start function that is used to start a configured device.
Add the dev_stop function that is used to stop a configured device.
Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
A DEK (Data encryption Key) is an mlx5 HW object which represents
the cipher algorithm key.
The DEKs are used during data encryption/decryption operations.
In symmetric algorithms like AES-XTS, we use the same DEK for both
encryption and decryption.
Use the mlx5 hash-list tool to manage the DEK objects in the PMD.
Provide the compare, create and destroy functions to manage DEKs in
hash-list and introduce an internal API to setup and unset the DEK
management and to prepare and destroy specific DEK object.
The DEK hash-list will be created in dev_configure routine and
destroyed in dev_close routine.
Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Add a new PMD for Mellanox devices- crypto PMD.
The crypto PMD will be supported starting Nvidia ConnectX6 and
BlueField2.
The crypto PMD will add the support of encryption and decryption using
the AES-XTS symmetric algorithm.
The crypto PMD requires rdma-core and uses mlx5 DevX.
This patch adds the PCI probing, basic functions, build files and
log utility.
Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Currently, private_data_offset for the sessionless is computed
wrongly which includes extra bytes added by
sizeof(struct rte_crypto_sym_xform) * 2. This causes buffer
overflow which leads to test application crash while freeing the
ops mempool. This patch provides fix for the same and also takes
care of increasing the length of ops to accommodate space for
rte_event_crypto_metadata while creating the crypto ops mempool.
Fixes: 3c2c535ecf ("test: add event crypto adapter auto-test")
Cc: stable@dpdk.org
Signed-off-by: Abhinandan Gujjar <abhinandan.gujjar@intel.com>
Reported-by: Ciara Power <ciara.power@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Feature flag in dev would be updated during config.
On reconfigure, the field need to be set again to
original value.
Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Isal compress PMD has build failures on Arm platform.
As dependent library ISA-L is supported on Arm platform,
support of the PMD is expanded to Arm architecture.
Fixed build failure caused by architecture specific code,
and made the PMD multi architecture compatible.
Bugzilla ID: 755
Signed-off-by: Ruifeng Wang <ruifeng.wang@arm.com>
The issue can cause illegal physical address access while a huge-page A
is released and huge-page B is allocated on the same virtual address.
The old MR can be matched using the virtual address of huge-page B but
the HW will access the physical address of huge-page A which is no more
part of the DPDK process.
Register a driver callback for memory event in order to free out all the
MRs of memory that is going to be freed from the dpdk process.
Fixes: f8c97babc9 ("compress/mlx5: add data-path functions")
Cc: stable@dpdk.org
Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Reduce session private data size by allocating
auth_key dynamically as required. Added auth_key_iova
to eliminate any impact on fastpath.
Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Reduce session private data size by allocating
auth_key dynamically as required.
Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
In case of AES-GCM/CCM, nonce/salt comes along
with IV, hence can be copied in a single memcpy.
This patch fixes the IV copy in lookaside IPsec
outbound instruction.
Fixes: fab634eb87 ("crypto/octeontx2: support security session data path")
Cc: stable@dpdk.org
Signed-off-by: Tejasree Kondoj <ktejasree@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
SA table entry would be reserved for inline inbound operations. Clear
valid bit of the SA so that CPT would treat SA entry as invalid. Also,
move setting of valid bit to the end in case of session_create() to
eliminate possibility of hardware seeing partial data.
Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Add locking for IPsec table updates.
Fixed error handling to clear SA entry if the SA
population functions encounters any error.
Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
The member 'dir' should not overlap with 'ip'. Usage of union for all
members would mean dir would get corrupt.
Fixes: e91b4f45ff ("net/octeontx2: support anti-replay for security session")
Cc: stable@dpdk.org
Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
The instruction queue is enabled in dev start and
is disabled in dev stop.
Signed-off-by: Ankur Dwivedi <adwivedi@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
The code for enabling instruction queue is moved to ROC API.
Signed-off-by: Ankur Dwivedi <adwivedi@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
This commit updates the QAT raw data-path API to support the
changes made to device and sessions. The QAT RAW data-path API
now works on Generation 1-3 devices and is disabled on GEN4.
Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com>
Signed-off-by: Adam Dybkowski <adamx.dybkowski@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
This commit adds service discovery to generation four
of Intel QuickAssist Technology devices.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
This commit resets ring pairs of particular vf before
setting PMD.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Add communication between physical device and virtual function
in Intel QucikAssist Technology PMD.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Rework init common header function for request
descriptor so it can be called only once.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Content descriptor functions are incorrectly named,
having them with proper name will improve readability and
facilitate further work.
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
