d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

qat: Import a new Intel (R) QAT driver

Browse Source 
QAT in-tree driver ported from out-of-tree release available
from 01.org.

The driver exposes complete cryptography and data compression
API in the kernel and integrates with Open Crypto Framework.
Details of supported operations, devices and usage can be found
in man and on 01.org.

Patch co-authored by: Krzysztof Zdziarski <krzysztofx.zdziarski@intel.com>
Patch co-authored by: Michal Jaraczewski <michalx.jaraczewski@intel.com>
Patch co-authored by: Michal Gulbicki <michalx.gulbicki@intel.com>
Patch co-authored by: Julian Grajkowski <julianx.grajkowski@intel.com>
Patch co-authored by: Piotr Kasierski <piotrx.kasierski@intel.com>
Patch co-authored by: Adam Czupryna <adamx.czupryna@intel.com>
Patch co-authored by: Konrad Zelazny <konradx.zelazny@intel.com>
Patch co-authored by: Katarzyna Rucinska <katarzynax.kargol@intel.com>
Patch co-authored by: Lukasz Kolodzinski <lukaszx.kolodzinski@intel.com>
Patch co-authored by: Zbigniew Jedlinski <zbigniewx.jedlinski@intel.com>

Reviewed by:	markj, jhb (OCF integration)
Reviewed by:	debdrup, pauamma (docs)
Sponsored by:	Intel Corporation
Differential Revision: https://reviews.freebsd.org/D34632
This commit is contained in:
Julian Grajkowski 2022-07-19 09:15:34 +01:00 committed by Mark Johnston
parent f4f56ff43d
commit 78ee8d1c4c
255 changed files with 99110 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

127
share/man/man4/qat.4 Normal file
View File

@ -0,0 +1,127 @@
.\" SPDX-License-Identifier: BSD-3-Clause
.\" Copyright(c) 2007-2022 Intel Corporation
.\" $FreeBSD$
.Dd June 30, 2022
.Dt QAT 4
.Os
.Sh NAME
.Nm qat
.Nd Intel (R) QuickAssist Technology (QAT) driver
.Sh SYNOPSIS
To load the driver call:
.Pp
.Bl -item -compact
.It
kldload qat
.El
.Pp
In order to load the driver on boot add these lines to
.Xr loader.conf 5 selecting firmware(s) suitable for installed device(s)
.Pp
.Bl -item -compact
.It
qat_200xx_fw_load="YES"
.It
qat_c3xxx_fw_load="YES"
.It
qat_c4xxx_fw_load="YES"
.It
qat_c62x_fw_load="YES"
.It
qat_dh895xcc_fw_load="YES"
.It
qat_load="YES"
.El
.Sh DESCRIPTION
The
.Nm
driver supports cryptography and compression acceleration of the
Intel (R) QuickAssist Technology (QAT) devices.
.Pp
The
.Nm
driver is intended for platforms that contain:
.Bl -bullet -compact
.It
Intel (R) C62x Chipset
.It
Intel (R) Atom C3000 processor product family
.It
Intel (R) QuickAssist Adapter 8960/Intel (R) QuickAssist Adapter 8970
(formerly known as "Lewis Hill")
.It
Intel (R) Communications Chipset 8925 to 8955 Series
.It
Intel (R) Atom P5300 processor product family
.El
.Pp
The
.Nm
driver supports cryptography and compression acceleration.
A complete API for offloading these operations is exposed in the kernel and may
be used by any other entity directly.
For details of usage and supported operations and algorithms refer to the
following documentation available from
.Lk 01.org :
.Bl -bullet -compact
.It
.Rs
.%A Intel (R)
.%T QuickAssist Technology API Programmer's Guide
.Re
.It
.Rs
.%A Intel (R)
.%T QuickAssist Technology Cryptographic API Reference Manual
.Re
.It
.Rs
.%A Intel (R)
.%T QuickAssist Technology Data Compression API Reference Manual
.Re
.It
.Rs
.%A Intel (R)
.%T QuickAssist Technology Performance Optimization Guide
.Re
.El
.Pp
In addition to exposing complete kernel API for offloading cryptography and
compression operations, the
.Nm
driver also integrates with
.Xr crypto 4 ,
allowing offloading supported cryptography operations to Intel (R) QuickAssist
Technology (QAT) devices.
For details of usage and supported operations and algorithms refer to the
documentation mentioned above and
.Sx SEE ALSO
section.
.Sh COMPATIBILITY
The
.Nm
driver replaced previous implementation introduced in
.Fx 13.0 .
Current version, in addition to
.Xr crypto 4
integration, supports also data compression and exposes a complete API for
offloading data compression and cryptography operations.
.Sh SEE ALSO
.Xr crypto 4 ,
.Xr ipsec 4 ,
.Xr pci 4 ,
.Xr crypto 7 ,
.Xr crypto 9
.Sh HISTORY
This
.Nm
driver was introduced in
.Fx 14.0 .
.Fx 13.0 included a different version of
.Nm
driver.
.Sh AUTHORS
The
.Nm
driver was written by
.An Intel (R) Corporation .

46
sys/contrib/dev/qat/LICENSE
View File

@ -1,11 +1,39 @@
Copyright (c) 2007-2016 Intel Corporation.
All rights reserved.
Redistribution. Redistribution and use in binary form, without modification, are permitted provided that the following conditions are met:
Copyright (c) 2021 Intel Corporation
Redistributions must reproduce the above copyright notice and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of Intel Corporation nor the names of its suppliers may be used to endorse or promote products derived from this software without specific prior written permission.
No reverse engineering, decompilation, or disassembly of this software is permitted.
Limited patent license. Intel Corporation grants a world-wide, royalty-free, non-exclusive license under patents it now or hereafter owns or controls to make, have made, use, import, offer to sell and sell ("Utilize") this software, but solely to the extent that any such patent is necessary to Utilize the software alone. The patent license shall not apply to any combinations which include this software. No hardware per se is licensed hereunder.
Redistribution. Redistribution and use in binary form, without
modification, are permitted provided that the following conditions are
met:
* Redistributions must reproduce the above copyright notice and the
following disclaimer in the documentation and/or other materials
provided with the distribution.
* Neither the name of Intel Corporation nor the names of its suppliers
may be used to endorse or promote products derived from this software
without specific prior written permission.
* No reverse engineering, decompilation, or disassembly of this software
is permitted.
Limited patent license. Intel Corporation grants a world-wide,
royalty-free, non-exclusive license under patents it now or hereafter
owns or controls to make, have made, use, import, offer to sell and
sell ("Utilize") this software, but solely to the extent that any
such patent is necessary to Utilize the software alone, or in
combination with an operating system licensed under an approved Open
Source license as listed by the Open Source Initiative at
http://opensource.org/licenses. The patent license shall not apply to
any other combinations which include this software. No hardware per
se is licensed hereunder.
DISCLAIMER. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
DISCLAIMER. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

BIN
sys/contrib/dev/qat/qat_200xx.bin Normal file
View File

Binary file not shown.

BIN
sys/contrib/dev/qat/qat_200xx_mmp.bin Normal file
View File

Binary file not shown.

BIN
sys/contrib/dev/qat/qat_895xcc.bin Normal file
View File

Binary file not shown.

BIN
sys/contrib/dev/qat/qat_895xcc_mmp.bin Normal file
View File

Binary file not shown.

BIN
sys/contrib/dev/qat/qat_c3xxx.bin Normal file
View File

Binary file not shown.

BIN
sys/contrib/dev/qat/qat_c3xxx_mmp.bin Normal file
View File

Binary file not shown.

BIN
sys/contrib/dev/qat/qat_c4xxx.bin Normal file
View File

Binary file not shown.

BIN
sys/contrib/dev/qat/qat_c4xxx_mmp.bin Normal file
View File

Binary file not shown.

BIN
sys/contrib/dev/qat/qat_c62x.bin Normal file
View File

Binary file not shown.

BIN
sys/contrib/dev/qat/qat_c62x_mmp.bin Normal file
View File

Binary file not shown.

12
sys/dev/qat/include/adf_cfg_dev_dbg.h Normal file
View File

@ -0,0 +1,12 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_CFG_DEV_DBG_H_
#define ADF_CFG_DEV_DBG_H_
struct adf_accel_dev;
int adf_cfg_dev_dbg_add(struct adf_accel_dev *accel_dev);
void adf_cfg_dev_dbg_remove(struct adf_accel_dev *accel_dev);
#endif /* ADF_CFG_DEV_DBG_H_ */

75
sys/dev/qat/include/adf_cfg_device.h Normal file
View File

@ -0,0 +1,75 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_CFG_DEVICE_H_
#define ADF_CFG_DEVICE_H_
#include "adf_cfg.h"
#include "sal_statistics_strings.h"
#define ADF_CFG_STATIC_CONF_VER 2
#define ADF_CFG_STATIC_CONF_CY_ASYM_RING_SIZE 64
#define ADF_CFG_STATIC_CONF_CY_SYM_RING_SIZE 512
#define ADF_CFG_STATIC_CONF_DC_INTER_BUF_SIZE 64
#define ADF_CFG_STATIC_CONF_SAL_STATS_CFG_ENABLED 1
#define ADF_CFG_STATIC_CONF_SAL_STATS_CFG_DC 1
#define ADF_CFG_STATIC_CONF_SAL_STATS_CFG_DH 0
#define ADF_CFG_STATIC_CONF_SAL_STATS_CFG_DRBG 0
#define ADF_CFG_STATIC_CONF_SAL_STATS_CFG_DSA 0
#define ADF_CFG_STATIC_CONF_SAL_STATS_CFG_ECC 0
#define ADF_CFG_STATIC_CONF_SAL_STATS_CFG_KEYGEN 0
#define ADF_CFG_STATIC_CONF_SAL_STATS_CFG_LN 0
#define ADF_CFG_STATIC_CONF_SAL_STATS_CFG_PRIME 0
#define ADF_CFG_STATIC_CONF_SAL_STATS_CFG_RSA 0
#define ADF_CFG_STATIC_CONF_SAL_STATS_CFG_SYM 1
#define ADF_CFG_STATIC_CONF_POLL 1
#define ADF_CFG_STATIC_CONF_IRQ 0
#define ADF_CFG_STATIC_CONF_AUTO_RESET 0
#define ADF_CFG_STATIC_CONF_NUM_DC_ACCEL_UNITS 2
#define ADF_CFG_STATIC_CONF_NUM_INLINE_ACCEL_UNITS 0
#define ADF_CFG_STATIC_CONF_INST_NUM_DC 2
#define ADF_CFG_STATIC_CONF_INST_NUM_CY_POLL 2
#define ADF_CFG_STATIC_CONF_INST_NUM_CY_IRQ 2
#define ADF_CFG_FW_STRING_TO_ID(str, acc, id) \
do { \
typeof(id) id_ = (id); \
typeof(str) str_; \
memcpy(str_, (str), sizeof(str_)); \
if (!strncmp(str_, \
ADF_SERVICES_DEFAULT, \
sizeof(ADF_SERVICES_DEFAULT))) \
*id_ = ADF_FW_IMAGE_DEFAULT; \
else if (!strncmp(str_, \
ADF_SERVICES_CRYPTO, \
sizeof(ADF_SERVICES_CRYPTO))) \
*id_ = ADF_FW_IMAGE_CRYPTO; \
else if (!strncmp(str_, \
ADF_SERVICES_COMPRESSION, \
sizeof(ADF_SERVICES_COMPRESSION))) \
*id_ = ADF_FW_IMAGE_COMPRESSION; \
else if (!strncmp(str_, \
ADF_SERVICES_CUSTOM1, \
sizeof(ADF_SERVICES_CUSTOM1))) \
*id_ = ADF_FW_IMAGE_CUSTOM1; \
else { \
*id_ = ADF_FW_IMAGE_DEFAULT; \
device_printf(GET_DEV(acc), \
"Invalid SerivesProfile: %s," \
"Using DEFAULT image\n", \
str_); \
} \
} while (0)
int adf_cfg_get_ring_pairs(struct adf_cfg_device *device,
struct adf_cfg_instance *inst,
const char *process_name,
struct adf_accel_dev *accel_dev);
int adf_cfg_device_init(struct adf_cfg_device *device,
struct adf_accel_dev *accel_dev);
void adf_cfg_device_clear(struct adf_cfg_device *device,
struct adf_accel_dev *accel_dev);
#endif

11
sys/dev/qat/include/adf_cnvnr_freq_counters.h Normal file
View File

@ -0,0 +1,11 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_CNVNR_CTRS_DBG_H_
#define ADF_CNVNR_CTRS_DBG_H_
struct adf_accel_dev;
int adf_cnvnr_freq_counters_add(struct adf_accel_dev *accel_dev);
void adf_cnvnr_freq_counters_remove(struct adf_accel_dev *accel_dev);
#endif /* ADF_CNVNR_CTRS_DBG_H_ */

80
sys/dev/qat/include/adf_dev_err.h Normal file
View File

@ -0,0 +1,80 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_DEV_ERR_H_
#define ADF_DEV_ERR_H_
#include <sys/types.h>
#include <dev/pci/pcivar.h>
#include "adf_accel_devices.h"
#define ADF_ERRSOU0 (0x3A000 + 0x00)
#define ADF_ERRSOU1 (0x3A000 + 0x04)
#define ADF_ERRSOU2 (0x3A000 + 0x08)
#define ADF_ERRSOU3 (0x3A000 + 0x0C)
#define ADF_ERRSOU4 (0x3A000 + 0xD0)
#define ADF_ERRSOU5 (0x3A000 + 0xD8)
#define ADF_ERRMSK0 (0x3A000 + 0x10)
#define ADF_ERRMSK1 (0x3A000 + 0x14)
#define ADF_ERRMSK2 (0x3A000 + 0x18)
#define ADF_ERRMSK3 (0x3A000 + 0x1C)
#define ADF_ERRMSK4 (0x3A000 + 0xD4)
#define ADF_ERRMSK5 (0x3A000 + 0xDC)
#define ADF_EMSK3_CPM0_MASK BIT(2)
#define ADF_EMSK3_CPM1_MASK BIT(3)
#define ADF_EMSK5_CPM2_MASK BIT(16)
#define ADF_EMSK5_CPM3_MASK BIT(17)
#define ADF_EMSK5_CPM4_MASK BIT(18)
#define ADF_RICPPINTSTS (0x3A000 + 0x114)
#define ADF_RIERRPUSHID (0x3A000 + 0x118)
#define ADF_RIERRPULLID (0x3A000 + 0x11C)
#define ADF_CPP_CFC_ERR_STATUS (0x30000 + 0xC04)
#define ADF_CPP_CFC_ERR_PPID (0x30000 + 0xC08)
#define ADF_TICPPINTSTS (0x3A400 + 0x13C)
#define ADF_TIERRPUSHID (0x3A400 + 0x140)
#define ADF_TIERRPULLID (0x3A400 + 0x144)
#define ADF_SECRAMUERR (0x3AC00 + 0x04)
#define ADF_SECRAMUERRAD (0x3AC00 + 0x0C)
#define ADF_CPPMEMTGTERR (0x3AC00 + 0x10)
#define ADF_ERRPPID (0x3AC00 + 0x14)
#define ADF_INTSTATSSM(i) ((i)*0x4000 + 0x04)
#define ADF_INTSTATSSM_SHANGERR BIT(13)
#define ADF_PPERR(i) ((i)*0x4000 + 0x08)
#define ADF_PPERRID(i) ((i)*0x4000 + 0x0C)
#define ADF_CERRSSMSH(i) ((i)*0x4000 + 0x10)
#define ADF_UERRSSMSH(i) ((i)*0x4000 + 0x18)
#define ADF_UERRSSMSHAD(i) ((i)*0x4000 + 0x1C)
#define ADF_SLICEHANGSTATUS(i) ((i)*0x4000 + 0x4C)
#define ADF_SLICE_HANG_AUTH0_MASK BIT(0)
#define ADF_SLICE_HANG_AUTH1_MASK BIT(1)
#define ADF_SLICE_HANG_AUTH2_MASK BIT(2)
#define ADF_SLICE_HANG_CPHR0_MASK BIT(4)
#define ADF_SLICE_HANG_CPHR1_MASK BIT(5)
#define ADF_SLICE_HANG_CPHR2_MASK BIT(6)
#define ADF_SLICE_HANG_CMP0_MASK BIT(8)
#define ADF_SLICE_HANG_CMP1_MASK BIT(9)
#define ADF_SLICE_HANG_XLT0_MASK BIT(12)
#define ADF_SLICE_HANG_XLT1_MASK BIT(13)
#define ADF_SLICE_HANG_MMP0_MASK BIT(16)
#define ADF_SLICE_HANG_MMP1_MASK BIT(17)
#define ADF_SLICE_HANG_MMP2_MASK BIT(18)
#define ADF_SLICE_HANG_MMP3_MASK BIT(19)
#define ADF_SLICE_HANG_MMP4_MASK BIT(20)
#define ADF_SSMWDT(i) ((i)*0x4000 + 0x54)
#define ADF_SSMWDTPKE(i) ((i)*0x4000 + 0x58)
#define ADF_SHINTMASKSSM(i) ((i)*0x4000 + 0x1018)
#define ADF_ENABLE_SLICE_HANG 0x000000
#define ADF_MAX_MMP (5)
#define ADF_MMP_BASE(i) ((i)*0x1000 % 0x3800)
#define ADF_CERRSSMMMP(i, n) ((i)*0x4000 + ADF_MMP_BASE(n) + 0x380)
#define ADF_UERRSSMMMP(i, n) ((i)*0x4000 + ADF_MMP_BASE(n) + 0x388)
#define ADF_UERRSSMMMPAD(i, n) ((i)*0x4000 + ADF_MMP_BASE(n) + 0x38C)
bool adf_handle_slice_hang(struct adf_accel_dev *accel_dev,
u8 accel_num,
struct resource *csr,
u32 slice_hang_offset);
bool adf_check_slice_hang(struct adf_accel_dev *accel_dev);
void adf_print_err_registers(struct adf_accel_dev *accel_dev);
#endif

10
sys/dev/qat/include/adf_freebsd_pfvf_ctrs_dbg.h Normal file
View File

@ -0,0 +1,10 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_PFVF_CTRS_DBG_H_
#define ADF_PFVF_CTRS_DBG_H_
struct adf_accel_dev;
int adf_pfvf_ctrs_dbg_add(struct adf_accel_dev *accel_dev);
#endif /* ADF_PFVF_CTRS_DBG_H_ */

40
sys/dev/qat/include/adf_fw_counters.h Normal file
View File

@ -0,0 +1,40 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_FW_COUNTERS_H_
#define ADF_FW_COUNTERS_H_
#include <linux/rwsem.h>
#include "adf_accel_devices.h"
#define FW_COUNTERS_MAX_STR_LEN 64
#define FW_COUNTERS_MAX_KEY_LEN_IN_BYTES FW_COUNTERS_MAX_STR_LEN
#define FW_COUNTERS_MAX_VAL_LEN_IN_BYTES FW_COUNTERS_MAX_STR_LEN
#define FW_COUNTERS_MAX_SECTION_LEN_IN_BYTES FW_COUNTERS_MAX_STR_LEN
#define ADF_FW_COUNTERS_NO_RESPONSE -1
struct adf_fw_counters_val {
char key[FW_COUNTERS_MAX_KEY_LEN_IN_BYTES];
char val[FW_COUNTERS_MAX_VAL_LEN_IN_BYTES];
struct list_head list;
};
struct adf_fw_counters_section {
char name[FW_COUNTERS_MAX_SECTION_LEN_IN_BYTES];
struct list_head list;
struct list_head param_head;
};
struct adf_fw_counters_data {
struct list_head ae_sec_list;
struct sysctl_oid *debug;
struct rw_semaphore lock;
};
int adf_fw_counters_add(struct adf_accel_dev *accel_dev);
void adf_fw_counters_remove(struct adf_accel_dev *accel_dev);
int adf_fw_count_ras_event(struct adf_accel_dev *accel_dev,
u32 *ras_event,
char *aeidstr);
#endif /* ADF_FW_COUNTERS_H_ */

33
sys/dev/qat/include/adf_heartbeat.h Normal file
View File

@ -0,0 +1,33 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_HEARTBEAT_H_
#define ADF_HEARTBEAT_H_
#include "adf_cfg_common.h"
struct adf_accel_dev;
struct qat_sysctl {
unsigned int hb_sysctlvar;
struct sysctl_oid *oid;
};
struct adf_heartbeat {
unsigned int hb_sent_counter;
unsigned int hb_failed_counter;
u64 last_hb_check_time;
enum adf_device_heartbeat_status last_hb_status;
struct qat_sysctl heartbeat;
struct qat_sysctl *heartbeat_sent;
struct qat_sysctl *heartbeat_failed;
};
int adf_heartbeat_init(struct adf_accel_dev *accel_dev);
void adf_heartbeat_clean(struct adf_accel_dev *accel_dev);
int adf_get_hb_timer(struct adf_accel_dev *accel_dev, unsigned int *value);
int adf_get_heartbeat_status(struct adf_accel_dev *accel_dev);
int adf_heartbeat_status(struct adf_accel_dev *accel_dev,
enum adf_device_heartbeat_status *hb_status);
#endif /* ADF_HEARTBEAT_H_ */

11
sys/dev/qat/include/adf_heartbeat_dbg.h Normal file
View File

@ -0,0 +1,11 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_HEARTBEAT_DBG_H_
#define ADF_HEARTBEAT_DBG_H_
struct adf_accel_dev;
int adf_heartbeat_dbg_add(struct adf_accel_dev *accel_dev);
int adf_heartbeat_dbg_del(struct adf_accel_dev *accel_dev);
#endif /* ADF_HEARTBEAT_DBG_H_ */

182
sys/dev/qat/include/adf_pf2vf_msg.h Normal file
View File

@ -0,0 +1,182 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_PF2VF_MSG_H
#define ADF_PF2VF_MSG_H
/*
* PF<->VF Messaging
* The PF has an array of 32-bit PF2VF registers, one for each VF. The
* PF can access all these registers; each VF can access only the one
* register associated with that particular VF.
*
* The register functionally is split into two parts:
* The bottom half is for PF->VF messages. In particular when the first
* bit of this register (bit 0) gets set an interrupt will be triggered
* in the respective VF.
* The top half is for VF->PF messages. In particular when the first bit
* of this half of register (bit 16) gets set an interrupt will be triggered
* in the PF.
*
* The remaining bits within this register are available to encode messages.
* and implement a collision control mechanism to prevent concurrent use of
* the PF2VF register by both the PF and VF.
*
* 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16
* _______________________________________________
* | | | | | | | | | | | | | | | | |
* +-----------------------------------------------+
* \___________________________/ \_________/ ^ ^
* ^ ^ | |
* | | | VF2PF Int
* | | Message Origin
* | Message Type
* Message-specific Data/Reserved
*
* 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
* _______________________________________________
* | | | | | | | | | | | | | | | | |
* +-----------------------------------------------+
* \___________________________/ \_________/ ^ ^
* ^ ^ | |
* | | | PF2VF Int
* | | Message Origin
* | Message Type
* Message-specific Data/Reserved
*
* Message Origin (Should always be 1)
* A legacy out-of-tree QAT driver allowed for a set of messages not supported
* by this driver; these had a Msg Origin of 0 and are ignored by this driver.
*
* When a PF or VF attempts to send a message in the lower or upper 16 bits,
* respectively, the other 16 bits are written to first with a defined
* IN_USE_BY pattern as part of a collision control scheme (see adf_iov_putmsg).
*/
/* VF/PF compatibility version. */
/* ADF_PFVF_COMPATIBILITY_EXT_CAP: Support for extended capabilities */
#define ADF_PFVF_COMPATIBILITY_CAPABILITIES 2
/* ADF_PFVF_COMPATIBILITY_FAST_ACK: In-use pattern cleared by receiver */
#define ADF_PFVF_COMPATIBILITY_FAST_ACK 3
#define ADF_PFVF_COMPATIBILITY_RING_TO_SVC_MAP 4
#define ADF_PFVF_COMPATIBILITY_VERSION 4 /* PF<->VF compat */
/* PF->VF messages */
#define ADF_PF2VF_INT BIT(0)
#define ADF_PF2VF_MSGORIGIN_SYSTEM BIT(1)
#define ADF_PF2VF_MSGTYPE_MASK 0x0000003C
#define ADF_PF2VF_MSGTYPE_SHIFT 2
#define ADF_PF2VF_MSGTYPE_RESTARTING 0x01
#define ADF_PF2VF_MSGTYPE_VERSION_RESP 0x02
#define ADF_PF2VF_MSGTYPE_BLOCK_RESP 0x03
#define ADF_PF2VF_MSGTYPE_FATAL_ERROR 0x04
#define ADF_PF2VF_IN_USE_BY_PF 0x6AC20000
#define ADF_PF2VF_IN_USE_BY_PF_MASK 0xFFFE0000
/* PF->VF Version Response */
#define ADF_PF2VF_VERSION_RESP_VERS_MASK 0x00003FC0
#define ADF_PF2VF_VERSION_RESP_VERS_SHIFT 6
#define ADF_PF2VF_VERSION_RESP_RESULT_MASK 0x0000C000
#define ADF_PF2VF_VERSION_RESP_RESULT_SHIFT 14
#define ADF_PF2VF_MINORVERSION_SHIFT 6
#define ADF_PF2VF_MAJORVERSION_SHIFT 10
#define ADF_PF2VF_VF_COMPATIBLE 1
#define ADF_PF2VF_VF_INCOMPATIBLE 2
#define ADF_PF2VF_VF_COMPAT_UNKNOWN 3
/* PF->VF Block Request Type */
#define ADF_VF2PF_MIN_SMALL_MESSAGE_TYPE 0
#define ADF_VF2PF_MAX_SMALL_MESSAGE_TYPE (ADF_VF2PF_MIN_SMALL_MESSAGE_TYPE + 15)
#define ADF_VF2PF_MIN_MEDIUM_MESSAGE_TYPE (ADF_VF2PF_MAX_SMALL_MESSAGE_TYPE + 1)
#define ADF_VF2PF_MAX_MEDIUM_MESSAGE_TYPE \
(ADF_VF2PF_MIN_MEDIUM_MESSAGE_TYPE + 7)
#define ADF_VF2PF_MIN_LARGE_MESSAGE_TYPE (ADF_VF2PF_MAX_MEDIUM_MESSAGE_TYPE + 1)
#define ADF_VF2PF_MAX_LARGE_MESSAGE_TYPE (ADF_VF2PF_MIN_LARGE_MESSAGE_TYPE + 3)
#define ADF_VF2PF_SMALL_PAYLOAD_SIZE 30
#define ADF_VF2PF_MEDIUM_PAYLOAD_SIZE 62
#define ADF_VF2PF_LARGE_PAYLOAD_SIZE 126
#define ADF_VF2PF_MAX_BLOCK_TYPE 3
#define ADF_VF2PF_BLOCK_REQ_TYPE_SHIFT 22
#define ADF_VF2PF_LARGE_BLOCK_BYTE_NUM_SHIFT 24
#define ADF_VF2PF_MEDIUM_BLOCK_BYTE_NUM_SHIFT 25
#define ADF_VF2PF_SMALL_BLOCK_BYTE_NUM_SHIFT 26
#define ADF_VF2PF_BLOCK_REQ_CRC_SHIFT 31
#define ADF_VF2PF_LARGE_BLOCK_BYTE_NUM_MASK 0x7F000000
#define ADF_VF2PF_MEDIUM_BLOCK_BYTE_NUM_MASK 0x7E000000
#define ADF_VF2PF_SMALL_BLOCK_BYTE_NUM_MASK 0x7C000000
#define ADF_VF2PF_LARGE_BLOCK_REQ_TYPE_MASK 0xC00000
#define ADF_VF2PF_MEDIUM_BLOCK_REQ_TYPE_MASK 0x1C00000
#define ADF_VF2PF_SMALL_BLOCK_REQ_TYPE_MASK 0x3C00000
/* PF->VF Block Response Type */
#define ADF_PF2VF_BLOCK_RESP_TYPE_DATA 0x0
#define ADF_PF2VF_BLOCK_RESP_TYPE_CRC 0x1
#define ADF_PF2VF_BLOCK_RESP_TYPE_ERROR 0x2
#define ADF_PF2VF_BLOCK_RESP_TYPE_SHIFT 6
#define ADF_PF2VF_BLOCK_RESP_DATA_SHIFT 8
#define ADF_PF2VF_BLOCK_RESP_TYPE_MASK 0x000000C0
#define ADF_PF2VF_BLOCK_RESP_DATA_MASK 0x0000FF00
/* PF-VF block message header bytes */
#define ADF_VF2PF_BLOCK_VERSION_BYTE 0
#define ADF_VF2PF_BLOCK_LEN_BYTE 1
#define ADF_VF2PF_BLOCK_DATA 2
/* PF->VF Block Error Code */
#define ADF_PF2VF_INVALID_BLOCK_TYPE 0x0
#define ADF_PF2VF_INVALID_BYTE_NUM_REQ 0x1
#define ADF_PF2VF_PAYLOAD_TRUNCATED 0x2
#define ADF_PF2VF_UNSPECIFIED_ERROR 0x3
/* VF->PF messages */
#define ADF_VF2PF_IN_USE_BY_VF 0x00006AC2
#define ADF_VF2PF_IN_USE_BY_VF_MASK 0x0000FFFE
#define ADF_VF2PF_INT BIT(16)
#define ADF_VF2PF_MSGORIGIN_SYSTEM BIT(17)
#define ADF_VF2PF_MSGTYPE_MASK 0x003C0000
#define ADF_VF2PF_MSGTYPE_SHIFT 18
#define ADF_VF2PF_MSGTYPE_INIT 0x3
#define ADF_VF2PF_MSGTYPE_SHUTDOWN 0x4
#define ADF_VF2PF_MSGTYPE_VERSION_REQ 0x5
#define ADF_VF2PF_MSGTYPE_COMPAT_VER_REQ 0x6
#define ADF_VF2PF_MSGTYPE_GET_LARGE_BLOCK_REQ 0x7
#define ADF_VF2PF_MSGTYPE_GET_MEDIUM_BLOCK_REQ 0x8
#define ADF_VF2PF_MSGTYPE_GET_SMALL_BLOCK_REQ 0x9
#define ADF_VF2PF_MSGTYPE_NOTIFY 0xa
#define ADF_VF2PF_MSGGENC_RESTARTING_COMPLETE 0x0
/* Block message types
* 0..15 - 32 byte message
* 16..23 - 64 byte message
* 24..27 - 128 byte message
* 2 - Get Capability Request message
*/
#define ADF_VF2PF_BLOCK_MSG_CAP_SUMMARY 2
#define ADF_VF2PF_BLOCK_MSG_GET_RING_TO_SVC_REQ 0x3
/* VF->PF Compatible Version Request */
#define ADF_VF2PF_COMPAT_VER_REQ_SHIFT 22
/* How long to wait for far side to acknowledge receipt */
#define ADF_IOV_MSG_ACK_DELAY_US 5
#define ADF_IOV_MSG_ACK_EXP_MAX_DELAY_US (5 * 1000)
#define ADF_IOV_MSG_ACK_DELAY_MS 5
#define ADF_IOV_MSG_ACK_LIN_MAX_DELAY_US (2 * 1000 * 1000)
/* If CSR is busy, how long to delay before retrying */
#define ADF_IOV_MSG_RETRY_DELAY 5
#define ADF_IOV_MSG_MAX_RETRIES 10
/* How long to wait for a response from the other side */
#define ADF_IOV_MSG_RESP_TIMEOUT 100
/* How often to retry when there is no response */
#define ADF_IOV_MSG_RESP_RETRIES 5
#define ADF_IOV_RATELIMIT_INTERVAL 8
#define ADF_IOV_RATELIMIT_BURST 130
/* CRC Calculation */
#define ADF_CRC8_INIT_VALUE 0xFF
/* PF VF message byte shift */
#define ADF_PFVF_DATA_SHIFT 8
#define ADF_PFVF_DATA_MASK 0xFF
#endif /* ADF_IOV_MSG_H */

11
sys/dev/qat/include/adf_ver_dbg.h Normal file
View File

@ -0,0 +1,11 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_VER_DBG_H_
#define ADF_VER_DBG_H_
struct adf_accel_dev;
int adf_ver_dbg_add(struct adf_accel_dev *accel_dev);
void adf_ver_dbg_del(struct adf_accel_dev *accel_dev);
#endif /* ADF_VER_DBG_H_ */

585
sys/dev/qat/include/common/adf_accel_devices.h Normal file
View File

@ -0,0 +1,585 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_ACCEL_DEVICES_H_
#define ADF_ACCEL_DEVICES_H_
#include "qat_freebsd.h"
#include "adf_cfg_common.h"
#define ADF_CFG_NUM_SERVICES 4
#define ADF_DH895XCC_DEVICE_NAME "dh895xcc"
#define ADF_DH895XCCVF_DEVICE_NAME "dh895xccvf"
#define ADF_C62X_DEVICE_NAME "c6xx"
#define ADF_C62XVF_DEVICE_NAME "c6xxvf"
#define ADF_C3XXX_DEVICE_NAME "c3xxx"
#define ADF_C3XXXVF_DEVICE_NAME "c3xxxvf"
#define ADF_200XX_DEVICE_NAME "200xx"
#define ADF_200XXVF_DEVICE_NAME "200xxvf"
#define ADF_C4XXX_DEVICE_NAME "c4xxx"
#define ADF_C4XXXVF_DEVICE_NAME "c4xxxvf"
#define ADF_DH895XCC_PCI_DEVICE_ID 0x435
#define ADF_DH895XCCIOV_PCI_DEVICE_ID 0x443
#define ADF_C62X_PCI_DEVICE_ID 0x37c8
#define ADF_C62XIOV_PCI_DEVICE_ID 0x37c9
#define ADF_C3XXX_PCI_DEVICE_ID 0x19e2
#define ADF_C3XXXIOV_PCI_DEVICE_ID 0x19e3
#define ADF_200XX_PCI_DEVICE_ID 0x18ee
#define ADF_200XXIOV_PCI_DEVICE_ID 0x18ef
#define ADF_D15XX_PCI_DEVICE_ID 0x6f54
#define ADF_D15XXIOV_PCI_DEVICE_ID 0x6f55
#define ADF_C4XXX_PCI_DEVICE_ID 0x18a0
#define ADF_C4XXXIOV_PCI_DEVICE_ID 0x18a1
#define IS_QAT_GEN3(ID) ({ (ID == ADF_C4XXX_PCI_DEVICE_ID); })
#define ADF_VF2PF_SET_SIZE 32
#define ADF_MAX_VF2PF_SET 4
#define ADF_VF2PF_SET_OFFSET(set_nr) ((set_nr)*ADF_VF2PF_SET_SIZE)
#define ADF_VF2PF_VFNR_TO_SET(vf_nr) ((vf_nr) / ADF_VF2PF_SET_SIZE)
#define ADF_VF2PF_VFNR_TO_MASK(vf_nr) \
({ \
u32 vf_nr_ = (vf_nr); \
BIT((vf_nr_)-ADF_VF2PF_SET_SIZE *ADF_VF2PF_VFNR_TO_SET( \
vf_nr_)); \
})
#define ADF_DEVICE_FUSECTL_OFFSET 0x40
#define ADF_DEVICE_LEGFUSE_OFFSET 0x4C
#define ADF_DEVICE_FUSECTL_MASK 0x80000000
#define ADF_PCI_MAX_BARS 3
#define ADF_DEVICE_NAME_LENGTH 32
#define ADF_ETR_MAX_RINGS_PER_BANK 16
#define ADF_MAX_MSIX_VECTOR_NAME 16
#define ADF_DEVICE_NAME_PREFIX "qat_"
#define ADF_STOP_RETRY 50
#define ADF_NUM_THREADS_PER_AE (8)
#define ADF_AE_ADMIN_THREAD (7)
#define ADF_NUM_PKE_STRAND (2)
#define ADF_AE_STRAND0_THREAD (8)
#define ADF_AE_STRAND1_THREAD (9)
#define ADF_NUM_HB_CNT_PER_AE (ADF_NUM_THREADS_PER_AE + ADF_NUM_PKE_STRAND)
#define ADF_CFG_NUM_SERVICES 4
#define ADF_SRV_TYPE_BIT_LEN 3
#define ADF_SRV_TYPE_MASK 0x7
#define ADF_RINGS_PER_SRV_TYPE 2
#define ADF_THRD_ABILITY_BIT_LEN 4
#define ADF_THRD_ABILITY_MASK 0xf
#define ADF_VF_OFFSET 0x8
#define ADF_MAX_FUNC_PER_DEV 0x7
#define ADF_PCI_DEV_OFFSET 0x3
#define ADF_SRV_TYPE_BIT_LEN 3
#define ADF_SRV_TYPE_MASK 0x7
#define GET_SRV_TYPE(ena_srv_mask, srv) \
(((ena_srv_mask) >> (ADF_SRV_TYPE_BIT_LEN * (srv))) & ADF_SRV_TYPE_MASK)
#define ADF_DEFAULT_RING_TO_SRV_MAP \
(CRYPTO | CRYPTO << ADF_CFG_SERV_RING_PAIR_1_SHIFT | \
NA << ADF_CFG_SERV_RING_PAIR_2_SHIFT | \
COMP << ADF_CFG_SERV_RING_PAIR_3_SHIFT)
enum adf_accel_capabilities {
ADF_ACCEL_CAPABILITIES_NULL = 0,
ADF_ACCEL_CAPABILITIES_CRYPTO_SYMMETRIC = 1,
ADF_ACCEL_CAPABILITIES_CRYPTO_ASYMMETRIC = 2,
ADF_ACCEL_CAPABILITIES_CIPHER = 4,
ADF_ACCEL_CAPABILITIES_AUTHENTICATION = 8,
ADF_ACCEL_CAPABILITIES_COMPRESSION = 32,
ADF_ACCEL_CAPABILITIES_DEPRECATED = 64,
ADF_ACCEL_CAPABILITIES_RANDOM_NUMBER = 128
};
struct adf_bar {
rman_res_t base_addr;
struct resource *virt_addr;
rman_res_t size;
} __packed;
struct adf_accel_msix {
struct msix_entry *entries;
u32 num_entries;
} __packed;
struct adf_accel_pci {
device_t pci_dev;
struct adf_accel_msix msix_entries;
struct adf_bar pci_bars[ADF_PCI_MAX_BARS];
uint8_t revid;
uint8_t sku;
int node;
} __packed;
enum dev_state { DEV_DOWN = 0, DEV_UP };
enum dev_sku_info {
DEV_SKU_1 = 0,
DEV_SKU_2,
DEV_SKU_3,
DEV_SKU_4,
DEV_SKU_VF,
DEV_SKU_1_CY,
DEV_SKU_2_CY,
DEV_SKU_3_CY,
DEV_SKU_UNKNOWN
};
static inline const char *
get_sku_info(enum dev_sku_info info)
{
switch (info) {
case DEV_SKU_1:
return "SKU1";
case DEV_SKU_1_CY:
return "SKU1CY";
case DEV_SKU_2:
return "SKU2";
case DEV_SKU_2_CY:
return "SKU2CY";
case DEV_SKU_3:
return "SKU3";
case DEV_SKU_3_CY:
return "SKU3CY";
case DEV_SKU_4:
return "SKU4";
case DEV_SKU_VF:
return "SKUVF";
case DEV_SKU_UNKNOWN:
default:
break;
}
return "Unknown SKU";
}
enum adf_accel_unit_services {
ADF_ACCEL_SERVICE_NULL = 0,
ADF_ACCEL_INLINE_CRYPTO = 1,
ADF_ACCEL_CRYPTO = 2,
ADF_ACCEL_COMPRESSION = 4
};
struct adf_ae_info {
u32 num_asym_thd;
u32 num_sym_thd;
u32 num_dc_thd;
} __packed;
struct adf_accel_unit {
u8 au_mask;
u32 accel_mask;
u64 ae_mask;
u64 comp_ae_mask;
u32 num_ae;
enum adf_accel_unit_services services;
} __packed;
struct adf_accel_unit_info {
u32 inline_ingress_msk;
u32 inline_egress_msk;
u32 sym_ae_msk;
u32 asym_ae_msk;
u32 dc_ae_msk;
u8 num_cy_au;
u8 num_dc_au;
u8 num_inline_au;
struct adf_accel_unit *au;
const struct adf_ae_info *ae_info;
} __packed;
struct adf_hw_aram_info {
/* Inline Egress mask. "1" = AE is working with egress traffic */
u32 inline_direction_egress_mask;
/* Inline congestion managmenet profiles set in config file */
u32 inline_congest_mngt_profile;
/* Initialise CY AE mask, "1" = AE is used for CY operations */
u32 cy_ae_mask;
/* Initialise DC AE mask, "1" = AE is used for DC operations */
u32 dc_ae_mask;
/* Number of long words used to define the ARAM regions */
u32 num_aram_lw_entries;
/* ARAM region definitions */
u32 mmp_region_size;
u32 mmp_region_offset;
u32 skm_region_size;
u32 skm_region_offset;
/*
* Defines size and offset of compression intermediate buffers stored
* in ARAM (device's on-chip memory).
*/
u32 inter_buff_aram_region_size;
u32 inter_buff_aram_region_offset;
u32 sadb_region_size;
u32 sadb_region_offset;
} __packed;
struct adf_hw_device_class {
const char *name;
const enum adf_device_type type;
uint32_t instances;
} __packed;
struct arb_info {
u32 arbiter_offset;
u32 wrk_thd_2_srv_arb_map;
u32 wrk_cfg_offset;
} __packed;
struct admin_info {
u32 admin_msg_ur;
u32 admin_msg_lr;
u32 mailbox_offset;
} __packed;
struct adf_cfg_device_data;
struct adf_accel_dev;
struct adf_etr_data;
struct adf_etr_ring_data;
struct adf_hw_device_data {
struct adf_hw_device_class *dev_class;
uint32_t (*get_accel_mask)(struct adf_accel_dev *accel_dev);
uint32_t (*get_ae_mask)(struct adf_accel_dev *accel_dev);
uint32_t (*get_sram_bar_id)(struct adf_hw_device_data *self);
uint32_t (*get_misc_bar_id)(struct adf_hw_device_data *self);
uint32_t (*get_etr_bar_id)(struct adf_hw_device_data *self);
uint32_t (*get_num_aes)(struct adf_hw_device_data *self);
uint32_t (*get_num_accels)(struct adf_hw_device_data *self);
void (*notify_and_wait_ethernet)(struct adf_accel_dev *accel_dev);
bool (*get_eth_doorbell_msg)(struct adf_accel_dev *accel_dev);
uint32_t (*get_pf2vf_offset)(uint32_t i);
uint32_t (*get_vintmsk_offset)(uint32_t i);
u32 (*get_vintsou_offset)(void);
void (*get_arb_info)(struct arb_info *arb_csrs_info);
void (*get_admin_info)(struct admin_info *admin_csrs_info);
void (*get_errsou_offset)(u32 *errsou3, u32 *errsou5);
uint32_t (*get_num_accel_units)(struct adf_hw_device_data *self);
int (*init_accel_units)(struct adf_accel_dev *accel_dev);
void (*exit_accel_units)(struct adf_accel_dev *accel_dev);
uint32_t (*get_clock_speed)(struct adf_hw_device_data *self);
enum dev_sku_info (*get_sku)(struct adf_hw_device_data *self);
bool (*check_prod_sku)(struct adf_accel_dev *accel_dev);
int (*alloc_irq)(struct adf_accel_dev *accel_dev);
void (*free_irq)(struct adf_accel_dev *accel_dev);
void (*enable_error_correction)(struct adf_accel_dev *accel_dev);
int (*check_uncorrectable_error)(struct adf_accel_dev *accel_dev);
void (*print_err_registers)(struct adf_accel_dev *accel_dev);
void (*disable_error_interrupts)(struct adf_accel_dev *accel_dev);
int (*init_ras)(struct adf_accel_dev *accel_dev);
void (*exit_ras)(struct adf_accel_dev *accel_dev);
void (*disable_arb)(struct adf_accel_dev *accel_dev);
void (*update_ras_errors)(struct adf_accel_dev *accel_dev, int error);
bool (*ras_interrupts)(struct adf_accel_dev *accel_dev,
bool *reset_required);
int (*init_admin_comms)(struct adf_accel_dev *accel_dev);
void (*exit_admin_comms)(struct adf_accel_dev *accel_dev);
int (*send_admin_init)(struct adf_accel_dev *accel_dev);
void (*set_asym_rings_mask)(struct adf_accel_dev *accel_dev);
int (*get_ring_to_svc_map)(struct adf_accel_dev *accel_dev,
u16 *ring_to_svc_map);
uint32_t (*get_accel_cap)(struct adf_accel_dev *accel_dev);
int (*init_arb)(struct adf_accel_dev *accel_dev);
void (*exit_arb)(struct adf_accel_dev *accel_dev);
void (*get_arb_mapping)(struct adf_accel_dev *accel_dev,
const uint32_t **cfg);
int (*get_heartbeat_status)(struct adf_accel_dev *accel_dev);
uint32_t (*get_ae_clock)(struct adf_hw_device_data *self);
void (*disable_iov)(struct adf_accel_dev *accel_dev);
void (*configure_iov_threads)(struct adf_accel_dev *accel_dev,
bool enable);
void (*enable_ints)(struct adf_accel_dev *accel_dev);
bool (*check_slice_hang)(struct adf_accel_dev *accel_dev);
int (*set_ssm_wdtimer)(struct adf_accel_dev *accel_dev);
int (*enable_vf2pf_comms)(struct adf_accel_dev *accel_dev);
int (*disable_vf2pf_comms)(struct adf_accel_dev *accel_dev);
void (*reset_device)(struct adf_accel_dev *accel_dev);
void (*reset_hw_units)(struct adf_accel_dev *accel_dev);
int (*measure_clock)(struct adf_accel_dev *accel_dev);
void (*restore_device)(struct adf_accel_dev *accel_dev);
uint32_t (*get_obj_cfg_ae_mask)(struct adf_accel_dev *accel_dev,
enum adf_accel_unit_services services);
int (*add_pke_stats)(struct adf_accel_dev *accel_dev);
void (*remove_pke_stats)(struct adf_accel_dev *accel_dev);
int (*add_misc_error)(struct adf_accel_dev *accel_dev);
int (*count_ras_event)(struct adf_accel_dev *accel_dev,
u32 *ras_event,
char *aeidstr);
void (*remove_misc_error)(struct adf_accel_dev *accel_dev);
int (*configure_accel_units)(struct adf_accel_dev *accel_dev);
uint32_t (*get_objs_num)(struct adf_accel_dev *accel_dev);
const char *(*get_obj_name)(struct adf_accel_dev *accel_dev,
enum adf_accel_unit_services services);
void (*pre_reset)(struct adf_accel_dev *accel_dev);
void (*post_reset)(struct adf_accel_dev *accel_dev);
const char *fw_name;
const char *fw_mmp_name;
bool reset_ack;
uint32_t fuses;
uint32_t accel_capabilities_mask;
uint32_t instance_id;
uint16_t accel_mask;
u32 aerucm_mask;
u32 ae_mask;
u32 service_mask;
uint16_t tx_rings_mask;
uint8_t tx_rx_gap;
uint8_t num_banks;
u8 num_rings_per_bank;
uint8_t num_accel;
uint8_t num_logical_accel;
uint8_t num_engines;
uint8_t min_iov_compat_ver;
int (*get_storage_enabled)(struct adf_accel_dev *accel_dev,
uint32_t *storage_enabled);
u8 query_storage_cap;
u32 clock_frequency;
u8 storage_enable;
u32 extended_dc_capabilities;
int (*config_device)(struct adf_accel_dev *accel_dev);
u16 asym_rings_mask;
int (*get_fw_image_type)(struct adf_accel_dev *accel_dev,
enum adf_cfg_fw_image_type *fw_image_type);
u16 ring_to_svc_map;
} __packed;
/* helper enum for performing CSR operations */
enum operation {
AND,
OR,
};
/* 32-bit CSR write macro */
#define ADF_CSR_WR(csr_base, csr_offset, val) \
bus_write_4(csr_base, csr_offset, val)
/* 64-bit CSR write macro */
#ifdef __x86_64__
#define ADF_CSR_WR64(csr_base, csr_offset, val) \
bus_write_8(csr_base, csr_offset, val)
#else
static __inline void
adf_csr_wr64(struct resource *csr_base, bus_size_t offset, uint64_t value)
{
bus_write_4(csr_base, offset, (uint32_t)value);
bus_write_4(csr_base, offset + 4, (uint32_t)(value >> 32));
}
#define ADF_CSR_WR64(csr_base, csr_offset, val) \
adf_csr_wr64(csr_base, csr_offset, val)
#endif
/* 32-bit CSR read macro */
#define ADF_CSR_RD(csr_base, csr_offset) bus_read_4(csr_base, csr_offset)
/* 64-bit CSR read macro */
#ifdef __x86_64__
#define ADF_CSR_RD64(csr_base, csr_offset) bus_read_8(csr_base, csr_offset)
#else
static __inline uint64_t
adf_csr_rd64(struct resource *csr_base, bus_size_t offset)
{
return (((uint64_t)bus_read_4(csr_base, offset)) |
(((uint64_t)bus_read_4(csr_base, offset + 4)) << 32));
}
#define ADF_CSR_RD64(csr_base, csr_offset) adf_csr_rd64(csr_base, csr_offset)
#endif
#define GET_DEV(accel_dev) ((accel_dev)->accel_pci_dev.pci_dev)
#define GET_BARS(accel_dev) ((accel_dev)->accel_pci_dev.pci_bars)
#define GET_HW_DATA(accel_dev) (accel_dev->hw_device)
#define GET_MAX_BANKS(accel_dev) (GET_HW_DATA(accel_dev)->num_banks)
#define GET_DEV_SKU(accel_dev) (accel_dev->accel_pci_dev.sku)
#define GET_NUM_RINGS_PER_BANK(accel_dev) \
(GET_HW_DATA(accel_dev)->num_rings_per_bank)
#define GET_MAX_ACCELENGINES(accel_dev) (GET_HW_DATA(accel_dev)->num_engines)
#define accel_to_pci_dev(accel_ptr) accel_ptr->accel_pci_dev.pci_dev
#define GET_SRV_TYPE(ena_srv_mask, srv) \
(((ena_srv_mask) >> (ADF_SRV_TYPE_BIT_LEN * (srv))) & ADF_SRV_TYPE_MASK)
#define SET_ASYM_MASK(asym_mask, srv) \
({ \
typeof(srv) srv_ = (srv); \
(asym_mask) |= ((1 << (srv_)*ADF_RINGS_PER_SRV_TYPE) | \
(1 << ((srv_)*ADF_RINGS_PER_SRV_TYPE + 1))); \
})
#define GET_NUM_RINGS_PER_BANK(accel_dev) \
(GET_HW_DATA(accel_dev)->num_rings_per_bank)
#define GET_MAX_PROCESSES(accel_dev) \
({ \
typeof(accel_dev) dev = (accel_dev); \
(GET_MAX_BANKS(dev) * (GET_NUM_RINGS_PER_BANK(dev) / 2)); \
})
#define GET_DU_TABLE(accel_dev) (accel_dev->du_table)
static inline void
adf_csr_fetch_and_and(struct resource *csr, size_t offs, unsigned long mask)
{
unsigned int val = ADF_CSR_RD(csr, offs);
val &= mask;
ADF_CSR_WR(csr, offs, val);
}
static inline void
adf_csr_fetch_and_or(struct resource *csr, size_t offs, unsigned long mask)
{
unsigned int val = ADF_CSR_RD(csr, offs);
val |= mask;
ADF_CSR_WR(csr, offs, val);
}
static inline void
adf_csr_fetch_and_update(enum operation op,
struct resource *csr,
size_t offs,
unsigned long mask)
{
switch (op) {
case AND:
adf_csr_fetch_and_and(csr, offs, mask);
break;
case OR:
adf_csr_fetch_and_or(csr, offs, mask);
break;
}
}
struct pfvf_stats {
struct dentry *stats_file;
/* Messages put in CSR */
unsigned int tx;
/* Messages read from CSR */
unsigned int rx;
/* Interrupt fired but int bit was clear */
unsigned int spurious;
/* Block messages sent */
unsigned int blk_tx;
/* Block messages received */
unsigned int blk_rx;
/* Blocks received with CRC errors */
unsigned int crc_err;
/* CSR in use by other side */
unsigned int busy;
/* Receiver did not acknowledge */
unsigned int no_ack;
/* Collision detected */
unsigned int collision;
/* Couldn't send a response */
unsigned int tx_timeout;
/* Didn't receive a response */
unsigned int rx_timeout;
/* Responses received */
unsigned int rx_rsp;
/* Messages re-transmitted */
unsigned int retry;
/* Event put timeout */
unsigned int event_timeout;
};
#define NUM_PFVF_COUNTERS 14
void adf_get_admin_info(struct admin_info *admin_csrs_info);
struct adf_admin_comms {
bus_addr_t phy_addr;
bus_addr_t const_tbl_addr;
bus_addr_t aram_map_phys_addr;
bus_addr_t phy_hb_addr;
bus_dmamap_t aram_map;
bus_dmamap_t const_tbl_map;
bus_dmamap_t hb_map;
char *virt_addr;
char *virt_hb_addr;
struct resource *mailbox_addr;
struct sx lock;
struct bus_dmamem dma_mem;
struct bus_dmamem dma_hb;
};
struct icp_qat_fw_loader_handle;
struct adf_fw_loader_data {
struct icp_qat_fw_loader_handle *fw_loader;
const struct firmware *uof_fw;
const struct firmware *mmp_fw;
};
struct adf_accel_vf_info {
struct adf_accel_dev *accel_dev;
struct mutex pf2vf_lock; /* protect CSR access for PF2VF messages */
u32 vf_nr;
bool init;
u8 compat_ver;
struct pfvf_stats pfvf_counters;
};
struct adf_fw_versions {
u8 fw_version_major;
u8 fw_version_minor;
u8 fw_version_patch;
u8 mmp_version_major;
u8 mmp_version_minor;
u8 mmp_version_patch;
};
#define ADF_COMPAT_CHECKER_MAX 8
typedef int (*adf_iov_compat_checker_t)(struct adf_accel_dev *accel_dev,
u8 vf_compat_ver);
struct adf_accel_compat_manager {
u8 num_chker;
adf_iov_compat_checker_t iov_compat_checkers[ADF_COMPAT_CHECKER_MAX];
};
struct adf_heartbeat;
struct adf_accel_dev {
struct adf_hw_aram_info *aram_info;
struct adf_accel_unit_info *au_info;
struct adf_etr_data *transport;
struct adf_hw_device_data *hw_device;
struct adf_cfg_device_data *cfg;
struct adf_fw_loader_data *fw_loader;
struct adf_admin_comms *admin;
struct adf_heartbeat *heartbeat;
struct adf_fw_versions fw_versions;
unsigned int autoreset_on_error;
struct adf_fw_counters_data *fw_counters_data;
struct sysctl_oid *debugfs_ae_config;
struct list_head crypto_list;
atomic_t *ras_counters;
unsigned long status;
atomic_t ref_count;
bus_dma_tag_t dma_tag;
struct sysctl_ctx_list sysctl_ctx;
struct sysctl_oid *ras_correctable;
struct sysctl_oid *ras_uncorrectable;
struct sysctl_oid *ras_fatal;
struct sysctl_oid *ras_reset;
struct sysctl_oid *pke_replay_dbgfile;
struct sysctl_oid *misc_error_dbgfile;
struct list_head list;
struct adf_accel_pci accel_pci_dev;
struct adf_accel_compat_manager *cm;
u8 compat_ver;
union {
struct {
/* vf_info is non-zero when SR-IOV is init'ed */
struct adf_accel_vf_info *vf_info;
int num_vfs;
} pf;
struct {
struct resource *irq;
void *cookie;
char *irq_name;
struct task pf2vf_bh_tasklet;
struct mutex vf2pf_lock; /* protect CSR access */
int iov_msg_completion;
uint8_t compatible;
uint8_t pf_version;
u8 pf2vf_block_byte;
u8 pf2vf_block_resp_type;
struct pfvf_stats pfvf_counters;
} vf;
} u1;
bool is_vf;
u32 accel_id;
void *lac_dev;
};
#endif

79
sys/dev/qat/include/common/adf_cfg.h Normal file
View File

@ -0,0 +1,79 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_CFG_H_
#define ADF_CFG_H_
#include <linux/rwsem.h>
#include "adf_accel_devices.h"
#include "adf_cfg_common.h"
#include "adf_cfg_strings.h"
struct adf_cfg_key_val {
char key[ADF_CFG_MAX_KEY_LEN_IN_BYTES];
char val[ADF_CFG_MAX_VAL_LEN_IN_BYTES];
enum adf_cfg_val_type type;
struct list_head list;
};
struct adf_cfg_section {
char name[ADF_CFG_MAX_SECTION_LEN_IN_BYTES];
bool processed;
bool is_derived;
struct list_head list;
struct list_head param_head;
};
struct adf_cfg_device_data {
struct adf_cfg_device *dev;
struct list_head sec_list;
struct sysctl_oid *debug;
struct sx lock;
};
struct adf_cfg_depot_list {
struct list_head sec_list;
};
int adf_cfg_dev_add(struct adf_accel_dev *accel_dev);
void adf_cfg_dev_remove(struct adf_accel_dev *accel_dev);
int adf_cfg_depot_restore_all(struct adf_accel_dev *accel_dev,
struct adf_cfg_depot_list *dev_hp_cfg);
int adf_cfg_section_add(struct adf_accel_dev *accel_dev, const char *name);
void adf_cfg_del_all(struct adf_accel_dev *accel_dev);
void adf_cfg_depot_del_all(struct list_head *head);
int adf_cfg_add_key_value_param(struct adf_accel_dev *accel_dev,
const char *section_name,
const char *key,
const void *val,
enum adf_cfg_val_type type);
int adf_cfg_get_param_value(struct adf_accel_dev *accel_dev,
const char *section,
const char *name,
char *value);
int adf_cfg_save_section(struct adf_accel_dev *accel_dev,
const char *name,
struct adf_cfg_section *section);
int adf_cfg_depot_save_all(struct adf_accel_dev *accel_dev,
struct adf_cfg_depot_list *dev_hp_cfg);
struct adf_cfg_section *adf_cfg_sec_find(struct adf_accel_dev *accel_dev,
const char *sec_name);
int adf_cfg_derived_section_add(struct adf_accel_dev *accel_dev,
const char *name);
int adf_cfg_remove_key_param(struct adf_accel_dev *accel_dev,
const char *section_name,
const char *key);
int adf_cfg_setup_irq(struct adf_accel_dev *accel_dev);
void adf_cfg_set_asym_rings_mask(struct adf_accel_dev *accel_dev);
void adf_cfg_gen_dispatch_arbiter(struct adf_accel_dev *accel_dev,
const u32 *thrd_to_arb_map,
u32 *thrd_to_arb_map_gen,
u32 total_engines);
int adf_cfg_get_fw_image_type(struct adf_accel_dev *accel_dev,
enum adf_cfg_fw_image_type *fw_image_type);
int adf_cfg_get_services_enabled(struct adf_accel_dev *accel_dev,
u16 *ring_to_svc_map);
int adf_cfg_restore_section(struct adf_accel_dev *accel_dev,
struct adf_cfg_section *section);
void adf_cfg_keyval_del_all(struct list_head *head);
#endif

211
sys/dev/qat/include/common/adf_cfg_common.h Normal file
View File

@ -0,0 +1,211 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_CFG_COMMON_H_
#define ADF_CFG_COMMON_H_
#include <sys/types.h>
#include <sys/ioccom.h>
#include <sys/cpuset.h>
#define ADF_CFG_MAX_STR_LEN 128
#define ADF_CFG_MAX_KEY_LEN_IN_BYTES ADF_CFG_MAX_STR_LEN
/*
* Max value length increased to 128 to support more length of values.
* like Dc0CoreAffinity = 0, 1, 2,... config values to max cores
*/
#define ADF_CFG_MAX_VAL_LEN_IN_BYTES 128
#define ADF_CFG_MAX_SECTION_LEN_IN_BYTES ADF_CFG_MAX_STR_LEN
#define ADF_CFG_NULL_TERM_SIZE 1
#define ADF_CFG_BASE_DEC 10
#define ADF_CFG_BASE_HEX 16
#define ADF_CFG_ALL_DEVICES 0xFFFE
#define ADF_CFG_NO_DEVICE 0xFFFF
#define ADF_CFG_AFFINITY_WHATEVER 0xFF
#define MAX_DEVICE_NAME_SIZE 32
#define ADF_MAX_DEVICES (32 * 32)
#define ADF_MAX_ACCELENGINES 12
#define ADF_CFG_STORAGE_ENABLED 1
#define ADF_DEVS_ARRAY_SIZE BITS_TO_LONGS(ADF_MAX_DEVICES)
#define ADF_SSM_WDT_PKE_DEFAULT_VALUE 0x3000000
#define ADF_WDT_TIMER_SYM_COMP_MS 3
#define ADF_MIN_HB_TIMER_MS 100
#define ADF_CFG_MAX_NUM_OF_SECTIONS 16
#define ADF_CFG_MAX_NUM_OF_TOKENS 16
#define ADF_CFG_MAX_TOKENS_IN_CONFIG 8
#define ADF_CFG_RESP_POLL 1
#define ADF_CFG_RESP_EPOLL 2
#define ADF_CFG_DEF_CY_RING_ASYM_SIZE 64
#define ADF_CFG_DEF_CY_RING_SYM_SIZE 512
#define ADF_CFG_DEF_DC_RING_SIZE 512
#define ADF_CFG_MAX_CORE_NUM 256
#define ADF_CFG_MAX_TOKENS ADF_CFG_MAX_CORE_NUM
#define ADF_CFG_MAX_TOKEN_LEN 10
#define ADF_CFG_ACCEL_DEF_COALES 1
#define ADF_CFG_ACCEL_DEF_COALES_TIMER 10000
#define ADF_CFG_ACCEL_DEF_COALES_NUM_MSG 0
#define ADF_CFG_ASYM_SRV_MASK 1
#define ADF_CFG_SYM_SRV_MASK 2
#define ADF_CFG_DC_SRV_MASK 8
#define ADF_CFG_UNKNOWN_SRV_MASK 0
#define ADF_CFG_DEF_ASYM_MASK 0x03
#define ADF_CFG_MAX_SERVICES 4
#define ADF_MAX_SERVICES 3
enum adf_svc_type {
ADF_SVC_ASYM = 0,
ADF_SVC_SYM = 1,
ADF_SVC_DC = 2,
ADF_SVC_NONE = 3
};
struct adf_pci_address {
unsigned char bus;
unsigned char dev;
unsigned char func;
} __packed;
#define ADF_CFG_SERV_RING_PAIR_0_SHIFT 0
#define ADF_CFG_SERV_RING_PAIR_1_SHIFT 3
#define ADF_CFG_SERV_RING_PAIR_2_SHIFT 6
#define ADF_CFG_SERV_RING_PAIR_3_SHIFT 9
enum adf_cfg_service_type { NA = 0, CRYPTO, COMP, SYM, ASYM, USED };
enum adf_cfg_bundle_type { FREE, KERNEL, USER };
enum adf_cfg_val_type { ADF_DEC, ADF_HEX, ADF_STR };
enum adf_device_type {
DEV_UNKNOWN = 0,
DEV_DH895XCC,
DEV_DH895XCCVF,
DEV_C62X,
DEV_C62XVF,
DEV_C3XXX,
DEV_C3XXXVF,
DEV_200XX,
DEV_200XXVF,
DEV_C4XXX,
DEV_C4XXXVF
};
enum adf_cfg_fw_image_type {
ADF_FW_IMAGE_DEFAULT = 0,
ADF_FW_IMAGE_CRYPTO,
ADF_FW_IMAGE_COMPRESSION,
ADF_FW_IMAGE_CUSTOM1
};
struct adf_dev_status_info {
enum adf_device_type type;
uint16_t accel_id;
uint16_t instance_id;
uint8_t num_ae;
uint8_t num_accel;
uint8_t num_logical_accel;
uint8_t banks_per_accel;
uint8_t state;
uint8_t bus;
uint8_t dev;
uint8_t fun;
int domain;
char name[MAX_DEVICE_NAME_SIZE];
u8 sku;
u32 node_id;
u32 device_mem_available;
u32 pci_device_id;
};
struct adf_cfg_device {
/* contains all the bundles info */
struct adf_cfg_bundle **bundles;
/* contains all the instances info */
struct adf_cfg_instance **instances;
int bundle_num;
int instance_index;
char name[ADF_CFG_MAX_STR_LEN];
int dev_id;
int max_kernel_bundle_nr;
u16 total_num_inst;
};
enum adf_accel_serv_type {
ADF_ACCEL_SERV_NA = 0x0,
ADF_ACCEL_SERV_ASYM,
ADF_ACCEL_SERV_SYM,
ADF_ACCEL_SERV_RND,
ADF_ACCEL_SERV_DC
};
struct adf_cfg_ring {
u8 mode : 1;
enum adf_accel_serv_type serv_type;
u8 number : 4;
};
struct adf_cfg_bundle {
/* Section(s) name this bundle is shared by */
char **sections;
int max_section;
int section_index;
int number;
enum adf_cfg_bundle_type type;
cpuset_t affinity_mask;
int polling_mode;
int instance_num;
int num_of_rings;
/* contains all the info about rings */
struct adf_cfg_ring **rings;
u16 in_use;
};
struct adf_cfg_instance {
enum adf_cfg_service_type stype;
char name[ADF_CFG_MAX_STR_LEN];
int polling_mode;
cpuset_t affinity_mask;
/* rings within an instance for services */
int asym_tx;
int asym_rx;
int sym_tx;
int sym_rx;
int dc_tx;
int dc_rx;
int bundle;
};
#define ADF_CFG_MAX_CORE_NUM 256
#define ADF_CFG_MAX_TOKENS_IN_CONFIG 8
#define ADF_CFG_MAX_TOKEN_LEN 10
#define ADF_CFG_MAX_TOKENS ADF_CFG_MAX_CORE_NUM
#define ADF_CFG_ACCEL_DEF_COALES 1
#define ADF_CFG_ACCEL_DEF_COALES_TIMER 10000
#define ADF_CFG_ACCEL_DEF_COALES_NUM_MSG 0
#define ADF_CFG_RESP_EPOLL 2
#define ADF_CFG_SERV_RING_PAIR_1_SHIFT 3
#define ADF_CFG_SERV_RING_PAIR_2_SHIFT 6
#define ADF_CFG_SERV_RING_PAIR_3_SHIFT 9
#define ADF_CFG_RESP_POLL 1
#define ADF_CFG_ASYM_SRV_MASK 1
#define ADF_CFG_SYM_SRV_MASK 2
#define ADF_CFG_DC_SRV_MASK 8
#define ADF_CFG_UNKNOWN_SRV_MASK 0
#define ADF_CFG_DEF_ASYM_MASK 0x03
#define ADF_CFG_MAX_SERVICES 4
#define ADF_CFG_HB_DEFAULT_VALUE 500
#define ADF_CFG_HB_COUNT_THRESHOLD 3
#define ADF_MIN_HB_TIMER_MS 100
enum adf_device_heartbeat_status {
DEV_HB_UNRESPONSIVE = 0,
DEV_HB_ALIVE,
DEV_HB_UNSUPPORTED
};
struct adf_dev_heartbeat_status_ctl {
uint16_t device_id;
enum adf_device_heartbeat_status status;
};
#endif

132
sys/dev/qat/include/common/adf_cfg_strings.h Normal file
View File

@ -0,0 +1,132 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_CFG_STRINGS_H_
#define ADF_CFG_STRINGS_H_
#define ADF_GENERAL_SEC "GENERAL"
#define ADF_KERNEL_SEC "KERNEL"
#define ADF_ACCEL_SEC "Accelerator"
#define ADF_NUM_CY "NumberCyInstances"
#define ADF_NUM_DC "NumberDcInstances"
#define ADF_RING_SYM_SIZE "NumConcurrentSymRequests"
#define ADF_RING_ASYM_SIZE "NumConcurrentAsymRequests"
#define ADF_RING_DC_SIZE "NumConcurrentRequests"
#define ADF_RING_ASYM_TX "RingAsymTx"
#define ADF_RING_SYM_TX "RingSymTx"
#define ADF_RING_RND_TX "RingNrbgTx"
#define ADF_RING_ASYM_RX "RingAsymRx"
#define ADF_RING_SYM_RX "RingSymRx"
#define ADF_RING_RND_RX "RingNrbgRx"
#define ADF_RING_DC_TX "RingTx"
#define ADF_RING_DC_RX "RingRx"
#define ADF_ETRMGR_BANK "Bank"
#define ADF_RING_BANK_NUM "BankNumber"
#define ADF_CY "Cy"
#define ADF_DC "Dc"
#define ADF_DC_EXTENDED_FEATURES "Device_DcExtendedFeatures"
#define ADF_ETRMGR_COALESCING_ENABLED "InterruptCoalescingEnabled"
#define ADF_ETRMGR_COALESCING_ENABLED_FORMAT \
ADF_ETRMGR_BANK "%d" ADF_ETRMGR_COALESCING_ENABLED
#define ADF_ETRMGR_COALESCE_TIMER "InterruptCoalescingTimerNs"
#define ADF_ETRMGR_COALESCE_TIMER_FORMAT \
ADF_ETRMGR_BANK "%d" ADF_ETRMGR_COALESCE_TIMER
#define ADF_ETRMGR_COALESCING_MSG_ENABLED "InterruptCoalescingNumResponses"
#define ADF_ETRMGR_COALESCING_MSG_ENABLED_FORMAT \
ADF_ETRMGR_BANK "%d" ADF_ETRMGR_COALESCING_MSG_ENABLED
#define ADF_ETRMGR_CORE_AFFINITY "CoreAffinity"
#define ADF_ETRMGR_CORE_AFFINITY_FORMAT \
ADF_ETRMGR_BANK "%d" ADF_ETRMGR_CORE_AFFINITY
#define ADF_ACCEL_STR "Accelerator%d"
#define ADF_INLINE_SEC "INLINE"
#define ADF_NUM_CY_ACCEL_UNITS "NumCyAccelUnits"
#define ADF_NUM_DC_ACCEL_UNITS "NumDcAccelUnits"
#define ADF_NUM_INLINE_ACCEL_UNITS "NumInlineAccelUnits"
#define ADF_INLINE_INGRESS "InlineIngress"
#define ADF_INLINE_EGRESS "InlineEgress"
#define ADF_INLINE_CONGEST_MNGT_PROFILE "InlineCongestionManagmentProfile"
#define ADF_INLINE_IPSEC_ALGO_GROUP "InlineIPsecAlgoGroup"
#define ADF_SERVICE_CY "cy"
#define ADF_SERVICE_SYM "sym"
#define ADF_SERVICE_DC "dc"
#define ADF_CFG_CY "cy"
#define ADF_CFG_DC "dc"
#define ADF_CFG_ASYM "asym"
#define ADF_CFG_SYM "sym"
#define ADF_SERVICE_INLINE "inline"
#define ADF_SERVICES_ENABLED "ServicesEnabled"
#define ADF_SERVICES_SEPARATOR ";"
#define ADF_DEV_SSM_WDT_BULK "CySymAndDcWatchDogTimer"
#define ADF_DEV_SSM_WDT_PKE "CyAsymWatchDogTimer"
#define ADF_DH895XCC_AE_FW_NAME "icp_qat_ae.uof"
#define ADF_CXXX_AE_FW_NAME "icp_qat_ae.suof"
#define ADF_HEARTBEAT_TIMER "HeartbeatTimer"
#define ADF_MMP_VER_KEY "Firmware_MmpVer"
#define ADF_UOF_VER_KEY "Firmware_UofVer"
#define ADF_HW_REV_ID_KEY "HW_RevId"
#define ADF_STORAGE_FIRMWARE_ENABLED "StorageEnabled"
#define ADF_DEV_MAX_BANKS "Device_Max_Banks"
#define ADF_DEV_CAPABILITIES_MASK "Device_Capabilities_Mask"
#define ADF_DEV_NODE_ID "Device_NodeId"
#define ADF_DEV_PKG_ID "Device_PkgId"
#define ADF_FIRST_USER_BUNDLE "FirstUserBundle"
#define ADF_INTERNAL_USERSPACE_SEC_SUFF "_INT_"
#define ADF_LIMIT_DEV_ACCESS "LimitDevAccess"
#define DEV_LIMIT_CFG_ACCESS_TMPL "_D_L_ACC"
#define ADF_DEV_MAX_RINGS_PER_BANK "Device_Max_Rings_Per_Bank"
#define ADF_NUM_PROCESSES "NumProcesses"
#define ADF_DH895XCC_AE_FW_NAME_COMPRESSION "compression.uof"
#define ADF_DH895XCC_AE_FW_NAME_CRYPTO "crypto.uof"
#define ADF_DH895XCC_AE_FW_NAME_CUSTOM1 "custom1.uof"
#define ADF_CXXX_AE_FW_NAME_COMPRESSION "compression.suof"
#define ADF_CXXX_AE_FW_NAME_CRYPTO "crypto.suof"
#define ADF_CXXX_AE_FW_NAME_CUSTOM1 "custom1.suof"
#define ADF_DC_EXTENDED_FEATURES "Device_DcExtendedFeatures"
#define ADF_PKE_DISABLED "PkeServiceDisabled"
#define ADF_INTER_BUF_SIZE "DcIntermediateBufferSizeInKB"
#define ADF_AUTO_RESET_ON_ERROR "AutoResetOnError"
#define ADF_KERNEL_SAL_SEC "KERNEL_QAT"
#define ADF_CFG_DEF_CY_RING_ASYM_SIZE 64
#define ADF_CFG_DEF_CY_RING_SYM_SIZE 512
#define ADF_CFG_DEF_DC_RING_SIZE 512
#define ADF_NUM_PROCESSES "NumProcesses"
#define ADF_SERVICES_ENABLED "ServicesEnabled"
#define ADF_CFG_CY "cy"
#define ADF_CFG_SYM "sym"
#define ADF_CFG_ASYM "asym"
#define ADF_CFG_DC "dc"
#define ADF_POLL_MODE "IsPolled"
#define ADF_DEV_KPT_ENABLE "KptEnabled"
#define ADF_STORAGE_FIRMWARE_ENABLED "StorageEnabled"
#define ADF_RL_FIRMWARE_ENABLED "RateLimitingEnabled"
#define ADF_SERVICES_PROFILE "ServicesProfile"
#define ADF_SERVICES_DEFAULT "DEFAULT"
#define ADF_SERVICES_CRYPTO "CRYPTO"
#define ADF_SERVICES_COMPRESSION "COMPRESSION"
#define ADF_SERVICES_CUSTOM1 "CUSTOM1"
#define ADF_DC_RING_SIZE (ADF_DC ADF_RING_DC_SIZE)
#define ADF_CY_RING_SYM_SIZE (ADF_CY ADF_RING_SYM_SIZE)
#define ADF_CY_RING_ASYM_SIZE (ADF_CY ADF_RING_ASYM_SIZE)
#define ADF_CY_CORE_AFFINITY_FORMAT ADF_CY "%d" ADF_ETRMGR_CORE_AFFINITY
#define ADF_DC_CORE_AFFINITY_FORMAT ADF_DC "%d" ADF_ETRMGR_CORE_AFFINITY
#define ADF_CY_BANK_NUM_FORMAT ADF_CY "%d" ADF_RING_BANK_NUM
#define ADF_DC_BANK_NUM_FORMAT ADF_DC "%d" ADF_RING_BANK_NUM
#define ADF_CY_ASYM_TX_FORMAT ADF_CY "%d" ADF_RING_ASYM_TX
#define ADF_CY_SYM_TX_FORMAT ADF_CY "%d" ADF_RING_SYM_TX
#define ADF_CY_ASYM_RX_FORMAT ADF_CY "%d" ADF_RING_ASYM_RX
#define ADF_CY_SYM_RX_FORMAT ADF_CY "%d" ADF_RING_SYM_RX
#define ADF_DC_TX_FORMAT ADF_DC "%d" ADF_RING_DC_TX
#define ADF_DC_RX_FORMAT ADF_DC "%d" ADF_RING_DC_RX
#define ADF_CY_RING_SYM_SIZE_FORMAT ADF_CY "%d" ADF_RING_SYM_SIZE
#define ADF_CY_RING_ASYM_SIZE_FORMAT ADF_CY "%d" ADF_RING_ASYM_SIZE
#define ADF_DC_RING_SIZE_FORMAT ADF_DC "%d" ADF_RING_DC_SIZE
#define ADF_CY_NAME_FORMAT ADF_CY "%dName"
#define ADF_DC_NAME_FORMAT ADF_DC "%dName"
#define ADF_CY_POLL_MODE_FORMAT ADF_CY "%d" ADF_POLL_MODE
#define ADF_DC_POLL_MODE_FORMAT ADF_DC "%d" ADF_POLL_MODE
#define ADF_USER_SECTION_NAME_FORMAT "%s_INT_%d"
#define ADF_LIMITED_USER_SECTION_NAME_FORMAT "%s_DEV%d_INT_%d"
#define ADF_CONFIG_VERSION "ConfigVersion"
#endif

46
sys/dev/qat/include/common/adf_cfg_user.h Normal file
View File

@ -0,0 +1,46 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_CFG_USER_H_
#define ADF_CFG_USER_H_
#include "adf_cfg_common.h"
#include "adf_cfg_strings.h"
struct adf_user_cfg_key_val {
char key[ADF_CFG_MAX_KEY_LEN_IN_BYTES];
char val[ADF_CFG_MAX_VAL_LEN_IN_BYTES];
union {
struct adf_user_cfg_key_val *next;
uint64_t padding3;
};
enum adf_cfg_val_type type;
};
struct adf_user_cfg_section {
char name[ADF_CFG_MAX_SECTION_LEN_IN_BYTES];
union {
struct adf_user_cfg_key_val *params;
uint64_t padding1;
};
union {
struct adf_user_cfg_section *next;
uint64_t padding3;
};
};
struct adf_user_cfg_ctl_data {
union {
struct adf_user_cfg_section *config_section;
uint64_t padding;
};
u32 device_id;
};
struct adf_user_reserve_ring {
u32 accel_id;
u32 bank_nr;
u32 ring_mask;
};
#endif

368
sys/dev/qat/include/common/adf_common_drv.h Normal file
View File

@ -0,0 +1,368 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_DRV_H
#define ADF_DRV_H
#include <dev/pci/pcivar.h>
#include "adf_accel_devices.h"
#include "icp_qat_fw_loader_handle.h"
#include "icp_qat_hal.h"
#include "adf_cfg_user.h"
#define ADF_MAJOR_VERSION 0
#define ADF_MINOR_VERSION 6
#define ADF_BUILD_VERSION 0
#define ADF_DRV_VERSION \
__stringify(ADF_MAJOR_VERSION) "." __stringify( \
ADF_MINOR_VERSION) "." __stringify(ADF_BUILD_VERSION)
#define ADF_STATUS_RESTARTING 0
#define ADF_STATUS_STARTING 1
#define ADF_STATUS_CONFIGURED 2
#define ADF_STATUS_STARTED 3
#define ADF_STATUS_AE_INITIALISED 4
#define ADF_STATUS_AE_UCODE_LOADED 5
#define ADF_STATUS_AE_STARTED 6
#define ADF_STATUS_PF_RUNNING 7
#define ADF_STATUS_IRQ_ALLOCATED 8
#define ADF_PCIE_FLR_ATTEMPT 10
#define ADF_STATUS_SYSCTL_CTX_INITIALISED 9
#define PCI_EXP_AERUCS 0x104
/* PMISC BAR upper and lower offsets in PCIe config space */
#define ADF_PMISC_L_OFFSET 0x18
#define ADF_PMISC_U_OFFSET 0x1c
enum adf_dev_reset_mode { ADF_DEV_RESET_ASYNC = 0, ADF_DEV_RESET_SYNC };
enum adf_event {
ADF_EVENT_INIT = 0,
ADF_EVENT_START,
ADF_EVENT_STOP,
ADF_EVENT_SHUTDOWN,
ADF_EVENT_RESTARTING,
ADF_EVENT_RESTARTED,
ADF_EVENT_ERROR,
};
struct adf_state {
enum adf_event dev_state;
int dev_id;
};
struct service_hndl {
int (*event_hld)(struct adf_accel_dev *accel_dev, enum adf_event event);
unsigned long init_status[ADF_DEVS_ARRAY_SIZE];
unsigned long start_status[ADF_DEVS_ARRAY_SIZE];
char *name;
struct list_head list;
};
static inline int
get_current_node(void)
{
return PCPU_GET(domain);
}
int adf_service_register(struct service_hndl *service);
int adf_service_unregister(struct service_hndl *service);
int adf_dev_init(struct adf_accel_dev *accel_dev);
int adf_dev_start(struct adf_accel_dev *accel_dev);
int adf_dev_stop(struct adf_accel_dev *accel_dev);
void adf_dev_shutdown(struct adf_accel_dev *accel_dev);
int adf_dev_autoreset(struct adf_accel_dev *accel_dev);
int adf_dev_reset(struct adf_accel_dev *accel_dev,
enum adf_dev_reset_mode mode);
int adf_dev_aer_schedule_reset(struct adf_accel_dev *accel_dev,
enum adf_dev_reset_mode mode);
void adf_error_notifier(uintptr_t arg);
int adf_init_fatal_error_wq(void);
void adf_exit_fatal_error_wq(void);
int adf_iov_putmsg(struct adf_accel_dev *accel_dev, u32 msg, u8 vf_nr);
int adf_iov_notify(struct adf_accel_dev *accel_dev, u32 msg, u8 vf_nr);
void adf_pf2vf_notify_restarting(struct adf_accel_dev *accel_dev);
int adf_notify_fatal_error(struct adf_accel_dev *accel_dev);
void adf_pf2vf_notify_fatal_error(struct adf_accel_dev *accel_dev);
void adf_pf2vf_notify_uncorrectable_error(struct adf_accel_dev *accel_dev);
void adf_pf2vf_notify_heartbeat_error(struct adf_accel_dev *accel_dev);
typedef int (*adf_iov_block_provider)(struct adf_accel_dev *accel_dev,
u8 **buffer,
u8 *length,
u8 *block_version,
u8 compatibility,
u8 byte_num);
int adf_iov_block_provider_register(u8 block_type,
const adf_iov_block_provider provider);
u8 adf_iov_is_block_provider_registered(u8 block_type);
int adf_iov_block_provider_unregister(u8 block_type,
const adf_iov_block_provider provider);
int adf_iov_block_get(struct adf_accel_dev *accel_dev,
u8 block_type,
u8 *block_version,
u8 *buffer,
u8 *length);
u8 adf_pfvf_crc(u8 start_crc, u8 *buf, u8 len);
int adf_iov_init_compat_manager(struct adf_accel_dev *accel_dev,
struct adf_accel_compat_manager **cm);
int adf_iov_shutdown_compat_manager(struct adf_accel_dev *accel_dev,
struct adf_accel_compat_manager **cm);
int adf_iov_register_compat_checker(struct adf_accel_dev *accel_dev,
const adf_iov_compat_checker_t cc);
int adf_iov_unregister_compat_checker(struct adf_accel_dev *accel_dev,
const adf_iov_compat_checker_t cc);
int adf_pf_enable_vf2pf_comms(struct adf_accel_dev *accel_dev);
int adf_pf_disable_vf2pf_comms(struct adf_accel_dev *accel_dev);
int adf_enable_vf2pf_comms(struct adf_accel_dev *accel_dev);
int adf_disable_vf2pf_comms(struct adf_accel_dev *accel_dev);
void adf_vf2pf_req_hndl(struct adf_accel_vf_info *vf_info);
void adf_devmgr_update_class_index(struct adf_hw_device_data *hw_data);
void adf_clean_vf_map(bool);
int adf_sysctl_add_fw_versions(struct adf_accel_dev *accel_dev);
int adf_sysctl_remove_fw_versions(struct adf_accel_dev *accel_dev);
int adf_ctl_dev_register(void);
void adf_ctl_dev_unregister(void);
int adf_pf_vf_capabilities_init(struct adf_accel_dev *accel_dev);
int adf_pf_ext_dc_cap_msg_provider(struct adf_accel_dev *accel_dev,
u8 **buffer,
u8 *length,
u8 *block_version,
u8 compatibility);
int adf_pf_vf_ring_to_svc_init(struct adf_accel_dev *accel_dev);
int adf_pf_ring_to_svc_msg_provider(struct adf_accel_dev *accel_dev,
u8 **buffer,
u8 *length,
u8 *block_version,
u8 compatibility,
u8 byte_num);
int adf_devmgr_add_dev(struct adf_accel_dev *accel_dev,
struct adf_accel_dev *pf);
void adf_devmgr_rm_dev(struct adf_accel_dev *accel_dev,
struct adf_accel_dev *pf);
struct list_head *adf_devmgr_get_head(void);
struct adf_accel_dev *adf_devmgr_get_dev_by_id(uint32_t id);
struct adf_accel_dev *adf_devmgr_get_first(void);
struct adf_accel_dev *adf_devmgr_pci_to_accel_dev(device_t pci_dev);
int adf_devmgr_verify_id(uint32_t *id);
void adf_devmgr_get_num_dev(uint32_t *num);
int adf_devmgr_in_reset(struct adf_accel_dev *accel_dev);
int adf_dev_started(struct adf_accel_dev *accel_dev);
int adf_dev_restarting_notify(struct adf_accel_dev *accel_dev);
int adf_dev_restarting_notify_sync(struct adf_accel_dev *accel_dev);
int adf_dev_restarted_notify(struct adf_accel_dev *accel_dev);
int adf_dev_stop_notify_sync(struct adf_accel_dev *accel_dev);
int adf_ae_init(struct adf_accel_dev *accel_dev);
int adf_ae_shutdown(struct adf_accel_dev *accel_dev);
int adf_ae_fw_load(struct adf_accel_dev *accel_dev);
void adf_ae_fw_release(struct adf_accel_dev *accel_dev);
int adf_ae_start(struct adf_accel_dev *accel_dev);
int adf_ae_stop(struct adf_accel_dev *accel_dev);
int adf_aer_store_ppaerucm_reg(device_t pdev,
struct adf_hw_device_data *hw_data);
int adf_enable_aer(struct adf_accel_dev *accel_dev, device_t *adf);
void adf_disable_aer(struct adf_accel_dev *accel_dev);
void adf_reset_sbr(struct adf_accel_dev *accel_dev);
void adf_reset_flr(struct adf_accel_dev *accel_dev);
void adf_dev_pre_reset(struct adf_accel_dev *accel_dev);
void adf_dev_post_reset(struct adf_accel_dev *accel_dev);
void adf_dev_restore(struct adf_accel_dev *accel_dev);
int adf_init_aer(void);
void adf_exit_aer(void);
int adf_put_admin_msg_sync(struct adf_accel_dev *accel_dev,
u32 ae,
void *in,
void *out);
struct icp_qat_fw_init_admin_req;
struct icp_qat_fw_init_admin_resp;
int adf_send_admin(struct adf_accel_dev *accel_dev,
struct icp_qat_fw_init_admin_req *req,
struct icp_qat_fw_init_admin_resp *resp,
u32 ae_mask);
int adf_config_device(struct adf_accel_dev *accel_dev);
int adf_init_admin_comms(struct adf_accel_dev *accel_dev);
void adf_exit_admin_comms(struct adf_accel_dev *accel_dev);
int adf_send_admin_init(struct adf_accel_dev *accel_dev);
int adf_get_fw_timestamp(struct adf_accel_dev *accel_dev, u64 *timestamp);
int adf_get_fw_pke_stats(struct adf_accel_dev *accel_dev,
u64 *suc_count,
u64 *unsuc_count);
int adf_dev_measure_clock(struct adf_accel_dev *accel_dev,
u32 *frequency,
u32 min,
u32 max);
int adf_clock_debugfs_add(struct adf_accel_dev *accel_dev);
u64 adf_clock_get_current_time(void);
int adf_init_arb(struct adf_accel_dev *accel_dev);
int adf_init_gen2_arb(struct adf_accel_dev *accel_dev);
void adf_exit_arb(struct adf_accel_dev *accel_dev);
void adf_disable_arb(struct adf_accel_dev *accel_dev);
void adf_update_ring_arb(struct adf_etr_ring_data *ring);
void
adf_enable_ring_arb(void *csr_addr, unsigned int bank_nr, unsigned int mask);
void
adf_disable_ring_arb(void *csr_addr, unsigned int bank_nr, unsigned int mask);
int adf_set_ssm_wdtimer(struct adf_accel_dev *accel_dev);
struct adf_accel_dev *adf_devmgr_get_dev_by_bdf(struct adf_pci_address *addr);
struct adf_accel_dev *adf_devmgr_get_dev_by_pci_bus(u8 bus);
int adf_get_vf_nr(struct adf_pci_address *vf_pci_addr, int *vf_nr);
u32 adf_get_slices_for_svc(struct adf_accel_dev *accel_dev,
enum adf_svc_type svc);
bool adf_is_bdf_equal(struct adf_pci_address *bdf1,
struct adf_pci_address *bdf2);
int adf_is_vf_nr_valid(struct adf_accel_dev *accel_dev, int vf_nr);
void adf_dev_get(struct adf_accel_dev *accel_dev);
void adf_dev_put(struct adf_accel_dev *accel_dev);
int adf_dev_in_use(struct adf_accel_dev *accel_dev);
int adf_init_etr_data(struct adf_accel_dev *accel_dev);
void adf_cleanup_etr_data(struct adf_accel_dev *accel_dev);
struct qat_crypto_instance *qat_crypto_get_instance_node(int node);
void qat_crypto_put_instance(struct qat_crypto_instance *inst);
void qat_alg_callback(void *resp);
void qat_alg_asym_callback(void *resp);
int qat_algs_register(void);
void qat_algs_unregister(void);
int qat_asym_algs_register(void);
void qat_asym_algs_unregister(void);
int adf_isr_resource_alloc(struct adf_accel_dev *accel_dev);
void adf_isr_resource_free(struct adf_accel_dev *accel_dev);
int adf_vf_isr_resource_alloc(struct adf_accel_dev *accel_dev);
void adf_vf_isr_resource_free(struct adf_accel_dev *accel_dev);
int qat_hal_init(struct adf_accel_dev *accel_dev);
void qat_hal_deinit(struct icp_qat_fw_loader_handle *handle);
void qat_hal_start(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned int ctx_mask);
void qat_hal_stop(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned int ctx_mask);
void qat_hal_reset(struct icp_qat_fw_loader_handle *handle);
int qat_hal_clr_reset(struct icp_qat_fw_loader_handle *handle);
void qat_hal_set_live_ctx(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned int ctx_mask);
int qat_hal_check_ae_active(struct icp_qat_fw_loader_handle *handle,
unsigned int ae);
int qat_hal_set_ae_lm_mode(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
enum icp_qat_uof_regtype lm_type,
unsigned char mode);
void qat_hal_set_ae_tindex_mode(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned char mode);
void qat_hal_set_ae_scs_mode(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned char mode);
int qat_hal_set_ae_ctx_mode(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned char mode);
int qat_hal_set_ae_nn_mode(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned char mode);
void qat_hal_set_pc(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned int ctx_mask,
unsigned int upc);
void qat_hal_wr_uwords(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned int uaddr,
unsigned int words_num,
const uint64_t *uword);
void qat_hal_wr_coalesce_uwords(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned int uaddr,
unsigned int words_num,
uint64_t *uword);
void qat_hal_wr_umem(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned int uword_addr,
unsigned int words_num,
unsigned int *data);
int qat_hal_get_ins_num(void);
int qat_hal_batch_wr_lm(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
struct icp_qat_uof_batch_init *lm_init_header);
int qat_hal_init_gpr(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned long ctx_mask,
enum icp_qat_uof_regtype reg_type,
unsigned short reg_num,
unsigned int regdata);
int qat_hal_init_wr_xfer(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned long ctx_mask,
enum icp_qat_uof_regtype reg_type,
unsigned short reg_num,
unsigned int regdata);
int qat_hal_init_rd_xfer(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned long ctx_mask,
enum icp_qat_uof_regtype reg_type,
unsigned short reg_num,
unsigned int regdata);
int qat_hal_init_nn(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned long ctx_mask,
unsigned short reg_num,
unsigned int regdata);
int qat_hal_wr_lm(struct icp_qat_fw_loader_handle *handle,
unsigned char ae,
unsigned short lm_addr,
unsigned int value);
int qat_uclo_wr_all_uimage(struct icp_qat_fw_loader_handle *handle);
void qat_uclo_del_obj(struct icp_qat_fw_loader_handle *handle);
void qat_uclo_del_mof(struct icp_qat_fw_loader_handle *handle);
int qat_uclo_wr_mimage(struct icp_qat_fw_loader_handle *handle,
const void *addr_ptr,
int mem_size);
int qat_uclo_map_obj(struct icp_qat_fw_loader_handle *handle,
const void *addr_ptr,
u32 mem_size,
const char *obj_name);
void qat_hal_get_scs_neigh_ae(unsigned char ae, unsigned char *ae_neigh);
int qat_uclo_set_cfg_ae_mask(struct icp_qat_fw_loader_handle *handle,
unsigned int cfg_ae_mask);
void adf_enable_pf2vf_interrupts(struct adf_accel_dev *accel_dev);
void adf_disable_pf2vf_interrupts(struct adf_accel_dev *accel_dev);
int adf_init_vf_wq(void);
void adf_exit_vf_wq(void);
void adf_flush_vf_wq(void);
int adf_vf2pf_init(struct adf_accel_dev *accel_dev);
void adf_vf2pf_shutdown(struct adf_accel_dev *accel_dev);
static inline int
adf_sriov_configure(device_t *pdev, int numvfs)
{
return 0;
}
static inline void
adf_disable_sriov(struct adf_accel_dev *accel_dev)
{
}
static inline void
adf_vf2pf_handler(struct adf_accel_vf_info *vf_info)
{
}
static inline int
adf_init_pf_wq(void)
{
return 0;
}
static inline void
adf_exit_pf_wq(void)
{
}
#endif

27
sys/dev/qat/include/common/adf_transport.h Normal file
View File

@ -0,0 +1,27 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_TRANSPORT_H
#define ADF_TRANSPORT_H
#include "adf_accel_devices.h"
struct adf_etr_ring_data;
typedef void (*adf_callback_fn)(void *resp_msg);
int adf_create_ring(struct adf_accel_dev *accel_dev,
const char *section,
u32 bank_num,
u32 num_mgs,
u32 msg_size,
const char *ring_name,
adf_callback_fn callback,
int poll_mode,
struct adf_etr_ring_data **ring_ptr);
int adf_send_message(struct adf_etr_ring_data *ring, u32 *msg);
void adf_remove_ring(struct adf_etr_ring_data *ring);
int adf_poll_bank(u32 accel_id, u32 bank_num, u32 quota);
int adf_poll_all_banks(u32 accel_id, u32 quota);
#endif /* ADF_TRANSPORT_H */

169
sys/dev/qat/include/common/adf_transport_access_macros.h Normal file
View File

@ -0,0 +1,169 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_TRANSPORT_ACCESS_MACROS_H
#define ADF_TRANSPORT_ACCESS_MACROS_H
#include "adf_accel_devices.h"
#define ADF_BANK_INT_SRC_SEL_MASK_0 0x4444444CUL
#define ADF_BANK_INT_SRC_SEL_MASK_X 0x44444444UL
#define ADF_BANK_INT_FLAG_CLEAR_MASK 0xFFFF
#define ADF_RING_CSR_RING_CONFIG 0x000
#define ADF_RING_CSR_RING_LBASE 0x040
#define ADF_RING_CSR_RING_UBASE 0x080
#define ADF_RING_CSR_RING_HEAD 0x0C0
#define ADF_RING_CSR_RING_TAIL 0x100
#define ADF_RING_CSR_E_STAT 0x14C
#define ADF_RING_CSR_INT_FLAG 0x170
#define ADF_RING_CSR_INT_SRCSEL 0x174
#define ADF_RING_CSR_INT_SRCSEL_2 0x178
#define ADF_RING_CSR_INT_COL_EN 0x17C
#define ADF_RING_CSR_INT_COL_CTL 0x180
#define ADF_RING_CSR_INT_FLAG_AND_COL 0x184
#define ADF_RING_CSR_INT_COL_CTL_ENABLE 0x80000000
#define ADF_RING_BUNDLE_SIZE 0x1000
#define ADF_RING_CONFIG_NEAR_FULL_WM 0x0A
#define ADF_RING_CONFIG_NEAR_EMPTY_WM 0x05
#define ADF_COALESCING_MIN_TIME 0x1FF
#define ADF_COALESCING_MAX_TIME 0xFFFFF
#define ADF_COALESCING_DEF_TIME 0x27FF
#define ADF_RING_NEAR_WATERMARK_512 0x08
#define ADF_RING_NEAR_WATERMARK_0 0x00
#define ADF_RING_EMPTY_SIG 0x7F7F7F7F
/* Valid internal ring size values */
#define ADF_RING_SIZE_128 0x01
#define ADF_RING_SIZE_256 0x02
#define ADF_RING_SIZE_512 0x03
#define ADF_RING_SIZE_4K 0x06
#define ADF_RING_SIZE_16K 0x08
#define ADF_RING_SIZE_4M 0x10
#define ADF_MIN_RING_SIZE ADF_RING_SIZE_128
#define ADF_MAX_RING_SIZE ADF_RING_SIZE_4M
#define ADF_DEFAULT_RING_SIZE ADF_RING_SIZE_16K
/* Valid internal msg size values */
#define ADF_MSG_SIZE_32 0x01
#define ADF_MSG_SIZE_64 0x02
#define ADF_MSG_SIZE_128 0x04
#define ADF_MIN_MSG_SIZE ADF_MSG_SIZE_32
#define ADF_MAX_MSG_SIZE ADF_MSG_SIZE_128
/* Size to bytes conversion macros for ring and msg size values */
#define ADF_MSG_SIZE_TO_BYTES(SIZE) (SIZE << 5)
#define ADF_BYTES_TO_MSG_SIZE(SIZE) (SIZE >> 5)
#define ADF_SIZE_TO_RING_SIZE_IN_BYTES(SIZE) ((1 << (SIZE - 1)) << 7)
#define ADF_RING_SIZE_IN_BYTES_TO_SIZE(SIZE) ((1 << (SIZE - 1)) >> 7)
/* Set the response quota to a high number */
#define ADF_NO_RESPONSE_QUOTA 0xFFFFFFFF
/* Minimum ring bufer size for memory allocation */
#define ADF_RING_SIZE_BYTES_MIN(SIZE) \
((SIZE < ADF_SIZE_TO_RING_SIZE_IN_BYTES(ADF_RING_SIZE_4K)) ? \
ADF_SIZE_TO_RING_SIZE_IN_BYTES(ADF_RING_SIZE_4K) : \
SIZE)
#define ADF_RING_SIZE_MODULO(SIZE) (SIZE + 0x6)
#define ADF_SIZE_TO_POW(SIZE) \
((((SIZE & 0x4) >> 1) | ((SIZE & 0x4) >> 2) | SIZE) & ~0x4)
/* Max outstanding requests */
#define ADF_MAX_INFLIGHTS(RING_SIZE, MSG_SIZE) \
((((1 << (RING_SIZE - 1)) << 3) >> ADF_SIZE_TO_POW(MSG_SIZE)) - 1)
#define BUILD_RING_CONFIG(size) \
((ADF_RING_NEAR_WATERMARK_0 << ADF_RING_CONFIG_NEAR_FULL_WM) | \
(ADF_RING_NEAR_WATERMARK_0 << ADF_RING_CONFIG_NEAR_EMPTY_WM) | size)
#define BUILD_RESP_RING_CONFIG(size, watermark_nf, watermark_ne) \
((watermark_nf << ADF_RING_CONFIG_NEAR_FULL_WM) | \
(watermark_ne << ADF_RING_CONFIG_NEAR_EMPTY_WM) | size)
#define BUILD_RING_BASE_ADDR(addr, size) \
((addr >> 6) & (0xFFFFFFFFFFFFFFFFULL << size))
#define READ_CSR_RING_HEAD(csr_base_addr, bank, ring) \
ADF_CSR_RD(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + ADF_RING_CSR_RING_HEAD + \
(ring << 2))
#define READ_CSR_RING_TAIL(csr_base_addr, bank, ring) \
ADF_CSR_RD(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + ADF_RING_CSR_RING_TAIL + \
(ring << 2))
#define READ_CSR_E_STAT(csr_base_addr, bank) \
ADF_CSR_RD(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + ADF_RING_CSR_E_STAT)
#define WRITE_CSR_RING_CONFIG(csr_base_addr, bank, ring, value) \
ADF_CSR_WR(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + ADF_RING_CSR_RING_CONFIG + \
(ring << 2), \
value)
#define WRITE_CSR_RING_BASE(csr_base_addr, bank, ring, value) \
do { \
uint32_t l_base = 0, u_base = 0; \
l_base = (uint32_t)(value & 0xFFFFFFFF); \
u_base = (uint32_t)((value & 0xFFFFFFFF00000000ULL) >> 32); \
ADF_CSR_WR(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + \
ADF_RING_CSR_RING_LBASE + (ring << 2), \
l_base); \
ADF_CSR_WR(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + \
ADF_RING_CSR_RING_UBASE + (ring << 2), \
u_base); \
} while (0)
static inline uint64_t
read_base(struct resource *csr_base_addr, uint32_t bank, uint32_t ring)
{
uint32_t l_base, u_base;
uint64_t addr;
l_base = ADF_CSR_RD(csr_base_addr,
(ADF_RING_BUNDLE_SIZE * bank) +
ADF_RING_CSR_RING_LBASE + (ring << 2));
u_base = ADF_CSR_RD(csr_base_addr,
(ADF_RING_BUNDLE_SIZE * bank) +
ADF_RING_CSR_RING_UBASE + (ring << 2));
addr = (uint64_t)l_base & 0x00000000FFFFFFFFULL;
addr |= (uint64_t)u_base << 32 & 0xFFFFFFFF00000000ULL;
return addr;
}
#define READ_CSR_RING_BASE(csr_base_addr, bank, ring) \
read_base(csr_base_addr, bank, ring)
#define WRITE_CSR_RING_HEAD(csr_base_addr, bank, ring, value) \
ADF_CSR_WR(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + ADF_RING_CSR_RING_HEAD + \
(ring << 2), \
value)
#define WRITE_CSR_RING_TAIL(csr_base_addr, bank, ring, value) \
ADF_CSR_WR(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + ADF_RING_CSR_RING_TAIL + \
(ring << 2), \
value)
#define WRITE_CSR_INT_FLAG(csr_base_addr, bank, value) \
ADF_CSR_WR(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * (bank)) + ADF_RING_CSR_INT_FLAG, \
value)
#define WRITE_CSR_INT_SRCSEL(csr_base_addr, bank) \
do { \
ADF_CSR_WR(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + \
ADF_RING_CSR_INT_SRCSEL, \
ADF_BANK_INT_SRC_SEL_MASK_0); \
ADF_CSR_WR(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + \
ADF_RING_CSR_INT_SRCSEL_2, \
ADF_BANK_INT_SRC_SEL_MASK_X); \
} while (0)
#define WRITE_CSR_INT_COL_EN(csr_base_addr, bank, value) \
ADF_CSR_WR(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + ADF_RING_CSR_INT_COL_EN, \
value)
#define WRITE_CSR_INT_COL_CTL(csr_base_addr, bank, value) \
ADF_CSR_WR(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + ADF_RING_CSR_INT_COL_CTL, \
ADF_RING_CSR_INT_COL_CTL_ENABLE | value)
#define WRITE_CSR_INT_FLAG_AND_COL(csr_base_addr, bank, value) \
ADF_CSR_WR(csr_base_addr, \
(ADF_RING_BUNDLE_SIZE * bank) + \
ADF_RING_CSR_INT_FLAG_AND_COL, \
value)
#endif

58
sys/dev/qat/include/common/adf_transport_internal.h Normal file
View File

@ -0,0 +1,58 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef ADF_TRANSPORT_INTRN_H
#define ADF_TRANSPORT_INTRN_H
#include "adf_transport.h"
struct adf_etr_ring_debug_entry {
char ring_name[ADF_CFG_MAX_KEY_LEN_IN_BYTES];
struct sysctl_oid *debug;
};
struct adf_etr_ring_data {
void *base_addr;
atomic_t *inflights;
struct mtx lock; /* protects ring data struct */
adf_callback_fn callback;
struct adf_etr_bank_data *bank;
bus_addr_t dma_addr;
uint16_t head;
uint16_t tail;
uint8_t ring_number;
uint8_t ring_size;
uint8_t msg_size;
uint8_t reserved;
struct adf_etr_ring_debug_entry *ring_debug;
struct bus_dmamem dma_mem;
u32 csr_tail_offset;
u32 max_inflights;
};
struct adf_etr_bank_data {
struct adf_etr_ring_data *rings;
struct task resp_handler;
struct resource *csr_addr;
struct adf_accel_dev *accel_dev;
uint32_t irq_coalesc_timer;
uint16_t ring_mask;
uint16_t irq_mask;
struct mtx lock; /* protects bank data struct */
struct sysctl_oid *bank_debug_dir;
struct sysctl_oid *bank_debug_cfg;
uint32_t bank_number;
};
struct adf_etr_data {
struct adf_etr_bank_data *banks;
struct sysctl_oid *debug;
};
void adf_response_handler(uintptr_t bank_addr);
int adf_handle_response(struct adf_etr_ring_data *ring, u32 quota);
int adf_bank_debugfs_add(struct adf_etr_bank_data *bank);
void adf_bank_debugfs_rm(struct adf_etr_bank_data *bank);
int adf_ring_debugfs_add(struct adf_etr_ring_data *ring, const char *name);
void adf_ring_debugfs_rm(struct adf_etr_ring_data *ring);
#endif

53
sys/dev/qat/include/common/icp_qat_fw_loader_handle.h Normal file
View File

@ -0,0 +1,53 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef __ICP_QAT_FW_LOADER_HANDLE_H__
#define __ICP_QAT_FW_LOADER_HANDLE_H__
#include "icp_qat_uclo.h"
struct icp_qat_fw_loader_ae_data {
unsigned int state;
unsigned int ustore_size;
unsigned int free_addr;
unsigned int free_size;
unsigned int live_ctx_mask;
};
struct icp_qat_fw_loader_hal_handle {
struct icp_qat_fw_loader_ae_data aes[ICP_QAT_UCLO_MAX_AE];
unsigned int ae_mask;
unsigned int slice_mask;
unsigned int revision_id;
unsigned int ae_max_num;
unsigned int upc_mask;
unsigned int max_ustore;
};
struct icp_qat_fw_loader_handle {
struct icp_qat_fw_loader_hal_handle *hal_handle;
struct adf_accel_dev *accel_dev;
device_t pci_dev;
void *obj_handle;
void *sobj_handle;
void *mobj_handle;
bool fw_auth;
unsigned int cfg_ae_mask;
rman_res_t hal_sram_size;
struct resource *hal_sram_addr_v;
unsigned int hal_sram_offset;
struct resource *hal_misc_addr_v;
uintptr_t hal_cap_g_ctl_csr_addr_v;
uintptr_t hal_cap_ae_xfer_csr_addr_v;
uintptr_t hal_cap_ae_local_csr_addr_v;
uintptr_t hal_ep_csr_addr_v;
};
struct icp_firml_dram_desc {
struct bus_dmamem dram_mem;
struct resource *dram_base_addr;
void *dram_base_addr_v;
bus_addr_t dram_bus_addr;
u64 dram_size;
};
#endif

196
sys/dev/qat/include/common/icp_qat_hal.h Normal file
View File

@ -0,0 +1,196 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef __ICP_QAT_HAL_H
#define __ICP_QAT_HAL_H
#include "adf_accel_devices.h"
#include "icp_qat_fw_loader_handle.h"
enum hal_global_csr {
MISC_CONTROL = 0x04,
ICP_RESET = 0x0c,
ICP_GLOBAL_CLK_ENABLE = 0x50
};
enum { MISC_CONTROL_C4XXX = 0xAA0,
ICP_RESET_CPP0 = 0x938,
ICP_RESET_CPP1 = 0x93c,
ICP_GLOBAL_CLK_ENABLE_CPP0 = 0x964,
ICP_GLOBAL_CLK_ENABLE_CPP1 = 0x968 };
enum hal_ae_csr {
USTORE_ADDRESS = 0x000,
USTORE_DATA_LOWER = 0x004,
USTORE_DATA_UPPER = 0x008,
ALU_OUT = 0x010,
CTX_ARB_CNTL = 0x014,
CTX_ENABLES = 0x018,
CC_ENABLE = 0x01c,
CSR_CTX_POINTER = 0x020,
CTX_STS_INDIRECT = 0x040,
ACTIVE_CTX_STATUS = 0x044,
CTX_SIG_EVENTS_INDIRECT = 0x048,
CTX_SIG_EVENTS_ACTIVE = 0x04c,
CTX_WAKEUP_EVENTS_INDIRECT = 0x050,
LM_ADDR_0_INDIRECT = 0x060,
LM_ADDR_1_INDIRECT = 0x068,
LM_ADDR_2_INDIRECT = 0x0cc,
LM_ADDR_3_INDIRECT = 0x0d4,
INDIRECT_LM_ADDR_0_BYTE_INDEX = 0x0e0,
INDIRECT_LM_ADDR_1_BYTE_INDEX = 0x0e8,
INDIRECT_LM_ADDR_2_BYTE_INDEX = 0x10c,
INDIRECT_LM_ADDR_3_BYTE_INDEX = 0x114,
INDIRECT_T_INDEX = 0x0f8,
INDIRECT_T_INDEX_BYTE_INDEX = 0x0fc,
FUTURE_COUNT_SIGNAL_INDIRECT = 0x078,
TIMESTAMP_LOW = 0x0c0,
TIMESTAMP_HIGH = 0x0c4,
PROFILE_COUNT = 0x144,
SIGNATURE_ENABLE = 0x150,
AE_MISC_CONTROL = 0x160,
LOCAL_CSR_STATUS = 0x180,
};
enum fcu_csr {
FCU_CONTROL = 0x0,
FCU_STATUS = 0x4,
FCU_DRAM_ADDR_LO = 0xc,
FCU_DRAM_ADDR_HI = 0x10,
FCU_RAMBASE_ADDR_HI = 0x14,
FCU_RAMBASE_ADDR_LO = 0x18
};
enum fcu_csr_c4xxx {
FCU_CONTROL_C4XXX = 0x0,
FCU_STATUS_C4XXX = 0x4,
FCU_STATUS1_C4XXX = 0xc,
FCU_AE_LOADED_C4XXX = 0x10,
FCU_DRAM_ADDR_LO_C4XXX = 0x14,
FCU_DRAM_ADDR_HI_C4XXX = 0x18,
};
enum fcu_cmd {
FCU_CTRL_CMD_NOOP = 0,
FCU_CTRL_CMD_AUTH = 1,
FCU_CTRL_CMD_LOAD = 2,
FCU_CTRL_CMD_START = 3
};
enum fcu_sts {
FCU_STS_NO_STS = 0,
FCU_STS_VERI_DONE = 1,
FCU_STS_LOAD_DONE = 2,
FCU_STS_VERI_FAIL = 3,
FCU_STS_LOAD_FAIL = 4,
FCU_STS_BUSY = 5
};
#define UA_ECS (0x1 << 31)
#define ACS_ABO_BITPOS 31
#define ACS_ACNO 0x7
#define CE_ENABLE_BITPOS 0x8
#define CE_LMADDR_0_GLOBAL_BITPOS 16
#define CE_LMADDR_1_GLOBAL_BITPOS 17
#define CE_LMADDR_2_GLOBAL_BITPOS 22
#define CE_LMADDR_3_GLOBAL_BITPOS 23
#define CE_T_INDEX_GLOBAL_BITPOS 21
#define CE_NN_MODE_BITPOS 20
#define CE_REG_PAR_ERR_BITPOS 25
#define CE_BREAKPOINT_BITPOS 27
#define CE_CNTL_STORE_PARITY_ERROR_BITPOS 29
#define CE_INUSE_CONTEXTS_BITPOS 31
#define CE_NN_MODE (0x1 << CE_NN_MODE_BITPOS)
#define CE_INUSE_CONTEXTS (0x1 << CE_INUSE_CONTEXTS_BITPOS)
#define XCWE_VOLUNTARY (0x1)
#define LCS_STATUS (0x1)
#define MMC_SHARE_CS_BITPOS 2
#define GLOBAL_CSR 0xA00
#define FCU_CTRL_AE_POS 0x8
#define FCU_AUTH_STS_MASK 0x7
#define FCU_STS_DONE_POS 0x9
#define FCU_STS_AUTHFWLD_POS 0X8
#define FCU_LOADED_AE_POS 0x16
#define FW_AUTH_WAIT_PERIOD 10
#define FW_AUTH_MAX_RETRY 300
#define FCU_OFFSET 0x8c0
#define FCU_OFFSET_C4XXX 0x1000
#define MAX_CPP_NUM 2
#define AE_CPP_NUM 2
#define AES_PER_CPP 16
#define SLICES_PER_CPP 6
#define ICP_QAT_AE_OFFSET 0x20000
#define ICP_QAT_AE_OFFSET_C4XXX 0x40000
#define ICP_QAT_CAP_OFFSET (ICP_QAT_AE_OFFSET + 0x10000)
#define ICP_QAT_CAP_OFFSET_C4XXX 0x70000
#define LOCAL_TO_XFER_REG_OFFSET 0x800
#define ICP_QAT_EP_OFFSET 0x3a000
#define ICP_QAT_EP_OFFSET_C4XXX 0x60000
#define MEM_CFG_ERR_BIT 0x20
#define CAP_CSR_ADDR(csr) (csr + handle->hal_cap_g_ctl_csr_addr_v)
#define SET_CAP_CSR(handle, csr, val) \
ADF_CSR_WR(handle->hal_misc_addr_v, CAP_CSR_ADDR(csr), val)
#define GET_CAP_CSR(handle, csr) \
ADF_CSR_RD(handle->hal_misc_addr_v, CAP_CSR_ADDR(csr))
#define SET_GLB_CSR(handle, csr, val) \
({ \
typeof(handle) handle_ = (handle); \
typeof(csr) csr_ = (csr); \
typeof(val) val_ = (val); \
(IS_QAT_GEN3(pci_get_device(GET_DEV(handle_->accel_dev)))) ? \
SET_CAP_CSR(handle_, (csr_), (val_)) : \
SET_CAP_CSR(handle_, csr_ + GLOBAL_CSR, val_); \
})
#define GET_GLB_CSR(handle, csr) \
({ \
typeof(handle) handle_ = (handle); \
typeof(csr) csr_ = (csr); \
(IS_QAT_GEN3(pci_get_device(GET_DEV(handle_->accel_dev)))) ? \
(GET_CAP_CSR(handle_, (csr_))) : \
(GET_CAP_CSR(handle_, (GLOBAL_CSR + (csr_)))); \
})
#define SET_FCU_CSR(handle, csr, val) \
({ \
typeof(handle) handle_ = (handle); \
typeof(csr) csr_ = (csr); \
typeof(val) val_ = (val); \
(IS_QAT_GEN3(pci_get_device(GET_DEV(handle_->accel_dev)))) ? \
SET_CAP_CSR(handle_, \
((csr_) + FCU_OFFSET_C4XXX), \
(val_)) : \
SET_CAP_CSR(handle_, ((csr_) + FCU_OFFSET), (val_)); \
})
#define GET_FCU_CSR(handle, csr) \
({ \
typeof(handle) handle_ = (handle); \
typeof(csr) csr_ = (csr); \
(IS_QAT_GEN3(pci_get_device(GET_DEV(handle_->accel_dev)))) ? \
GET_CAP_CSR(handle_, (FCU_OFFSET_C4XXX + (csr_))) : \
GET_CAP_CSR(handle_, (FCU_OFFSET + (csr_))); \
})
#define AE_CSR(handle, ae) \
((handle)->hal_cap_ae_local_csr_addr_v + ((ae) << 12))
#define AE_CSR_ADDR(handle, ae, csr) (AE_CSR(handle, ae) + (0x3ff & (csr)))
#define SET_AE_CSR(handle, ae, csr, val) \
ADF_CSR_WR(handle->hal_misc_addr_v, AE_CSR_ADDR(handle, ae, csr), val)
#define GET_AE_CSR(handle, ae, csr) \
ADF_CSR_RD(handle->hal_misc_addr_v, AE_CSR_ADDR(handle, ae, csr))
#define AE_XFER(handle, ae) \
((handle)->hal_cap_ae_xfer_csr_addr_v + ((ae) << 12))
#define AE_XFER_ADDR(handle, ae, reg) \
(AE_XFER(handle, ae) + (((reg)&0xff) << 2))
#define SET_AE_XFER(handle, ae, reg, val) \
ADF_CSR_WR(handle->hal_misc_addr_v, AE_XFER_ADDR(handle, ae, reg), val)
#define SRAM_WRITE(handle, addr, val) \
ADF_CSR_WR((handle)->hal_sram_addr_v, addr, val)
#define GET_CSR_OFFSET(device_id, cap_offset_, ae_offset_, ep_offset_) \
({ \
int gen3 = IS_QAT_GEN3(device_id); \
cap_offset_ = \
(gen3 ? ICP_QAT_CAP_OFFSET_C4XXX : ICP_QAT_CAP_OFFSET); \
ae_offset_ = \
(gen3 ? ICP_QAT_AE_OFFSET_C4XXX : ICP_QAT_AE_OFFSET); \
ep_offset_ = \
(gen3 ? ICP_QAT_EP_OFFSET_C4XXX : ICP_QAT_EP_OFFSET); \
})
#endif

558
sys/dev/qat/include/common/icp_qat_uclo.h Normal file
View File

@ -0,0 +1,558 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef __ICP_QAT_UCLO_H__
#define __ICP_QAT_UCLO_H__
#define ICP_QAT_AC_895XCC_DEV_TYPE 0x00400000
#define ICP_QAT_AC_C62X_DEV_TYPE 0x01000000
#define ICP_QAT_AC_C3XXX_DEV_TYPE 0x02000000
#define ICP_QAT_AC_200XX_DEV_TYPE 0x02000000
#define ICP_QAT_AC_C4XXX_DEV_TYPE 0x04000000
#define ICP_QAT_UCLO_MAX_AE 32
#define ICP_QAT_UCLO_MAX_CTX 8
#define ICP_QAT_UCLO_MAX_CPPNUM 2
#define ICP_QAT_UCLO_MAX_UIMAGE (ICP_QAT_UCLO_MAX_AE * ICP_QAT_UCLO_MAX_CTX)
#define ICP_QAT_UCLO_MAX_USTORE 0x4000
#define ICP_QAT_UCLO_MAX_XFER_REG 128
#define ICP_QAT_UCLO_MAX_GPR_REG 128
#define ICP_QAT_UCLO_MAX_LMEM_REG 1024
#define ICP_QAT_UCLO_AE_ALL_CTX 0xff
#define ICP_QAT_UOF_OBJID_LEN 8
#define ICP_QAT_UOF_FID 0xc6c2
#define ICP_QAT_UOF_MAJVER 0x4
#define ICP_QAT_UOF_MINVER 0x11
#define ICP_QAT_UOF_OBJS "UOF_OBJS"
#define ICP_QAT_UOF_STRT "UOF_STRT"
#define ICP_QAT_UOF_IMAG "UOF_IMAG"
#define ICP_QAT_UOF_IMEM "UOF_IMEM"
#define ICP_QAT_UOF_LOCAL_SCOPE 1
#define ICP_QAT_UOF_INIT_EXPR 0
#define ICP_QAT_UOF_INIT_REG 1
#define ICP_QAT_UOF_INIT_REG_CTX 2
#define ICP_QAT_UOF_INIT_EXPR_ENDIAN_SWAP 3
#define ICP_QAT_SUOF_OBJ_ID_LEN 8
#define ICP_QAT_SUOF_FID 0x53554f46
#define ICP_QAT_SUOF_MAJVER 0x0
#define ICP_QAT_SUOF_MINVER 0x1
#define ICP_QAT_SUOF_OBJ_NAME_LEN 128
#define ICP_QAT_MOF_OBJ_ID_LEN 8
#define ICP_QAT_MOF_OBJ_CHUNKID_LEN 8
#define ICP_QAT_MOF_FID 0x00666f6d
#define ICP_QAT_MOF_MAJVER 0x0
#define ICP_QAT_MOF_MINVER 0x1
#define ICP_QAT_MOF_SYM_OBJS "SYM_OBJS"
#define ICP_QAT_SUOF_OBJS "SUF_OBJS"
#define ICP_QAT_SUOF_IMAG "SUF_IMAG"
#define ICP_QAT_SIMG_AE_INIT_SEQ_LEN (50 * sizeof(unsigned long long))
#define ICP_QAT_SIMG_AE_INSTS_LEN (0x4000 * sizeof(unsigned long long))
#define ICP_QAT_CSS_FWSK_MODULUS_LEN 256
#define ICP_QAT_CSS_FWSK_EXPONENT_LEN 4
#define ICP_QAT_CSS_FWSK_PAD_LEN 252
#define ICP_QAT_CSS_FWSK_PUB_LEN \
(ICP_QAT_CSS_FWSK_MODULUS_LEN + ICP_QAT_CSS_FWSK_EXPONENT_LEN + \
ICP_QAT_CSS_FWSK_PAD_LEN)
#define ICP_QAT_CSS_SIGNATURE_LEN 256
#define ICP_QAT_CSS_AE_IMG_LEN \
(sizeof(struct icp_qat_simg_ae_mode) + ICP_QAT_SIMG_AE_INIT_SEQ_LEN + \
ICP_QAT_SIMG_AE_INSTS_LEN)
#define ICP_QAT_CSS_AE_SIMG_LEN \
(sizeof(struct icp_qat_css_hdr) + ICP_QAT_CSS_FWSK_PUB_LEN + \
ICP_QAT_CSS_SIGNATURE_LEN + ICP_QAT_CSS_AE_IMG_LEN)
#define ICP_QAT_AE_IMG_OFFSET \
(sizeof(struct icp_qat_css_hdr) + ICP_QAT_CSS_FWSK_MODULUS_LEN + \
ICP_QAT_CSS_FWSK_EXPONENT_LEN + ICP_QAT_CSS_SIGNATURE_LEN)
#define ICP_QAT_CSS_MAX_IMAGE_LEN 0x40000
#define ICP_QAT_CTX_MODE(ae_mode) ((ae_mode)&0xf)
#define ICP_QAT_NN_MODE(ae_mode) (((ae_mode) >> 0x4) & 0xf)
#define ICP_QAT_SHARED_USTORE_MODE(ae_mode) (((ae_mode) >> 0xb) & 0x1)
#define RELOADABLE_CTX_SHARED_MODE(ae_mode) (((ae_mode) >> 0xc) & 0x1)
#define ICP_QAT_LOC_MEM0_MODE(ae_mode) (((ae_mode) >> 0x8) & 0x1)
#define ICP_QAT_LOC_MEM1_MODE(ae_mode) (((ae_mode) >> 0x9) & 0x1)
#define ICP_QAT_LOC_MEM2_MODE(ae_mode) (((ae_mode) >> 0x6) & 0x1)
#define ICP_QAT_LOC_MEM3_MODE(ae_mode) (((ae_mode) >> 0x7) & 0x1)
#define ICP_QAT_LOC_TINDEX_MODE(ae_mode) (((ae_mode) >> 0xe) & 0x1)
enum icp_qat_uof_mem_region {
ICP_QAT_UOF_SRAM_REGION = 0x0,
ICP_QAT_UOF_LMEM_REGION = 0x3,
ICP_QAT_UOF_UMEM_REGION = 0x5
};
enum icp_qat_uof_regtype {
ICP_NO_DEST = 0,
ICP_GPA_REL = 1,
ICP_GPA_ABS = 2,
ICP_GPB_REL = 3,
ICP_GPB_ABS = 4,
ICP_SR_REL = 5,
ICP_SR_RD_REL = 6,
ICP_SR_WR_REL = 7,
ICP_SR_ABS = 8,
ICP_SR_RD_ABS = 9,
ICP_SR_WR_ABS = 10,
ICP_DR_REL = 19,
ICP_DR_RD_REL = 20,
ICP_DR_WR_REL = 21,
ICP_DR_ABS = 22,
ICP_DR_RD_ABS = 23,
ICP_DR_WR_ABS = 24,
ICP_LMEM = 26,
ICP_LMEM0 = 27,
ICP_LMEM1 = 28,
ICP_NEIGH_REL = 31,
ICP_LMEM2 = 61,
ICP_LMEM3 = 62,
};
enum icp_qat_css_fwtype { CSS_AE_FIRMWARE = 0, CSS_MMP_FIRMWARE = 1 };
struct icp_qat_uclo_page {
struct icp_qat_uclo_encap_page *encap_page;
struct icp_qat_uclo_region *region;
unsigned int flags;
};
struct icp_qat_uclo_region {
struct icp_qat_uclo_page *loaded;
struct icp_qat_uclo_page *page;
};
struct icp_qat_uclo_aeslice {
struct icp_qat_uclo_region *region;
struct icp_qat_uclo_page *page;
struct icp_qat_uclo_page *cur_page[ICP_QAT_UCLO_MAX_CTX];
struct icp_qat_uclo_encapme *encap_image;
unsigned int ctx_mask_assigned;
unsigned int new_uaddr[ICP_QAT_UCLO_MAX_CTX];
};
struct icp_qat_uclo_aedata {
unsigned int slice_num;
unsigned int eff_ustore_size;
struct icp_qat_uclo_aeslice ae_slices[ICP_QAT_UCLO_MAX_CTX];
unsigned int shareable_ustore;
};
struct icp_qat_uof_encap_obj {
char *beg_uof;
struct icp_qat_uof_objhdr *obj_hdr;
struct icp_qat_uof_chunkhdr *chunk_hdr;
struct icp_qat_uof_varmem_seg *var_mem_seg;
};
struct icp_qat_uclo_encap_uwblock {
unsigned int start_addr;
unsigned int words_num;
uint64_t micro_words;
};
struct icp_qat_uclo_encap_page {
unsigned int def_page;
unsigned int page_region;
unsigned int beg_addr_v;
unsigned int beg_addr_p;
unsigned int micro_words_num;
unsigned int uwblock_num;
struct icp_qat_uclo_encap_uwblock *uwblock;
};
struct icp_qat_uclo_encapme {
struct icp_qat_uof_image *img_ptr;
struct icp_qat_uclo_encap_page *page;
unsigned int ae_reg_num;
struct icp_qat_uof_ae_reg *ae_reg;
unsigned int init_regsym_num;
struct icp_qat_uof_init_regsym *init_regsym;
unsigned int sbreak_num;
struct icp_qat_uof_sbreak *sbreak;
unsigned int uwords_num;
};
struct icp_qat_uclo_init_mem_table {
unsigned int entry_num;
struct icp_qat_uof_initmem *init_mem;
};
struct icp_qat_uclo_objhdr {
char *file_buff;
unsigned int checksum;
unsigned int size;
};
struct icp_qat_uof_strtable {
unsigned int table_len;
unsigned int reserved;
uint64_t strings;
};
struct icp_qat_uclo_objhandle {
unsigned int prod_type;
unsigned int prod_rev;
struct icp_qat_uclo_objhdr *obj_hdr;
struct icp_qat_uof_encap_obj encap_uof_obj;
struct icp_qat_uof_strtable str_table;
struct icp_qat_uclo_encapme ae_uimage[ICP_QAT_UCLO_MAX_UIMAGE];
struct icp_qat_uclo_aedata ae_data[ICP_QAT_UCLO_MAX_AE];
struct icp_qat_uclo_init_mem_table init_mem_tab;
struct icp_qat_uof_batch_init *lm_init_tab[ICP_QAT_UCLO_MAX_AE];
struct icp_qat_uof_batch_init *umem_init_tab[ICP_QAT_UCLO_MAX_AE];
int uimage_num;
int uword_in_bytes;
int global_inited;
unsigned int ae_num;
unsigned int ustore_phy_size;
void *obj_buf;
uint64_t *uword_buf;
};
struct icp_qat_uof_uword_block {
unsigned int start_addr;
unsigned int words_num;
unsigned int uword_offset;
unsigned int reserved;
};
struct icp_qat_uof_filehdr {
unsigned short file_id;
unsigned short reserved1;
char min_ver;
char maj_ver;
unsigned short reserved2;
unsigned short max_chunks;
unsigned short num_chunks;
};
struct icp_qat_uof_filechunkhdr {
char chunk_id[ICP_QAT_UOF_OBJID_LEN];
unsigned int checksum;
unsigned int offset;
unsigned int size;
};
struct icp_qat_uof_objhdr {
unsigned int ac_dev_type;
unsigned short min_cpu_ver;
unsigned short max_cpu_ver;
short max_chunks;
short num_chunks;
unsigned int reserved1;
unsigned int reserved2;
};
struct icp_qat_uof_chunkhdr {
char chunk_id[ICP_QAT_UOF_OBJID_LEN];
unsigned int offset;
unsigned int size;
};
struct icp_qat_uof_memvar_attr {
unsigned int offset_in_byte;
unsigned int value;
};
struct icp_qat_uof_initmem {
unsigned int sym_name;
char region;
char scope;
unsigned short reserved1;
unsigned int addr;
unsigned int num_in_bytes;
unsigned int val_attr_num;
};
struct icp_qat_uof_init_regsym {
unsigned int sym_name;
char init_type;
char value_type;
char reg_type;
unsigned char ctx;
unsigned int reg_addr;
unsigned int value;
};
struct icp_qat_uof_varmem_seg {
unsigned int sram_base;
unsigned int sram_size;
unsigned int sram_alignment;
unsigned int sdram_base;
unsigned int sdram_size;
unsigned int sdram_alignment;
unsigned int sdram1_base;
unsigned int sdram1_size;
unsigned int sdram1_alignment;
unsigned int scratch_base;
unsigned int scratch_size;
unsigned int scratch_alignment;
};
struct icp_qat_uof_gtid {
char tool_id[ICP_QAT_UOF_OBJID_LEN];
int tool_ver;
unsigned int reserved1;
unsigned int reserved2;
};
struct icp_qat_uof_sbreak {
unsigned int page_num;
unsigned int virt_uaddr;
unsigned char sbreak_type;
unsigned char reg_type;
unsigned short reserved1;
unsigned int addr_offset;
unsigned int reg_addr;
};
struct icp_qat_uof_code_page {
unsigned int page_region;
unsigned int page_num;
unsigned char def_page;
unsigned char reserved2;
unsigned short reserved1;
unsigned int beg_addr_v;
unsigned int beg_addr_p;
unsigned int neigh_reg_tab_offset;
unsigned int uc_var_tab_offset;
unsigned int imp_var_tab_offset;
unsigned int imp_expr_tab_offset;
unsigned int code_area_offset;
};
struct icp_qat_uof_image {
unsigned int img_name;
unsigned int ae_assigned;
unsigned int ctx_assigned;
unsigned int ac_dev_type;
unsigned int entry_address;
unsigned int fill_pattern[2];
unsigned int reloadable_size;
unsigned char sensitivity;
unsigned char reserved;
unsigned short ae_mode;
unsigned short max_ver;
unsigned short min_ver;
unsigned short image_attrib;
unsigned short reserved2;
unsigned short page_region_num;
unsigned short numpages;
unsigned int reg_tab_offset;
unsigned int init_reg_sym_tab;
unsigned int sbreak_tab;
unsigned int app_metadata;
};
struct icp_qat_uof_objtable {
unsigned int entry_num;
};
struct icp_qat_uof_ae_reg {
unsigned int name;
unsigned int vis_name;
unsigned short type;
unsigned short addr;
unsigned short access_mode;
unsigned char visible;
unsigned char reserved1;
unsigned short ref_count;
unsigned short reserved2;
unsigned int xo_id;
};
struct icp_qat_uof_code_area {
unsigned int micro_words_num;
unsigned int uword_block_tab;
};
struct icp_qat_uof_batch_init {
unsigned int ae;
unsigned int addr;
unsigned int *value;
unsigned int size;
struct icp_qat_uof_batch_init *next;
};
struct icp_qat_suof_img_hdr {
const char *simg_buf;
unsigned long simg_len;
const char *css_header;
const char *css_key;
const char *css_signature;
const char *css_simg;
unsigned long simg_size;
unsigned int ae_num;
unsigned int ae_mask;
unsigned int fw_type;
unsigned long simg_name;
unsigned long appmeta_data;
};
struct icp_qat_suof_img_tbl {
unsigned int num_simgs;
struct icp_qat_suof_img_hdr *simg_hdr;
};
struct icp_qat_suof_handle {
unsigned int file_id;
unsigned int check_sum;
char min_ver;
char maj_ver;
char fw_type;
const char *suof_buf;
unsigned int suof_size;
char *sym_str;
unsigned int sym_size;
struct icp_qat_suof_img_tbl img_table;
};
struct icp_qat_fw_auth_desc {
unsigned int img_len;
unsigned int ae_mask;
unsigned int css_hdr_high;
unsigned int css_hdr_low;
unsigned int img_high;
unsigned int img_low;
unsigned int signature_high;
unsigned int signature_low;
unsigned int fwsk_pub_high;
unsigned int fwsk_pub_low;
unsigned int img_ae_mode_data_high;
unsigned int img_ae_mode_data_low;
unsigned int img_ae_init_data_high;
unsigned int img_ae_init_data_low;
unsigned int img_ae_insts_high;
unsigned int img_ae_insts_low;
};
struct icp_qat_auth_chunk {
struct icp_qat_fw_auth_desc fw_auth_desc;
u64 chunk_size;
u64 chunk_bus_addr;
};
struct icp_qat_css_hdr {
unsigned int module_type;
unsigned int header_len;
unsigned int header_ver;
unsigned int module_id;
unsigned int module_vendor;
unsigned int date;
unsigned int size;
unsigned int key_size;
unsigned int module_size;
unsigned int exponent_size;
unsigned int fw_type;
unsigned int reserved[21];
};
struct icp_qat_simg_ae_mode {
unsigned int file_id;
unsigned short maj_ver;
unsigned short min_ver;
unsigned int dev_type;
unsigned short devmax_ver;
unsigned short devmin_ver;
unsigned int ae_mask;
unsigned int ctx_enables;
char fw_type;
char ctx_mode;
char nn_mode;
char lm0_mode;
char lm1_mode;
char scs_mode;
char lm2_mode;
char lm3_mode;
char tindex_mode;
unsigned char reserved[7];
char simg_name[256];
char appmeta_data[256];
};
struct icp_qat_suof_filehdr {
unsigned int file_id;
unsigned int check_sum;
char min_ver;
char maj_ver;
char fw_type;
char reserved;
unsigned short max_chunks;
unsigned short num_chunks;
};
struct icp_qat_suof_chunk_hdr {
char chunk_id[ICP_QAT_SUOF_OBJ_ID_LEN];
u64 offset;
u64 size;
};
struct icp_qat_suof_strtable {
unsigned int tab_length;
unsigned int strings;
};
struct icp_qat_suof_objhdr {
unsigned int img_length;
unsigned int reserved;
};
struct icp_qat_mof_file_hdr {
unsigned int file_id;
unsigned int checksum;
char min_ver;
char maj_ver;
unsigned short reserved;
unsigned short max_chunks;
unsigned short num_chunks;
};
struct icp_qat_mof_chunkhdr {
char chunk_id[ICP_QAT_MOF_OBJ_ID_LEN];
u64 offset;
u64 size;
};
struct icp_qat_mof_str_table {
unsigned int tab_len;
unsigned int strings;
};
struct icp_qat_mof_obj_hdr {
unsigned short max_chunks;
unsigned short num_chunks;
unsigned int reserved;
};
struct icp_qat_mof_obj_chunkhdr {
char chunk_id[ICP_QAT_MOF_OBJ_CHUNKID_LEN];
u64 offset;
u64 size;
unsigned int name;
unsigned int reserved;
};
struct icp_qat_mof_objhdr {
char *obj_name;
const char *obj_buf;
unsigned int obj_size;
};
struct icp_qat_mof_table {
unsigned int num_objs;
struct icp_qat_mof_objhdr *obj_hdr;
};
struct icp_qat_mof_handle {
unsigned int file_id;
unsigned int checksum;
char min_ver;
char maj_ver;
const char *mof_buf;
u32 mof_size;
char *sym_str;
unsigned int sym_size;
const char *uobjs_hdr;
const char *sobjs_hdr;
struct icp_qat_mof_table obj_table;
};
#endif

156
sys/dev/qat/include/common/qat_freebsd.h Normal file
View File

@ -0,0 +1,156 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef QAT_FREEBSD_H_
#define QAT_FREEBSD_H_
#include <sys/param.h>
#include <sys/module.h>
#include <sys/bus.h>
#include <sys/param.h>
#include <sys/malloc.h>
#include <sys/firmware.h>
#include <sys/rman.h>
#include <sys/types.h>
#include <sys/ctype.h>
#include <sys/ioccom.h>
#include <sys/param.h>
#include <sys/lock.h>
#include <linux/device.h>
#include <linux/dma-mapping.h>
#include <linux/completion.h>
#include <linux/list.h>
#include <machine/bus.h>
#include <machine/bus_dma.h>
#include <sys/firmware.h>
#include <asm/uaccess.h>
#include <linux/math64.h>
#include <linux/spinlock.h>
#define PCI_VENDOR_ID_INTEL 0x8086
#if !defined(__bool_true_false_are_defined)
#define __bool_true_false_are_defined 1
#define false 0
#define true 1
#if __STDC_VERSION__ < 199901L && __GNUC__ < 3 && !defined(__INTEL_COMPILER)
typedef int _Bool;
#endif
typedef _Bool bool;
#endif /* !__bool_true_false_are_defined && !__cplusplus */
#if __STDC_VERSION__ < 199901L && __GNUC__ < 3 && !defined(__INTEL_COMPILER)
typedef int _Bool;
#endif
#define pause_ms(wmesg, ms) pause_sbt(wmesg, (ms)*SBT_1MS, 0, C_HARDCLOCK)
/* Function sets the MaxPayload size of a PCI device. */
int pci_set_max_payload(device_t dev, int payload_size);
device_t pci_find_pf(device_t vf);
MALLOC_DECLARE(M_QAT);
struct msix_entry {
struct resource *irq;
void *cookie;
};
struct pci_device_id {
uint16_t vendor;
uint16_t device;
};
struct bus_dmamem {
bus_dma_tag_t dma_tag;
bus_dmamap_t dma_map;
void *dma_vaddr;
bus_addr_t dma_baddr;
};
/*
* Allocate a mapping. On success, zero is returned and the 'dma_vaddr'
* and 'dma_baddr' fields are populated with the virtual and bus addresses,
* respectively, of the mapping.
*/
int bus_dma_mem_create(struct bus_dmamem *mem,
bus_dma_tag_t parent,
bus_size_t alignment,
bus_addr_t lowaddr,
bus_size_t len,
int flags);
/*
* Release a mapping created by bus_dma_mem_create().
*/
void bus_dma_mem_free(struct bus_dmamem *mem);
#define list_for_each_prev_safe(p, n, h) \
for (p = (h)->prev, n = (p)->prev; p != (h); p = n, n = (p)->prev)
static inline int
compat_strtoul(const char *cp, unsigned int base, unsigned long *res)
{
char *end;
*res = strtoul(cp, &end, base);
/* skip newline character, if any */
if (*end == '\n')
end++;
if (*cp == 0 || *end != 0)
return (-EINVAL);
return (0);
}
static inline int
compat_strtouint(const char *cp, unsigned int base, unsigned int *res)
{
char *end;
unsigned long temp;
*res = temp = strtoul(cp, &end, base);
/* skip newline character, if any */
if (*end == '\n')
end++;
if (*cp == 0 || *end != 0)
return (-EINVAL);
if (temp != (unsigned int)temp)
return (-ERANGE);
return (0);
}
static inline int
compat_strtou8(const char *cp, unsigned int base, unsigned char *res)
{
char *end;
unsigned long temp;
*res = temp = strtoul(cp, &end, base);
/* skip newline character, if any */
if (*end == '\n')
end++;
if (*cp == 0 || *end != 0)
return -EINVAL;
if (temp != (unsigned char)temp)
return -ERANGE;
return 0;
}
#if __FreeBSD_version >= 1300500
#undef dev_to_node
static inline int
dev_to_node(device_t dev)
{
int numa_domain;
if (!dev || bus_get_domain(dev, &numa_domain) != 0)
return (-1);
else
return (numa_domain);
}
#endif
#endif

33
sys/dev/qat/include/common/sal_statistics_strings.h Normal file
View File

@ -0,0 +1,33 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef SAL_STATISTICS_STRINGS_H
#define SAL_STATISTICS_STRINGS_H
/*
* Config values names for statistics
*/
#define SAL_STATS_CFG_ENABLED "statsGeneral"
/**< Config value name for enabling/disabling statistics */
#define SAL_STATS_CFG_DC "statsDc"
/**< Config value name for enabling/disabling Compression statistics */
#define SAL_STATS_CFG_DH "statsDh"
/**< Config value name for enabling/disabling Diffie-Helman statistics */
#define SAL_STATS_CFG_DRBG "statsDrbg"
/**< Config value name for enabling/disabling DRBG statistics */
#define SAL_STATS_CFG_DSA "statsDsa"
/**< Config value name for enabling/disabling DSA statistics */
#define SAL_STATS_CFG_ECC "statsEcc"
/**< Config value name for enabling/disabling ECC statistics */
#define SAL_STATS_CFG_KEYGEN "statsKeyGen"
/**< Config value name for enabling/disabling Key Gen statistics */
#define SAL_STATS_CFG_LN "statsLn"
/**< Config value name for enabling/disabling Large Number statistics */
#define SAL_STATS_CFG_PRIME "statsPrime"
/**< Config value name for enabling/disabling Prime statistics */
#define SAL_STATS_CFG_RSA "statsRsa"
/**< Config value name for enabling/disabling RSA statistics */
#define SAL_STATS_CFG_SYM "statsSym"
/**< Config value name for enabling/disabling Symmetric Crypto statistics */
#endif

292
sys/dev/qat/include/icp_qat_fw.h Normal file
View File

@ -0,0 +1,292 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef _ICP_QAT_FW_H_
#define _ICP_QAT_FW_H_
#include <sys/types.h>
#include "icp_qat_hw.h"
#define QAT_FIELD_SET(flags, val, bitpos, mask) \
{ \
(flags) = (((flags) & (~((mask) << (bitpos)))) | \
(((val) & (mask)) << (bitpos))); \
}
#define QAT_FIELD_GET(flags, bitpos, mask) (((flags) >> (bitpos)) & (mask))
#define ICP_QAT_FW_REQ_DEFAULT_SZ 128
#define ICP_QAT_FW_RESP_DEFAULT_SZ 32
#define ICP_QAT_FW_COMN_ONE_BYTE_SHIFT 8
#define ICP_QAT_FW_COMN_SINGLE_BYTE_MASK 0xFF
#define ICP_QAT_FW_NUM_LONGWORDS_1 1
#define ICP_QAT_FW_NUM_LONGWORDS_2 2
#define ICP_QAT_FW_NUM_LONGWORDS_3 3
#define ICP_QAT_FW_NUM_LONGWORDS_4 4
#define ICP_QAT_FW_NUM_LONGWORDS_5 5
#define ICP_QAT_FW_NUM_LONGWORDS_6 6
#define ICP_QAT_FW_NUM_LONGWORDS_7 7
#define ICP_QAT_FW_NUM_LONGWORDS_10 10
#define ICP_QAT_FW_NUM_LONGWORDS_13 13
#define ICP_QAT_FW_NULL_REQ_SERV_ID 1
enum icp_qat_fw_comn_resp_serv_id {
ICP_QAT_FW_COMN_RESP_SERV_NULL,
ICP_QAT_FW_COMN_RESP_SERV_CPM_FW,
ICP_QAT_FW_COMN_RESP_SERV_DELIMITER
};
enum icp_qat_fw_comn_request_id {
ICP_QAT_FW_COMN_REQ_NULL = 0,
ICP_QAT_FW_COMN_REQ_CPM_FW_PKE = 3,
ICP_QAT_FW_COMN_REQ_CPM_FW_LA = 4,
ICP_QAT_FW_COMN_REQ_CPM_FW_DMA = 7,
ICP_QAT_FW_COMN_REQ_CPM_FW_COMP = 9,
ICP_QAT_FW_COMN_REQ_DELIMITER
};
struct icp_qat_fw_comn_req_hdr_cd_pars {
union {
struct {
uint64_t content_desc_addr;
uint16_t content_desc_resrvd1;
uint8_t content_desc_params_sz;
uint8_t content_desc_hdr_resrvd2;
uint32_t content_desc_resrvd3;
} s;
struct {
uint32_t serv_specif_fields[4];
} s1;
} u;
};
struct icp_qat_fw_comn_req_mid {
uint64_t opaque_data;
uint64_t src_data_addr;
uint64_t dest_data_addr;
uint32_t src_length;
uint32_t dst_length;
};
struct icp_qat_fw_comn_req_cd_ctrl {
uint32_t content_desc_ctrl_lw[ICP_QAT_FW_NUM_LONGWORDS_5];
};
struct icp_qat_fw_comn_req_hdr {
uint8_t resrvd1;
uint8_t service_cmd_id;
uint8_t service_type;
uint8_t hdr_flags;
uint16_t serv_specif_flags;
uint16_t comn_req_flags;
};
struct icp_qat_fw_comn_req_rqpars {
uint32_t serv_specif_rqpars_lw[ICP_QAT_FW_NUM_LONGWORDS_13];
};
struct icp_qat_fw_comn_req {
struct icp_qat_fw_comn_req_hdr comn_hdr;
struct icp_qat_fw_comn_req_hdr_cd_pars cd_pars;
struct icp_qat_fw_comn_req_mid comn_mid;
struct icp_qat_fw_comn_req_rqpars serv_specif_rqpars;
struct icp_qat_fw_comn_req_cd_ctrl cd_ctrl;
};
struct icp_qat_fw_comn_error {
uint8_t xlat_err_code;
uint8_t cmp_err_code;
};
struct icp_qat_fw_comn_resp_hdr {
uint8_t resrvd1;
uint8_t service_id;
uint8_t response_type;
uint8_t hdr_flags;
struct icp_qat_fw_comn_error comn_error;
uint8_t comn_status;
uint8_t cmd_id;
};
struct icp_qat_fw_comn_resp {
struct icp_qat_fw_comn_resp_hdr comn_hdr;
uint64_t opaque_data;
uint32_t resrvd[ICP_QAT_FW_NUM_LONGWORDS_4];
};
#define ICP_QAT_FW_COMN_REQ_FLAG_SET 1
#define ICP_QAT_FW_COMN_REQ_FLAG_CLR 0
#define ICP_QAT_FW_COMN_VALID_FLAG_BITPOS 7
#define ICP_QAT_FW_COMN_VALID_FLAG_MASK 0x1
#define ICP_QAT_FW_COMN_HDR_RESRVD_FLD_MASK 0x7F
#define ICP_QAT_FW_COMN_OV_SRV_TYPE_GET(icp_qat_fw_comn_req_hdr_t) \
icp_qat_fw_comn_req_hdr_t.service_type
#define ICP_QAT_FW_COMN_OV_SRV_TYPE_SET(icp_qat_fw_comn_req_hdr_t, val) \
icp_qat_fw_comn_req_hdr_t.service_type = val
#define ICP_QAT_FW_COMN_OV_SRV_CMD_ID_GET(icp_qat_fw_comn_req_hdr_t) \
icp_qat_fw_comn_req_hdr_t.service_cmd_id
#define ICP_QAT_FW_COMN_OV_SRV_CMD_ID_SET(icp_qat_fw_comn_req_hdr_t, val) \
icp_qat_fw_comn_req_hdr_t.service_cmd_id = val
#define ICP_QAT_FW_COMN_HDR_VALID_FLAG_GET(hdr_t) \
ICP_QAT_FW_COMN_VALID_FLAG_GET(hdr_t.hdr_flags)
#define ICP_QAT_FW_COMN_HDR_VALID_FLAG_SET(hdr_t, val) \
ICP_QAT_FW_COMN_VALID_FLAG_SET(hdr_t, val)
#define ICP_QAT_FW_COMN_VALID_FLAG_GET(hdr_flags) \
QAT_FIELD_GET(hdr_flags, \
ICP_QAT_FW_COMN_VALID_FLAG_BITPOS, \
ICP_QAT_FW_COMN_VALID_FLAG_MASK)
#define ICP_QAT_FW_COMN_HDR_RESRVD_FLD_GET(hdr_flags) \
(hdr_flags & ICP_QAT_FW_COMN_HDR_RESRVD_FLD_MASK)
#define ICP_QAT_FW_COMN_VALID_FLAG_SET(hdr_t, val) \
QAT_FIELD_SET((hdr_t.hdr_flags), \
(val), \
ICP_QAT_FW_COMN_VALID_FLAG_BITPOS, \
ICP_QAT_FW_COMN_VALID_FLAG_MASK)
#define ICP_QAT_FW_COMN_HDR_FLAGS_BUILD(valid) \
(((valid)&ICP_QAT_FW_COMN_VALID_FLAG_MASK) \
<< ICP_QAT_FW_COMN_VALID_FLAG_BITPOS)
#define QAT_COMN_PTR_TYPE_BITPOS 0
#define QAT_COMN_PTR_TYPE_MASK 0x1
#define QAT_COMN_CD_FLD_TYPE_BITPOS 1
#define QAT_COMN_CD_FLD_TYPE_MASK 0x1
#define QAT_COMN_PTR_TYPE_FLAT 0x0
#define QAT_COMN_PTR_TYPE_SGL 0x1
#define QAT_COMN_CD_FLD_TYPE_64BIT_ADR 0x0
#define QAT_COMN_CD_FLD_TYPE_16BYTE_DATA 0x1
#define ICP_QAT_FW_COMN_FLAGS_BUILD(cdt, ptr) \
((((cdt)&QAT_COMN_CD_FLD_TYPE_MASK) << QAT_COMN_CD_FLD_TYPE_BITPOS) | \
(((ptr)&QAT_COMN_PTR_TYPE_MASK) << QAT_COMN_PTR_TYPE_BITPOS))
#define ICP_QAT_FW_COMN_PTR_TYPE_GET(flags) \
QAT_FIELD_GET(flags, QAT_COMN_PTR_TYPE_BITPOS, QAT_COMN_PTR_TYPE_MASK)
#define ICP_QAT_FW_COMN_CD_FLD_TYPE_GET(flags) \
QAT_FIELD_GET(flags, \
QAT_COMN_CD_FLD_TYPE_BITPOS, \
QAT_COMN_CD_FLD_TYPE_MASK)
#define ICP_QAT_FW_COMN_PTR_TYPE_SET(flags, val) \
QAT_FIELD_SET(flags, \
val, \
QAT_COMN_PTR_TYPE_BITPOS, \
QAT_COMN_PTR_TYPE_MASK)
#define ICP_QAT_FW_COMN_CD_FLD_TYPE_SET(flags, val) \
QAT_FIELD_SET(flags, \
val, \
QAT_COMN_CD_FLD_TYPE_BITPOS, \
QAT_COMN_CD_FLD_TYPE_MASK)
#define ICP_QAT_FW_COMN_NEXT_ID_BITPOS 4
#define ICP_QAT_FW_COMN_NEXT_ID_MASK 0xF0
#define ICP_QAT_FW_COMN_CURR_ID_BITPOS 0
#define ICP_QAT_FW_COMN_CURR_ID_MASK 0x0F
#define ICP_QAT_FW_COMN_NEXT_ID_GET(cd_ctrl_hdr_t) \
((((cd_ctrl_hdr_t)->next_curr_id) & ICP_QAT_FW_COMN_NEXT_ID_MASK) >> \
(ICP_QAT_FW_COMN_NEXT_ID_BITPOS))
#define ICP_QAT_FW_COMN_NEXT_ID_SET(cd_ctrl_hdr_t, val) \
{ \
((cd_ctrl_hdr_t)->next_curr_id) = \
((((cd_ctrl_hdr_t)->next_curr_id) & \
ICP_QAT_FW_COMN_CURR_ID_MASK) | \
((val << ICP_QAT_FW_COMN_NEXT_ID_BITPOS) & \
ICP_QAT_FW_COMN_NEXT_ID_MASK)); \
}
#define ICP_QAT_FW_COMN_CURR_ID_GET(cd_ctrl_hdr_t) \
(((cd_ctrl_hdr_t)->next_curr_id) & ICP_QAT_FW_COMN_CURR_ID_MASK)
#define ICP_QAT_FW_COMN_CURR_ID_SET(cd_ctrl_hdr_t, val) \
{ \
((cd_ctrl_hdr_t)->next_curr_id) = \
((((cd_ctrl_hdr_t)->next_curr_id) & \
ICP_QAT_FW_COMN_NEXT_ID_MASK) | \
((val)&ICP_QAT_FW_COMN_CURR_ID_MASK)); \
}
#define QAT_COMN_RESP_CRYPTO_STATUS_BITPOS 7
#define QAT_COMN_RESP_CRYPTO_STATUS_MASK 0x1
#define QAT_COMN_RESP_PKE_STATUS_BITPOS 6
#define QAT_COMN_RESP_PKE_STATUS_MASK 0x1
#define QAT_COMN_RESP_CMP_STATUS_BITPOS 5
#define QAT_COMN_RESP_CMP_STATUS_MASK 0x1
#define QAT_COMN_RESP_XLAT_STATUS_BITPOS 4
#define QAT_COMN_RESP_XLAT_STATUS_MASK 0x1
#define QAT_COMN_RESP_CMP_END_OF_LAST_BLK_BITPOS 3
#define QAT_COMN_RESP_CMP_END_OF_LAST_BLK_MASK 0x1
#define ICP_QAT_FW_COMN_RESP_STATUS_BUILD(crypto, comp, xlat, eolb) \
((((crypto)&QAT_COMN_RESP_CRYPTO_STATUS_MASK) \
<< QAT_COMN_RESP_CRYPTO_STATUS_BITPOS) | \
(((comp)&QAT_COMN_RESP_CMP_STATUS_MASK) \
<< QAT_COMN_RESP_CMP_STATUS_BITPOS) | \
(((xlat)&QAT_COMN_RESP_XLAT_STATUS_MASK) \
<< QAT_COMN_RESP_XLAT_STATUS_BITPOS) | \
(((eolb)&QAT_COMN_RESP_CMP_END_OF_LAST_BLK_MASK) \
<< QAT_COMN_RESP_CMP_END_OF_LAST_BLK_BITPOS))
#define ICP_QAT_FW_COMN_RESP_CRYPTO_STAT_GET(status) \
QAT_FIELD_GET(status, \
QAT_COMN_RESP_CRYPTO_STATUS_BITPOS, \
QAT_COMN_RESP_CRYPTO_STATUS_MASK)
#define ICP_QAT_FW_COMN_RESP_CMP_STAT_GET(status) \
QAT_FIELD_GET(status, \
QAT_COMN_RESP_CMP_STATUS_BITPOS, \
QAT_COMN_RESP_CMP_STATUS_MASK)
#define ICP_QAT_FW_COMN_RESP_XLAT_STAT_GET(status) \
QAT_FIELD_GET(status, \
QAT_COMN_RESP_XLAT_STATUS_BITPOS, \
QAT_COMN_RESP_XLAT_STATUS_MASK)
#define ICP_QAT_FW_COMN_RESP_CMP_END_OF_LAST_BLK_FLAG_GET(status) \
QAT_FIELD_GET(status, \
QAT_COMN_RESP_CMP_END_OF_LAST_BLK_BITPOS, \
QAT_COMN_RESP_CMP_END_OF_LAST_BLK_MASK)
#define ICP_QAT_FW_COMN_STATUS_FLAG_OK 0
#define ICP_QAT_FW_COMN_STATUS_FLAG_ERROR 1
#define ICP_QAT_FW_COMN_STATUS_CMP_END_OF_LAST_BLK_FLAG_CLR 0
#define ICP_QAT_FW_COMN_STATUS_CMP_END_OF_LAST_BLK_FLAG_SET 1
#define ERR_CODE_NO_ERROR 0
#define ERR_CODE_INVALID_BLOCK_TYPE -1
#define ERR_CODE_NO_MATCH_ONES_COMP -2
#define ERR_CODE_TOO_MANY_LEN_OR_DIS -3
#define ERR_CODE_INCOMPLETE_LEN -4
#define ERR_CODE_RPT_LEN_NO_FIRST_LEN -5
#define ERR_CODE_RPT_GT_SPEC_LEN -6
#define ERR_CODE_INV_LIT_LEN_CODE_LEN -7
#define ERR_CODE_INV_DIS_CODE_LEN -8
#define ERR_CODE_INV_LIT_LEN_DIS_IN_BLK -9
#define ERR_CODE_DIS_TOO_FAR_BACK -10
#define ERR_CODE_OVERFLOW_ERROR -11
#define ERR_CODE_SOFT_ERROR -12
#define ERR_CODE_FATAL_ERROR -13
#define ERR_CODE_SSM_ERROR -14
#define ERR_CODE_ENDPOINT_ERROR -15
enum icp_qat_fw_slice {
ICP_QAT_FW_SLICE_NULL = 0,
ICP_QAT_FW_SLICE_CIPHER = 1,
ICP_QAT_FW_SLICE_AUTH = 2,
ICP_QAT_FW_SLICE_DRAM_RD = 3,
ICP_QAT_FW_SLICE_DRAM_WR = 4,
ICP_QAT_FW_SLICE_COMP = 5,
ICP_QAT_FW_SLICE_XLAT = 6,
ICP_QAT_FW_SLICE_DELIMITER
};
#endif

222
sys/dev/qat/include/icp_qat_fw_init_admin.h Normal file
View File

@ -0,0 +1,222 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef _ICP_QAT_FW_INIT_ADMIN_H_
#define _ICP_QAT_FW_INIT_ADMIN_H_
#include "icp_qat_fw.h"
enum icp_qat_fw_init_admin_cmd_id {
ICP_QAT_FW_INIT_ME = 0,
ICP_QAT_FW_TRNG_ENABLE = 1,
ICP_QAT_FW_TRNG_DISABLE = 2,
ICP_QAT_FW_CONSTANTS_CFG = 3,
ICP_QAT_FW_STATUS_GET = 4,
ICP_QAT_FW_COUNTERS_GET = 5,
ICP_QAT_FW_LOOPBACK = 6,
ICP_QAT_FW_HEARTBEAT_SYNC = 7,
ICP_QAT_FW_HEARTBEAT_GET = 8,
ICP_QAT_FW_COMP_CAPABILITY_GET = 9,
ICP_QAT_FW_CRYPTO_CAPABILITY_GET = 10,
ICP_QAT_FW_HEARTBEAT_TIMER_SET = 13,
ICP_QAT_FW_RL_SLA_CONFIG = 14,
ICP_QAT_FW_RL_INIT = 15,
ICP_QAT_FW_RL_DU_START = 16,
ICP_QAT_FW_RL_DU_STOP = 17,
ICP_QAT_FW_TIMER_GET = 19,
ICP_QAT_FW_CNV_STATS_GET = 20,
ICP_QAT_FW_PKE_REPLAY_STATS_GET = 21
};
enum icp_qat_fw_init_admin_resp_status {
ICP_QAT_FW_INIT_RESP_STATUS_SUCCESS = 0,
ICP_QAT_FW_INIT_RESP_STATUS_FAIL = 1,
ICP_QAT_FW_INIT_RESP_STATUS_UNSUPPORTED = 4
};
enum icp_qat_fw_cnv_error_type {
CNV_ERR_TYPE_NO_ERROR = 0,
CNV_ERR_TYPE_CHECKSUM_ERROR,
CNV_ERR_TYPE_DECOMP_PRODUCED_LENGTH_ERROR,
CNV_ERR_TYPE_DECOMPRESSION_ERROR,
CNV_ERR_TYPE_TRANSLATION_ERROR,
CNV_ERR_TYPE_DECOMP_CONSUMED_LENGTH_ERROR,
CNV_ERR_TYPE_UNKNOWN_ERROR
};
#define CNV_ERROR_TYPE_GET(latest_error) \
({ \
__typeof__(latest_error) _lerror = latest_error; \
(_lerror >> 12) > CNV_ERR_TYPE_UNKNOWN_ERROR ? \
CNV_ERR_TYPE_UNKNOWN_ERROR : \
(enum icp_qat_fw_cnv_error_type)(_lerror >> 12); \
})
#define CNV_ERROR_LENGTH_DELTA_GET(latest_error) \
({ \
__typeof__(latest_error) _lerror = latest_error; \
((s16)((_lerror & 0x0FFF) | (_lerror & 0x0800 ? 0xF000 : 0))); \
})
#define CNV_ERROR_DECOMP_STATUS_GET(latest_error) ((s8)(latest_error & 0xFF))
struct icp_qat_fw_init_admin_req {
u16 init_cfg_sz;
u8 resrvd1;
u8 cmd_id;
u32 max_req_duration;
u64 opaque_data;
union {
/* ICP_QAT_FW_INIT_ME */
struct {
u64 resrvd2;
u16 ibuf_size_in_kb;
u16 resrvd3;
u32 resrvd4;
};
/* ICP_QAT_FW_CONSTANTS_CFG */
struct {
u64 init_cfg_ptr;
u64 resrvd5;
};
/* ICP_QAT_FW_HEARTBEAT_TIMER_SET */
struct {
u64 hb_cfg_ptr;
u32 heartbeat_ticks;
u32 resrvd6;
};
/* ICP_QAT_FW_RL_SLA_CONFIG */
struct {
u32 credit_per_sla;
u8 service_id;
u8 vf_id;
u8 resrvd7;
u8 resrvd8;
u32 resrvd9;
u32 resrvd10;
};
/* ICP_QAT_FW_RL_INIT */
struct {
u32 rl_period;
u8 config;
u8 resrvd11;
u8 num_me;
u8 resrvd12;
u8 pke_svc_arb_map;
u8 bulk_crypto_svc_arb_map;
u8 compression_svc_arb_map;
u8 resrvd13;
u32 resrvd14;
};
/* ICP_QAT_FW_RL_DU_STOP */
struct {
u64 cfg_ptr;
u32 resrvd15;
u32 resrvd16;
};
};
} __packed;
struct icp_qat_fw_init_admin_resp {
u8 flags;
u8 resrvd1;
u8 status;
u8 cmd_id;
union {
u32 resrvd2;
u32 ras_event_count;
/* ICP_QAT_FW_STATUS_GET */
struct {
u16 version_minor_num;
u16 version_major_num;
};
/* ICP_QAT_FW_COMP_CAPABILITY_GET */
u32 extended_features;
/* ICP_QAT_FW_CNV_STATS_GET */
struct {
u16 error_count;
u16 latest_error;
};
};
u64 opaque_data;
union {
u32 resrvd3[4];
/* ICP_QAT_FW_STATUS_GET */
struct {
u32 version_patch_num;
u8 context_id;
u8 ae_id;
u16 resrvd4;
u64 resrvd5;
};
/* ICP_QAT_FW_COMP_CAPABILITY_GET */
struct {
u16 compression_algos;
u16 checksum_algos;
u32 deflate_capabilities;
u32 resrvd6;
u32 deprecated;
};
/* ICP_QAT_FW_CRYPTO_CAPABILITY_GET */
struct {
u32 cipher_algos;
u32 hash_algos;
u16 keygen_algos;
u16 other;
u16 public_key_algos;
u16 prime_algos;
};
/* ICP_QAT_FW_RL_DU_STOP */
struct {
u32 resrvd7;
u8 granularity;
u8 resrvd8;
u16 resrvd9;
u32 total_du_time;
u32 resrvd10;
};
/* ICP_QAT_FW_TIMER_GET */
struct {
u64 timestamp;
u64 resrvd11;
};
/* ICP_QAT_FW_COUNTERS_GET */
struct {
u64 req_rec_count;
u64 resp_sent_count;
};
/* ICP_QAT_FW_PKE_REPLAY_STATS_GET */
struct {
u32 successful_count;
u32 unsuccessful_count;
u64 resrvd12;
};
};
} __packed;
enum icp_qat_fw_init_admin_init_flag { ICP_QAT_FW_INIT_FLAG_PKE_DISABLED = 0 };
struct icp_qat_fw_init_admin_hb_cnt {
u16 resp_heartbeat_cnt;
u16 req_heartbeat_cnt;
};
struct icp_qat_fw_init_admin_hb_stats {
struct icp_qat_fw_init_admin_hb_cnt stats[ADF_NUM_HB_CNT_PER_AE];
};
#define ICP_QAT_FW_COMN_HEARTBEAT_OK 0
#define ICP_QAT_FW_COMN_HEARTBEAT_BLOCKED 1
#define ICP_QAT_FW_COMN_HEARTBEAT_FLAG_BITPOS 0
#define ICP_QAT_FW_COMN_HEARTBEAT_FLAG_MASK 0x1
#define ICP_QAT_FW_COMN_STATUS_RESRVD_FLD_MASK 0xFE
#define ICP_QAT_FW_COMN_HEARTBEAT_HDR_FLAG_GET(hdr_t) \
ICP_QAT_FW_COMN_HEARTBEAT_FLAG_GET(hdr_t.flags)
#define ICP_QAT_FW_COMN_HEARTBEAT_HDR_FLAG_SET(hdr_t, val) \
ICP_QAT_FW_COMN_HEARTBEAT_FLAG_SET(hdr_t, val)
#define ICP_QAT_FW_COMN_HEARTBEAT_FLAG_GET(flags) \
QAT_FIELD_GET(flags, \
ICP_QAT_FW_COMN_HEARTBEAT_FLAG_BITPOS, \
ICP_QAT_FW_COMN_HEARTBEAT_FLAG_MASK)
#endif

326
sys/dev/qat/include/icp_qat_hw.h Normal file
View File

@ -0,0 +1,326 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef _ICP_QAT_HW_H_
#define _ICP_QAT_HW_H_
enum icp_qat_hw_ae_id {
ICP_QAT_HW_AE_0 = 0,
ICP_QAT_HW_AE_1 = 1,
ICP_QAT_HW_AE_2 = 2,
ICP_QAT_HW_AE_3 = 3,
ICP_QAT_HW_AE_4 = 4,
ICP_QAT_HW_AE_5 = 5,
ICP_QAT_HW_AE_6 = 6,
ICP_QAT_HW_AE_7 = 7,
ICP_QAT_HW_AE_8 = 8,
ICP_QAT_HW_AE_9 = 9,
ICP_QAT_HW_AE_10 = 10,
ICP_QAT_HW_AE_11 = 11,
ICP_QAT_HW_AE_DELIMITER = 12
};
enum icp_qat_hw_qat_id {
ICP_QAT_HW_QAT_0 = 0,
ICP_QAT_HW_QAT_1 = 1,
ICP_QAT_HW_QAT_2 = 2,
ICP_QAT_HW_QAT_3 = 3,
ICP_QAT_HW_QAT_4 = 4,
ICP_QAT_HW_QAT_5 = 5,
ICP_QAT_HW_QAT_DELIMITER = 6
};
enum icp_qat_hw_auth_algo {
ICP_QAT_HW_AUTH_ALGO_NULL = 0,
ICP_QAT_HW_AUTH_ALGO_SHA1 = 1,
ICP_QAT_HW_AUTH_ALGO_MD5 = 2,
ICP_QAT_HW_AUTH_ALGO_SHA224 = 3,
ICP_QAT_HW_AUTH_ALGO_SHA256 = 4,
ICP_QAT_HW_AUTH_ALGO_SHA384 = 5,
ICP_QAT_HW_AUTH_ALGO_SHA512 = 6,
ICP_QAT_HW_AUTH_ALGO_AES_XCBC_MAC = 7,
ICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC = 8,
ICP_QAT_HW_AUTH_ALGO_AES_F9 = 9,
ICP_QAT_HW_AUTH_ALGO_GALOIS_128 = 10,
ICP_QAT_HW_AUTH_ALGO_GALOIS_64 = 11,
ICP_QAT_HW_AUTH_ALGO_KASUMI_F9 = 12,
ICP_QAT_HW_AUTH_ALGO_SNOW_3G_UIA2 = 13,
ICP_QAT_HW_AUTH_ALGO_ZUC_3G_128_EIA3 = 14,
ICP_QAT_HW_AUTH_RESERVED_1 = 15,
ICP_QAT_HW_AUTH_RESERVED_2 = 16,
ICP_QAT_HW_AUTH_ALGO_SHA3_256 = 17,
ICP_QAT_HW_AUTH_RESERVED_3 = 18,
ICP_QAT_HW_AUTH_ALGO_SHA3_512 = 19,
ICP_QAT_HW_AUTH_ALGO_DELIMITER = 20
};
enum icp_qat_hw_auth_mode {
ICP_QAT_HW_AUTH_MODE0 = 0,
ICP_QAT_HW_AUTH_MODE1 = 1,
ICP_QAT_HW_AUTH_MODE2 = 2,
ICP_QAT_HW_AUTH_MODE_DELIMITER = 3
};
struct icp_qat_hw_auth_config {
uint32_t config;
uint32_t reserved;
};
enum icp_qat_slice_mask {
ICP_ACCEL_MASK_CIPHER_SLICE = 0x01,
ICP_ACCEL_MASK_AUTH_SLICE = 0x02,
ICP_ACCEL_MASK_PKE_SLICE = 0x04,
ICP_ACCEL_MASK_COMPRESS_SLICE = 0x08,
ICP_ACCEL_MASK_DEPRECATED = 0x10,
ICP_ACCEL_MASK_EIA3_SLICE = 0x20,
ICP_ACCEL_MASK_SHA3_SLICE = 0x40,
ICP_ACCEL_MASK_CRYPTO0_SLICE = 0x80,
ICP_ACCEL_MASK_CRYPTO1_SLICE = 0x100,
ICP_ACCEL_MASK_CRYPTO2_SLICE = 0x200,
ICP_ACCEL_MASK_SM3_SLICE = 0x400,
ICP_ACCEL_MASK_SM4_SLICE = 0x800
};
enum icp_qat_capabilities_mask {
ICP_ACCEL_CAPABILITIES_CRYPTO_SYMMETRIC = BIT(0),
ICP_ACCEL_CAPABILITIES_CRYPTO_ASYMMETRIC = BIT(1),
ICP_ACCEL_CAPABILITIES_CIPHER = BIT(2),
ICP_ACCEL_CAPABILITIES_AUTHENTICATION = BIT(3),
ICP_ACCEL_CAPABILITIES_RESERVED_1 = BIT(4),
ICP_ACCEL_CAPABILITIES_COMPRESSION = BIT(5),
ICP_ACCEL_CAPABILITIES_DEPRECATED = BIT(6),
ICP_ACCEL_CAPABILITIES_RAND = BIT(7),
ICP_ACCEL_CAPABILITIES_ZUC = BIT(8),
ICP_ACCEL_CAPABILITIES_SHA3 = BIT(9),
ICP_ACCEL_CAPABILITIES_KPT = BIT(10),
ICP_ACCEL_CAPABILITIES_RL = BIT(11),
ICP_ACCEL_CAPABILITIES_HKDF = BIT(12),
ICP_ACCEL_CAPABILITIES_ECEDMONT = BIT(13),
ICP_ACCEL_CAPABILITIES_EXT_ALGCHAIN = BIT(14),
ICP_ACCEL_CAPABILITIES_SHA3_EXT = BIT(15),
ICP_ACCEL_CAPABILITIES_AESGCM_SPC = BIT(16),
ICP_ACCEL_CAPABILITIES_CHACHA_POLY = BIT(17),
ICP_ACCEL_CAPABILITIES_SM2 = BIT(18),
ICP_ACCEL_CAPABILITIES_SM3 = BIT(19),
ICP_ACCEL_CAPABILITIES_SM4 = BIT(20),
ICP_ACCEL_CAPABILITIES_INLINE = BIT(21),
ICP_ACCEL_CAPABILITIES_CNV_INTEGRITY = BIT(22),
ICP_ACCEL_CAPABILITIES_CNV_INTEGRITY64 = BIT(23),
ICP_ACCEL_CAPABILITIES_LZ4_COMPRESSION = BIT(24),
ICP_ACCEL_CAPABILITIES_LZ4S_COMPRESSION = BIT(25),
ICP_ACCEL_CAPABILITIES_AES_V2 = BIT(26),
ICP_ACCEL_CAPABILITIES_KPT2 = BIT(27),
};
enum icp_qat_extended_dc_capabilities_mask {
ICP_ACCEL_CAPABILITIES_ADVANCED_COMPRESSION = 0x101
};
#define QAT_AUTH_MODE_BITPOS 4
#define QAT_AUTH_MODE_MASK 0xF
#define QAT_AUTH_ALGO_BITPOS 0
#define QAT_AUTH_ALGO_MASK 0xF
#define QAT_AUTH_CMP_BITPOS 8
#define QAT_AUTH_HIGH_BIT 4
#define QAT_AUTH_CMP_MASK 0x7F
#define QAT_AUTH_SHA3_PADDING_BITPOS 16
#define QAT_AUTH_SHA3_PADDING_MASK 0x1
#define QAT_AUTH_ALGO_SHA3_BITPOS 22
#define QAT_AUTH_ALGO_SHA3_MASK 0x3
#define ICP_QAT_HW_AUTH_CONFIG_BUILD(mode, algo, cmp_len) \
(((mode & QAT_AUTH_MODE_MASK) << QAT_AUTH_MODE_BITPOS) | \
((algo & QAT_AUTH_ALGO_MASK) << QAT_AUTH_ALGO_BITPOS) | \
(((algo >> 4) & QAT_AUTH_ALGO_SHA3_MASK) \
<< QAT_AUTH_ALGO_SHA3_BITPOS) | \
(((((algo == ICP_QAT_HW_AUTH_ALGO_SHA3_256) || \
(algo == ICP_QAT_HW_AUTH_ALGO_SHA3_512)) ? \
1 : \
0) & \
QAT_AUTH_SHA3_PADDING_MASK) \
<< QAT_AUTH_SHA3_PADDING_BITPOS) | \
((cmp_len & QAT_AUTH_CMP_MASK) << QAT_AUTH_CMP_BITPOS))
struct icp_qat_hw_auth_counter {
__be32 counter;
uint32_t reserved;
};
#define QAT_AUTH_COUNT_MASK 0xFFFFFFFF
#define QAT_AUTH_COUNT_BITPOS 0
#define ICP_QAT_HW_AUTH_COUNT_BUILD(val) \
(((val)&QAT_AUTH_COUNT_MASK) << QAT_AUTH_COUNT_BITPOS)
struct icp_qat_hw_auth_setup {
struct icp_qat_hw_auth_config auth_config;
struct icp_qat_hw_auth_counter auth_counter;
};
#define QAT_HW_DEFAULT_ALIGNMENT 8
#define QAT_HW_ROUND_UP(val, n) (((val) + ((n)-1)) & (~(n - 1)))
#define ICP_QAT_HW_NULL_STATE1_SZ 32
#define ICP_QAT_HW_MD5_STATE1_SZ 16
#define ICP_QAT_HW_SHA1_STATE1_SZ 20
#define ICP_QAT_HW_SHA224_STATE1_SZ 32
#define ICP_QAT_HW_SHA256_STATE1_SZ 32
#define ICP_QAT_HW_SHA3_256_STATE1_SZ 32
#define ICP_QAT_HW_SHA384_STATE1_SZ 64
#define ICP_QAT_HW_SHA512_STATE1_SZ 64
#define ICP_QAT_HW_SHA3_512_STATE1_SZ 64
#define ICP_QAT_HW_SHA3_224_STATE1_SZ 28
#define ICP_QAT_HW_SHA3_384_STATE1_SZ 48
#define ICP_QAT_HW_AES_XCBC_MAC_STATE1_SZ 16
#define ICP_QAT_HW_AES_CBC_MAC_STATE1_SZ 16
#define ICP_QAT_HW_AES_F9_STATE1_SZ 32
#define ICP_QAT_HW_KASUMI_F9_STATE1_SZ 16
#define ICP_QAT_HW_GALOIS_128_STATE1_SZ 16
#define ICP_QAT_HW_SNOW_3G_UIA2_STATE1_SZ 8
#define ICP_QAT_HW_ZUC_3G_EIA3_STATE1_SZ 8
#define ICP_QAT_HW_NULL_STATE2_SZ 32
#define ICP_QAT_HW_MD5_STATE2_SZ 16
#define ICP_QAT_HW_SHA1_STATE2_SZ 20
#define ICP_QAT_HW_SHA224_STATE2_SZ 32
#define ICP_QAT_HW_SHA256_STATE2_SZ 32
#define ICP_QAT_HW_SHA3_256_STATE2_SZ 0
#define ICP_QAT_HW_SHA384_STATE2_SZ 64
#define ICP_QAT_HW_SHA512_STATE2_SZ 64
#define ICP_QAT_HW_SHA3_512_STATE2_SZ 0
#define ICP_QAT_HW_SHA3_224_STATE2_SZ 0
#define ICP_QAT_HW_SHA3_384_STATE2_SZ 0
#define ICP_QAT_HW_AES_XCBC_MAC_KEY_SZ 16
#define ICP_QAT_HW_AES_CBC_MAC_KEY_SZ 16
#define ICP_QAT_HW_AES_CCM_CBC_E_CTR0_SZ 16
#define ICP_QAT_HW_F9_IK_SZ 16
#define ICP_QAT_HW_F9_FK_SZ 16
#define ICP_QAT_HW_KASUMI_F9_STATE2_SZ \
(ICP_QAT_HW_F9_IK_SZ + ICP_QAT_HW_F9_FK_SZ)
#define ICP_QAT_HW_AES_F9_STATE2_SZ ICP_QAT_HW_KASUMI_F9_STATE2_SZ
#define ICP_QAT_HW_SNOW_3G_UIA2_STATE2_SZ 24
#define ICP_QAT_HW_ZUC_3G_EIA3_STATE2_SZ 32
#define ICP_QAT_HW_GALOIS_H_SZ 16
#define ICP_QAT_HW_GALOIS_LEN_A_SZ 8
#define ICP_QAT_HW_GALOIS_E_CTR0_SZ 16
struct icp_qat_hw_auth_sha512 {
struct icp_qat_hw_auth_setup inner_setup;
uint8_t state1[ICP_QAT_HW_SHA512_STATE1_SZ];
struct icp_qat_hw_auth_setup outer_setup;
uint8_t state2[ICP_QAT_HW_SHA512_STATE2_SZ];
};
struct icp_qat_hw_auth_algo_blk {
struct icp_qat_hw_auth_sha512 sha;
};
#define ICP_QAT_HW_GALOIS_LEN_A_BITPOS 0
#define ICP_QAT_HW_GALOIS_LEN_A_MASK 0xFFFFFFFF
enum icp_qat_hw_cipher_algo {
ICP_QAT_HW_CIPHER_ALGO_NULL = 0,
ICP_QAT_HW_CIPHER_ALGO_DES = 1,
ICP_QAT_HW_CIPHER_ALGO_3DES = 2,
ICP_QAT_HW_CIPHER_ALGO_AES128 = 3,
ICP_QAT_HW_CIPHER_ALGO_AES192 = 4,
ICP_QAT_HW_CIPHER_ALGO_AES256 = 5,
ICP_QAT_HW_CIPHER_ALGO_ARC4 = 6,
ICP_QAT_HW_CIPHER_ALGO_KASUMI = 7,
ICP_QAT_HW_CIPHER_ALGO_SNOW_3G_UEA2 = 8,
ICP_QAT_HW_CIPHER_ALGO_ZUC_3G_128_EEA3 = 9,
ICP_QAT_HW_CIPHER_ALGO_SM4 = 10,
ICP_QAT_HW_CIPHER_ALGO_CHACHA20_POLY1305 = 11,
ICP_QAT_HW_CIPHER_DELIMITER = 12
};
enum icp_qat_hw_cipher_mode {
ICP_QAT_HW_CIPHER_ECB_MODE = 0,
ICP_QAT_HW_CIPHER_CBC_MODE = 1,
ICP_QAT_HW_CIPHER_CTR_MODE = 2,
ICP_QAT_HW_CIPHER_F8_MODE = 3,
ICP_QAT_HW_CIPHER_AEAD_MODE = 4,
ICP_QAT_HW_CIPHER_RESERVED_MODE = 5,
ICP_QAT_HW_CIPHER_XTS_MODE = 6,
ICP_QAT_HW_CIPHER_MODE_DELIMITER = 7
};
struct icp_qat_hw_cipher_config {
uint32_t val;
uint32_t reserved;
};
enum icp_qat_hw_cipher_dir {
ICP_QAT_HW_CIPHER_ENCRYPT = 0,
ICP_QAT_HW_CIPHER_DECRYPT = 1,
};
enum icp_qat_hw_cipher_convert {
ICP_QAT_HW_CIPHER_NO_CONVERT = 0,
ICP_QAT_HW_CIPHER_KEY_CONVERT = 1,
};
#define QAT_CIPHER_MODE_BITPOS 4
#define QAT_CIPHER_MODE_MASK 0xF
#define QAT_CIPHER_ALGO_BITPOS 0
#define QAT_CIPHER_ALGO_MASK 0xF
#define QAT_CIPHER_CONVERT_BITPOS 9
#define QAT_CIPHER_CONVERT_MASK 0x1
#define QAT_CIPHER_DIR_BITPOS 8
#define QAT_CIPHER_DIR_MASK 0x1
#define QAT_CIPHER_AEAD_HASH_CMP_LEN_MASK 0x1F
#define QAT_CIPHER_AEAD_HASH_CMP_LEN_BITPOS 10
#define QAT_CIPHER_AEAD_AAD_SIZE_LOWER_MASK 0xFF
#define QAT_CIPHER_AEAD_AAD_SIZE_UPPER_MASK 0x3F
#define QAT_CIPHER_AEAD_AAD_UPPER_SHIFT 8
#define QAT_CIPHER_AEAD_AAD_LOWER_SHIFT 24
#define QAT_CIPHER_AEAD_AAD_SIZE_BITPOS 16
#define QAT_CIPHER_MODE_F8_KEY_SZ_MULT 2
#define QAT_CIPHER_MODE_XTS_KEY_SZ_MULT 2
#define ICP_QAT_HW_CIPHER_CONFIG_BUILD(mode, algo, convert, dir) \
(((mode & QAT_CIPHER_MODE_MASK) << QAT_CIPHER_MODE_BITPOS) | \
((algo & QAT_CIPHER_ALGO_MASK) << QAT_CIPHER_ALGO_BITPOS) | \
((convert & QAT_CIPHER_CONVERT_MASK) << QAT_CIPHER_CONVERT_BITPOS) | \
((dir & QAT_CIPHER_DIR_MASK) << QAT_CIPHER_DIR_BITPOS))
#define ICP_QAT_HW_DES_BLK_SZ 8
#define ICP_QAT_HW_3DES_BLK_SZ 8
#define ICP_QAT_HW_NULL_BLK_SZ 8
#define ICP_QAT_HW_AES_BLK_SZ 16
#define ICP_QAT_HW_KASUMI_BLK_SZ 8
#define ICP_QAT_HW_SNOW_3G_BLK_SZ 8
#define ICP_QAT_HW_ZUC_3G_BLK_SZ 8
#define ICP_QAT_HW_NULL_KEY_SZ 256
#define ICP_QAT_HW_DES_KEY_SZ 8
#define ICP_QAT_HW_3DES_KEY_SZ 24
#define ICP_QAT_HW_AES_128_KEY_SZ 16
#define ICP_QAT_HW_AES_192_KEY_SZ 24
#define ICP_QAT_HW_AES_256_KEY_SZ 32
#define ICP_QAT_HW_AES_128_F8_KEY_SZ \
(ICP_QAT_HW_AES_128_KEY_SZ * QAT_CIPHER_MODE_F8_KEY_SZ_MULT)
#define ICP_QAT_HW_AES_192_F8_KEY_SZ \
(ICP_QAT_HW_AES_192_KEY_SZ * QAT_CIPHER_MODE_F8_KEY_SZ_MULT)
#define ICP_QAT_HW_AES_256_F8_KEY_SZ \
(ICP_QAT_HW_AES_256_KEY_SZ * QAT_CIPHER_MODE_F8_KEY_SZ_MULT)
#define ICP_QAT_HW_AES_128_XTS_KEY_SZ \
(ICP_QAT_HW_AES_128_KEY_SZ * QAT_CIPHER_MODE_XTS_KEY_SZ_MULT)
#define ICP_QAT_HW_AES_256_XTS_KEY_SZ \
(ICP_QAT_HW_AES_256_KEY_SZ * QAT_CIPHER_MODE_XTS_KEY_SZ_MULT)
#define ICP_QAT_HW_KASUMI_KEY_SZ 16
#define ICP_QAT_HW_KASUMI_F8_KEY_SZ \
(ICP_QAT_HW_KASUMI_KEY_SZ * QAT_CIPHER_MODE_F8_KEY_SZ_MULT)
#define ICP_QAT_HW_AES_128_XTS_KEY_SZ \
(ICP_QAT_HW_AES_128_KEY_SZ * QAT_CIPHER_MODE_XTS_KEY_SZ_MULT)
#define ICP_QAT_HW_AES_256_XTS_KEY_SZ \
(ICP_QAT_HW_AES_256_KEY_SZ * QAT_CIPHER_MODE_XTS_KEY_SZ_MULT)
#define ICP_QAT_HW_ARC4_KEY_SZ 256
#define ICP_QAT_HW_SNOW_3G_UEA2_KEY_SZ 16
#define ICP_QAT_HW_SNOW_3G_UEA2_IV_SZ 16
#define ICP_QAT_HW_ZUC_3G_EEA3_KEY_SZ 16
#define ICP_QAT_HW_ZUC_3G_EEA3_IV_SZ 16
#define ICP_QAT_HW_MODE_F8_NUM_REG_TO_CLEAR 2
#define INIT_SHRAM_CONSTANTS_TABLE_SZ 1024
struct icp_qat_hw_cipher_aes256_f8 {
struct icp_qat_hw_cipher_config cipher_config;
uint8_t key[ICP_QAT_HW_AES_256_F8_KEY_SZ];
};
struct icp_qat_hw_cipher_algo_blk {
struct icp_qat_hw_cipher_aes256_f8 aes;
} __aligned(64);
#endif

142
sys/dev/qat/include/qat_ocf_mem_pool.h Normal file
View File

@ -0,0 +1,142 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef _QAT_OCF_MEM_POOL_H_
#define _QAT_OCF_MEM_POOL_H_
/* System headers */
#include <sys/types.h>
/* QAT specific headers */
#include "cpa.h"
#include "cpa_cy_sym_dp.h"
#include "icp_qat_fw_la.h"
#define QAT_OCF_MAX_LEN (64 * 1024)
#define QAT_OCF_MAX_FLATS (32)
#define QAT_OCF_MAX_DIGEST SHA512_DIGEST_LENGTH
#define QAT_OCF_MAX_SYMREQ (256)
#define QAT_OCF_MEM_POOL_SIZE ((QAT_OCF_MAX_SYMREQ * 2 + 1) * 2)
#define QAT_OCF_MAXLEN 64 * 1024
/* Dedicated structure due to flexible arrays not allowed to be
* allocated on stack */
struct qat_ocf_buffer_list {
Cpa64U reserved0;
Cpa32U numBuffers;
Cpa32U reserved1;
CpaPhysFlatBuffer flatBuffers[QAT_OCF_MAX_FLATS];
};
struct qat_ocf_dma_mem {
bus_dma_tag_t dma_tag;
bus_dmamap_t dma_map;
bus_dma_segment_t dma_seg;
void *dma_vaddr;
} __aligned(64);
struct qat_ocf_cookie {
/* Source SGLs */
struct qat_ocf_buffer_list src_buffers;
/* Destination SGL */
struct qat_ocf_buffer_list dst_buffers;
/* Cache OP data */
CpaCySymDpOpData pOpdata;
/* IV max size taken from cryptdev */
uint8_t qat_ocf_iv_buf[EALG_MAX_BLOCK_LEN];
bus_addr_t qat_ocf_iv_buf_paddr;
uint8_t qat_ocf_digest[QAT_OCF_MAX_DIGEST];
bus_addr_t qat_ocf_digest_paddr;
/* Used only in case of separated AAD and GCM, CCM and RC4 */
uint8_t qat_ocf_gcm_aad[ICP_QAT_FW_CCM_GCM_AAD_SZ_MAX];
bus_addr_t qat_ocf_gcm_aad_paddr;
/* Source SGLs */
struct qat_ocf_dma_mem src_dma_mem;
bus_addr_t src_buffer_list_paddr;
/* Destination SGL */
struct qat_ocf_dma_mem dst_dma_mem;
bus_addr_t dst_buffer_list_paddr;
/* AAD - used only if separated AAD is used by OCF and HW requires
* to have it at the beginning of source buffer */
struct qat_ocf_dma_mem gcm_aad_dma_mem;
bus_addr_t gcm_aad_buffer_list_paddr;
CpaBoolean is_sep_aad_used;
/* Cache OP data */
bus_addr_t pOpData_paddr;
/* misc */
struct cryptop *crp_op;
/* This cookie tag and map */
bus_dma_tag_t dma_tag;
bus_dmamap_t dma_map;
};
struct qat_ocf_session {
CpaCySymSessionCtx sessionCtx;
Cpa32U sessionCtxSize;
Cpa32U authLen;
Cpa32U aadLen;
};
struct qat_ocf_dsession {
struct qat_ocf_instance *qatInstance;
struct qat_ocf_session encSession;
struct qat_ocf_session decSession;
};
struct qat_ocf_load_cb_arg {
struct cryptop *crp_op;
struct qat_ocf_cookie *qat_cookie;
CpaCySymDpOpData *pOpData;
int error;
};
struct qat_ocf_instance {
CpaInstanceHandle cyInstHandle;
struct mtx cyInstMtx;
struct qat_ocf_dma_mem cookie_dmamem[QAT_OCF_MEM_POOL_SIZE];
struct qat_ocf_cookie *cookie_pool[QAT_OCF_MEM_POOL_SIZE];
struct qat_ocf_cookie *free_cookie[QAT_OCF_MEM_POOL_SIZE];
int free_cookie_ptr;
struct mtx cookie_pool_mtx;
int32_t driver_id;
};
/* Init/deinit */
CpaStatus qat_ocf_cookie_pool_init(struct qat_ocf_instance *instance,
device_t dev);
void qat_ocf_cookie_pool_deinit(struct qat_ocf_instance *instance);
/* Alloc/free */
CpaStatus qat_ocf_cookie_alloc(struct qat_ocf_instance *instance,
struct qat_ocf_cookie **buffers_out);
void qat_ocf_cookie_free(struct qat_ocf_instance *instance,
struct qat_ocf_cookie *cookie);
/* Pre/post sync */
CpaStatus qat_ocf_cookie_dma_pre_sync(struct cryptop *crp,
CpaCySymDpOpData *pOpData);
CpaStatus qat_ocf_cookie_dma_post_sync(struct cryptop *crp,
CpaCySymDpOpData *pOpData);
/* Bus DMA unload */
CpaStatus qat_ocf_cookie_dma_unload(struct cryptop *crp,
CpaCySymDpOpData *pOpData);
/* Bus DMA load callbacks */
void qat_ocf_crypto_load_buf_cb(void *_arg,
bus_dma_segment_t *segs,
int nseg,
int error);
void qat_ocf_crypto_load_obuf_cb(void *_arg,
bus_dma_segment_t *segs,
int nseg,
int error);
void qat_ocf_crypto_load_aadbuf_cb(void *_arg,
bus_dma_segment_t *segs,
int nseg,
int error);
#endif /* _QAT_OCF_MEM_POOL_H_ */

61
sys/dev/qat/include/qat_ocf_utils.h Normal file
View File

@ -0,0 +1,61 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
#ifndef _QAT_OCF_UTILS_H_
#define _QAT_OCF_UTILS_H_
/* System headers */
#include <sys/types.h>
#include <sys/mbuf.h>
#include <machine/bus_dma.h>
/* Cryptodev headers */
#include <opencrypto/cryptodev.h>
#include <crypto/sha2/sha512.h>
/* QAT specific headers */
#include "qat_ocf_mem_pool.h"
#include "cpa.h"
#include "cpa_cy_sym_dp.h"
static inline CpaBoolean
is_gmac_exception(const struct crypto_session_params *csp)
{
if (CSP_MODE_DIGEST == csp->csp_mode)
if (CRYPTO_AES_NIST_GMAC == csp->csp_auth_alg)
return CPA_TRUE;
return CPA_FALSE;
}
static inline CpaBoolean
is_sep_aad_supported(const struct crypto_session_params *csp)
{
if (CPA_TRUE == is_gmac_exception(csp))
return CPA_FALSE;
if (CSP_MODE_AEAD == csp->csp_mode)
if (CRYPTO_AES_NIST_GCM_16 == csp->csp_cipher_alg ||
CRYPTO_AES_NIST_GMAC == csp->csp_cipher_alg)
return CPA_TRUE;
return CPA_FALSE;
}
static inline CpaBoolean
is_use_sep_digest(const struct crypto_session_params *csp)
{
/* Use separated digest for all digest/hash operations,
* including GMAC */
if (CSP_MODE_DIGEST == csp->csp_mode || CSP_MODE_ETA == csp->csp_mode)
return CPA_TRUE;
return CPA_FALSE;
}
int qat_ocf_handle_session_update(struct qat_ocf_dsession *ocf_dsession,
struct cryptop *crp);
CpaStatus qat_ocf_wait_for_session(CpaCySymSessionCtx sessionCtx,
Cpa32U timeoutMS);
#endif /* _QAT_OCF_UTILS_H_ */

1228
sys/dev/qat/qat/qat_ocf.c Normal file
View File

File diff suppressed because it is too large Load Diff

564
sys/dev/qat/qat/qat_ocf_mem_pool.c Normal file
View File

@ -0,0 +1,564 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/* System headers */
#include <sys/param.h>
#include <sys/systm.h>
#include <sys/bus.h>
#include <sys/kernel.h>
#include <sys/mbuf.h>
#include <sys/mutex.h>
#include <machine/bus.h>
/* Cryptodev headers */
#include <opencrypto/cryptodev.h>
#include <opencrypto/xform.h>
/* QAT specific headers */
#include "qat_ocf_mem_pool.h"
#include "qat_ocf_utils.h"
#include "cpa.h"
/* Private functions */
static void
qat_ocf_alloc_single_cb(void *arg, bus_dma_segment_t *segs, int nseg, int error)
{
struct qat_ocf_dma_mem *dma_mem;
if (error != 0)
return;
dma_mem = arg;
dma_mem->dma_seg = segs[0];
}
static int
qat_ocf_populate_buf_list_cb(struct qat_ocf_buffer_list *buffers,
bus_dma_segment_t *segs,
int niseg,
int skip_seg,
int skip_bytes)
{
CpaPhysFlatBuffer *flatBuffer;
bus_addr_t segment_addr;
bus_size_t segment_len;
int iseg, oseg;
for (iseg = 0, oseg = skip_seg;
iseg < niseg && oseg < QAT_OCF_MAX_FLATS;
iseg++) {
segment_addr = segs[iseg].ds_addr;
segment_len = segs[iseg].ds_len;
if (skip_bytes > 0) {
if (skip_bytes < segment_len) {
segment_addr += skip_bytes;
segment_len -= skip_bytes;
skip_bytes = 0;
} else {
skip_bytes -= segment_len;
continue;
}
}
flatBuffer = &buffers->flatBuffers[oseg++];
flatBuffer->dataLenInBytes = (Cpa32U)segment_len;
flatBuffer->bufferPhysAddr = (CpaPhysicalAddr)segment_addr;
};
buffers->numBuffers = oseg;
return iseg < niseg ? E2BIG : 0;
}
void
qat_ocf_crypto_load_aadbuf_cb(void *_arg,
bus_dma_segment_t *segs,
int nseg,
int error)
{
struct qat_ocf_load_cb_arg *arg;
struct qat_ocf_cookie *qat_cookie;
arg = _arg;
if (error != 0) {
arg->error = error;
return;
}
qat_cookie = arg->qat_cookie;
arg->error = qat_ocf_populate_buf_list_cb(
&qat_cookie->src_buffers, segs, nseg, 0, 0);
}
void
qat_ocf_crypto_load_buf_cb(void *_arg,
bus_dma_segment_t *segs,
int nseg,
int error)
{
struct qat_ocf_cookie *qat_cookie;
struct qat_ocf_load_cb_arg *arg;
int start_segment = 0, skip_bytes = 0;
arg = _arg;
if (error != 0) {
arg->error = error;
return;
}
qat_cookie = arg->qat_cookie;
skip_bytes = 0;
start_segment = qat_cookie->src_buffers.numBuffers;
arg->error = qat_ocf_populate_buf_list_cb(
&qat_cookie->src_buffers, segs, nseg, start_segment, skip_bytes);
}
void
qat_ocf_crypto_load_obuf_cb(void *_arg,
bus_dma_segment_t *segs,
int nseg,
int error)
{
struct qat_ocf_load_cb_arg *arg;
struct cryptop *crp;
struct qat_ocf_cookie *qat_cookie;
const struct crypto_session_params *csp;
int osegs = 0, to_copy = 0;
arg = _arg;
if (error != 0) {
arg->error = error;
return;
}
crp = arg->crp_op;
qat_cookie = arg->qat_cookie;
csp = crypto_get_params(crp->crp_session);
/*
* The payload must start at the same offset in the output SG list as in
* the input SG list. Copy over SG entries from the input corresponding
* to the AAD buffer.
*/
if (crp->crp_aad_length == 0 ||
(CPA_TRUE == is_sep_aad_supported(csp) && crp->crp_aad)) {
arg->error =
qat_ocf_populate_buf_list_cb(&qat_cookie->dst_buffers,
segs,
nseg,
0,
crp->crp_payload_output_start);
return;
}
/* Copy AAD from source SGL to keep payload in the same position in
* destination buffers */
if (NULL == crp->crp_aad)
to_copy = crp->crp_payload_start - crp->crp_aad_start;
else
to_copy = crp->crp_aad_length;
for (; osegs < qat_cookie->src_buffers.numBuffers; osegs++) {
CpaPhysFlatBuffer *src_flat;
CpaPhysFlatBuffer *dst_flat;
int data_len;
if (to_copy <= 0)
break;
src_flat = &qat_cookie->src_buffers.flatBuffers[osegs];
dst_flat = &qat_cookie->dst_buffers.flatBuffers[osegs];
dst_flat->bufferPhysAddr = src_flat->bufferPhysAddr;
data_len = imin(src_flat->dataLenInBytes, to_copy);
dst_flat->dataLenInBytes = data_len;
to_copy -= data_len;
}
arg->error =
qat_ocf_populate_buf_list_cb(&qat_cookie->dst_buffers,
segs,
nseg,
osegs,
crp->crp_payload_output_start);
}
static int
qat_ocf_alloc_dma_mem(device_t dev,
struct qat_ocf_dma_mem *dma_mem,
int nseg,
bus_size_t size,
bus_size_t alignment)
{
int error;
error = bus_dma_tag_create(bus_get_dma_tag(dev),
alignment,
0, /* alignment, boundary */
BUS_SPACE_MAXADDR, /* lowaddr */
BUS_SPACE_MAXADDR, /* highaddr */
NULL,
NULL, /* filter, filterarg */
size, /* maxsize */
nseg, /* nsegments */
size, /* maxsegsize */
BUS_DMA_COHERENT, /* flags */
NULL,
NULL, /* lockfunc, lockarg */
&dma_mem->dma_tag);
if (error != 0) {
device_printf(dev,
"couldn't create DMA tag, error = %d\n",
error);
return error;
}
error =
bus_dmamem_alloc(dma_mem->dma_tag,
&dma_mem->dma_vaddr,
BUS_DMA_NOWAIT | BUS_DMA_ZERO | BUS_DMA_COHERENT,
&dma_mem->dma_map);
if (error != 0) {
device_printf(dev,
"couldn't allocate dmamem, error = %d\n",
error);
goto fail_0;
}
error = bus_dmamap_load(dma_mem->dma_tag,
dma_mem->dma_map,
dma_mem->dma_vaddr,
size,
qat_ocf_alloc_single_cb,
dma_mem,
BUS_DMA_NOWAIT);
if (error) {
device_printf(dev,
"couldn't load dmamem map, error = %d\n",
error);
goto fail_1;
}
return 0;
fail_1:
bus_dmamem_free(dma_mem->dma_tag, dma_mem->dma_vaddr, dma_mem->dma_map);
fail_0:
bus_dma_tag_destroy(dma_mem->dma_tag);
return error;
}
static void
qat_ocf_free_dma_mem(struct qat_ocf_dma_mem *qdm)
{
if (qdm->dma_tag != NULL && qdm->dma_vaddr != NULL) {
bus_dmamap_unload(qdm->dma_tag, qdm->dma_map);
bus_dmamem_free(qdm->dma_tag, qdm->dma_vaddr, qdm->dma_map);
bus_dma_tag_destroy(qdm->dma_tag);
explicit_bzero(qdm, sizeof(*qdm));
}
}
static int
qat_ocf_dma_tag_and_map(device_t dev,
struct qat_ocf_dma_mem *dma_mem,
bus_size_t size,
bus_size_t segs)
{
int error;
error = bus_dma_tag_create(bus_get_dma_tag(dev),
1,
0, /* alignment, boundary */
BUS_SPACE_MAXADDR, /* lowaddr */
BUS_SPACE_MAXADDR, /* highaddr */
NULL,
NULL, /* filter, filterarg */
size, /* maxsize */
segs, /* nsegments */
size, /* maxsegsize */
BUS_DMA_COHERENT, /* flags */
NULL,
NULL, /* lockfunc, lockarg */
&dma_mem->dma_tag);
if (error != 0)
return error;
error = bus_dmamap_create(dma_mem->dma_tag,
BUS_DMA_COHERENT,
&dma_mem->dma_map);
if (error != 0)
return error;
return 0;
}
static void
qat_ocf_clear_cookie(struct qat_ocf_cookie *qat_cookie)
{
qat_cookie->src_buffers.numBuffers = 0;
qat_cookie->dst_buffers.numBuffers = 0;
qat_cookie->is_sep_aad_used = CPA_FALSE;
explicit_bzero(qat_cookie->qat_ocf_iv_buf,
sizeof(qat_cookie->qat_ocf_iv_buf));
explicit_bzero(qat_cookie->qat_ocf_digest,
sizeof(qat_cookie->qat_ocf_digest));
explicit_bzero(qat_cookie->qat_ocf_gcm_aad,
sizeof(qat_cookie->qat_ocf_gcm_aad));
qat_cookie->crp_op = NULL;
}
/* Public functions */
CpaStatus
qat_ocf_cookie_dma_pre_sync(struct cryptop *crp, CpaCySymDpOpData *pOpData)
{
struct qat_ocf_cookie *qat_cookie;
if (NULL == pOpData->pCallbackTag)
return CPA_STATUS_FAIL;
qat_cookie = (struct qat_ocf_cookie *)pOpData->pCallbackTag;
if (CPA_TRUE == qat_cookie->is_sep_aad_used) {
bus_dmamap_sync(qat_cookie->gcm_aad_dma_mem.dma_tag,
qat_cookie->gcm_aad_dma_mem.dma_map,
BUS_DMASYNC_PREWRITE | BUS_DMASYNC_PREREAD);
}
bus_dmamap_sync(qat_cookie->src_dma_mem.dma_tag,
qat_cookie->src_dma_mem.dma_map,
BUS_DMASYNC_PREWRITE | BUS_DMASYNC_PREREAD);
if (CRYPTO_HAS_OUTPUT_BUFFER(crp)) {
bus_dmamap_sync(qat_cookie->dst_dma_mem.dma_tag,
qat_cookie->dst_dma_mem.dma_map,
BUS_DMASYNC_PREWRITE | BUS_DMASYNC_PREREAD);
}
bus_dmamap_sync(qat_cookie->dma_tag,
qat_cookie->dma_map,
BUS_DMASYNC_PREWRITE | BUS_DMASYNC_PREREAD);
return CPA_STATUS_SUCCESS;
}
CpaStatus
qat_ocf_cookie_dma_post_sync(struct cryptop *crp, CpaCySymDpOpData *pOpData)
{
struct qat_ocf_cookie *qat_cookie;
if (NULL == pOpData->pCallbackTag)
return CPA_STATUS_FAIL;
qat_cookie = (struct qat_ocf_cookie *)pOpData->pCallbackTag;
bus_dmamap_sync(qat_cookie->src_dma_mem.dma_tag,
qat_cookie->src_dma_mem.dma_map,
BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE);
if (CRYPTO_HAS_OUTPUT_BUFFER(crp)) {
bus_dmamap_sync(qat_cookie->dst_dma_mem.dma_tag,
qat_cookie->dst_dma_mem.dma_map,
BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE);
}
bus_dmamap_sync(qat_cookie->dma_tag,
qat_cookie->dma_map,
BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE);
if (qat_cookie->is_sep_aad_used)
bus_dmamap_sync(qat_cookie->gcm_aad_dma_mem.dma_tag,
qat_cookie->gcm_aad_dma_mem.dma_map,
BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE);
return CPA_STATUS_SUCCESS;
}
CpaStatus
qat_ocf_cookie_dma_unload(struct cryptop *crp, CpaCySymDpOpData *pOpData)
{
struct qat_ocf_cookie *qat_cookie;
qat_cookie = pOpData->pCallbackTag;
if (NULL == qat_cookie)
return CPA_STATUS_FAIL;
bus_dmamap_unload(qat_cookie->src_dma_mem.dma_tag,
qat_cookie->src_dma_mem.dma_map);
if (CRYPTO_HAS_OUTPUT_BUFFER(crp))
bus_dmamap_unload(qat_cookie->dst_dma_mem.dma_tag,
qat_cookie->dst_dma_mem.dma_map);
if (qat_cookie->is_sep_aad_used)
bus_dmamap_unload(qat_cookie->gcm_aad_dma_mem.dma_tag,
qat_cookie->gcm_aad_dma_mem.dma_map);
return CPA_STATUS_SUCCESS;
}
CpaStatus
qat_ocf_cookie_pool_init(struct qat_ocf_instance *instance, device_t dev)
{
int i, error = 0;
mtx_init(&instance->cookie_pool_mtx,
"QAT cookie pool MTX",
NULL,
MTX_DEF);
instance->free_cookie_ptr = 0;
for (i = 0; i < QAT_OCF_MEM_POOL_SIZE; i++) {
struct qat_ocf_cookie *qat_cookie;
struct qat_ocf_dma_mem *entry_dma_mem;
entry_dma_mem = &instance->cookie_dmamem[i];
/* Allocate DMA segment for cache entry.
* Cache has to be stored in DMAable mem due to
* it contains i.a src and dst flat buffer
* lists.
*/
error = qat_ocf_alloc_dma_mem(dev,
entry_dma_mem,
1,
sizeof(struct qat_ocf_cookie),
(1 << 6));
if (error)
break;
qat_cookie = entry_dma_mem->dma_vaddr;
instance->cookie_pool[i] = qat_cookie;
qat_cookie->dma_map = entry_dma_mem->dma_map;
qat_cookie->dma_tag = entry_dma_mem->dma_tag;
qat_ocf_clear_cookie(qat_cookie);
/* Physical address of IV buffer */
qat_cookie->qat_ocf_iv_buf_paddr =
entry_dma_mem->dma_seg.ds_addr +
offsetof(struct qat_ocf_cookie, qat_ocf_iv_buf);
/* Physical address of digest buffer */
qat_cookie->qat_ocf_digest_paddr =
entry_dma_mem->dma_seg.ds_addr +
offsetof(struct qat_ocf_cookie, qat_ocf_digest);
/* Physical address of AAD buffer */
qat_cookie->qat_ocf_gcm_aad_paddr =
entry_dma_mem->dma_seg.ds_addr +
offsetof(struct qat_ocf_cookie, qat_ocf_gcm_aad);
/* We already got physical address of src and dest SGL header */
qat_cookie->src_buffer_list_paddr =
entry_dma_mem->dma_seg.ds_addr +
offsetof(struct qat_ocf_cookie, src_buffers);
qat_cookie->dst_buffer_list_paddr =
entry_dma_mem->dma_seg.ds_addr +
offsetof(struct qat_ocf_cookie, dst_buffers);
/* We already have physical address of pOpdata */
qat_cookie->pOpData_paddr = entry_dma_mem->dma_seg.ds_addr +
offsetof(struct qat_ocf_cookie, pOpdata);
/* Init QAT DP API OP data with const values */
qat_cookie->pOpdata.pCallbackTag = (void *)qat_cookie;
qat_cookie->pOpdata.thisPhys =
(CpaPhysicalAddr)qat_cookie->pOpData_paddr;
error = qat_ocf_dma_tag_and_map(dev,
&qat_cookie->src_dma_mem,
QAT_OCF_MAXLEN,
QAT_OCF_MAX_FLATS);
if (error)
break;
error = qat_ocf_dma_tag_and_map(dev,
&qat_cookie->dst_dma_mem,
QAT_OCF_MAXLEN,
QAT_OCF_MAX_FLATS);
if (error)
break;
/* Max one flat buffer for embedded AAD if provided as separated
* by OCF and it's not supported by QAT */
error = qat_ocf_dma_tag_and_map(dev,
&qat_cookie->gcm_aad_dma_mem,
QAT_OCF_MAXLEN,
1);
if (error)
break;
instance->free_cookie[i] = qat_cookie;
instance->free_cookie_ptr++;
}
return error;
}
CpaStatus
qat_ocf_cookie_alloc(struct qat_ocf_instance *qat_instance,
struct qat_ocf_cookie **cookie_out)
{
mtx_lock(&qat_instance->cookie_pool_mtx);
if (qat_instance->free_cookie_ptr == 0) {
mtx_unlock(&qat_instance->cookie_pool_mtx);
return CPA_STATUS_FAIL;
}
*cookie_out =
qat_instance->free_cookie[--qat_instance->free_cookie_ptr];
mtx_unlock(&qat_instance->cookie_pool_mtx);
return CPA_STATUS_SUCCESS;
}
void
qat_ocf_cookie_free(struct qat_ocf_instance *qat_instance,
struct qat_ocf_cookie *cookie)
{
qat_ocf_clear_cookie(cookie);
mtx_lock(&qat_instance->cookie_pool_mtx);
qat_instance->free_cookie[qat_instance->free_cookie_ptr++] = cookie;
mtx_unlock(&qat_instance->cookie_pool_mtx);
}
void
qat_ocf_cookie_pool_deinit(struct qat_ocf_instance *qat_instance)
{
int i;
for (i = 0; i < QAT_OCF_MEM_POOL_SIZE; i++) {
struct qat_ocf_cookie *cookie;
struct qat_ocf_dma_mem *cookie_dma;
cookie = qat_instance->cookie_pool[i];
if (NULL == cookie)
continue;
/* Destroy tag and map for source SGL */
if (cookie->src_dma_mem.dma_tag) {
bus_dmamap_destroy(cookie->src_dma_mem.dma_tag,
cookie->src_dma_mem.dma_map);
bus_dma_tag_destroy(cookie->src_dma_mem.dma_tag);
}
/* Destroy tag and map for dest SGL */
if (cookie->dst_dma_mem.dma_tag) {
bus_dmamap_destroy(cookie->dst_dma_mem.dma_tag,
cookie->dst_dma_mem.dma_map);
bus_dma_tag_destroy(cookie->dst_dma_mem.dma_tag);
}
/* Destroy tag and map for separated AAD */
if (cookie->gcm_aad_dma_mem.dma_tag) {
bus_dmamap_destroy(cookie->gcm_aad_dma_mem.dma_tag,
cookie->gcm_aad_dma_mem.dma_map);
bus_dma_tag_destroy(cookie->gcm_aad_dma_mem.dma_tag);
}
/* Free DMA memory */
cookie_dma = &qat_instance->cookie_dmamem[i];
qat_ocf_free_dma_mem(cookie_dma);
qat_instance->cookie_pool[i] = NULL;
}
mtx_destroy(&qat_instance->cookie_pool_mtx);
return;
}

172
sys/dev/qat/qat/qat_ocf_utils.c Normal file
View File

@ -0,0 +1,172 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/* System headers */
#include <sys/param.h>
#include <sys/systm.h>
#include <sys/bus.h>
#include <sys/kernel.h>
#include <sys/mutex.h>
#include <sys/timespec.h>
/* QAT specific headers */
#include "qat_ocf_utils.h"
#include "cpa.h"
#include "lac_common.h"
#include "lac_log.h"
#include "lac_mem.h"
#include "lac_mem_pools.h"
#include "lac_list.h"
#include "lac_sym.h"
#include "lac_sym_qat.h"
#include "lac_sal.h"
#include "lac_sal_ctrl.h"
#include "lac_session.h"
#include "lac_sym_cipher.h"
#include "lac_sym_hash.h"
#include "lac_sym_alg_chain.h"
#include "lac_sym_stats.h"
#include "lac_sym_partial.h"
#include "lac_sym_qat_hash_defs_lookup.h"
#define QAT_OCF_AAD_NOCHANGE (-1)
CpaStatus
qat_ocf_wait_for_session(CpaCySymSessionCtx sessionCtx, Cpa32U timeoutMS)
{
CpaBoolean sessionInUse = CPA_TRUE;
CpaStatus status;
struct timespec start_ts;
struct timespec current_ts;
struct timespec delta;
u64 delta_ms;
nanotime(&start_ts);
for (;;) {
status = cpaCySymSessionInUse(sessionCtx, &sessionInUse);
if (CPA_STATUS_SUCCESS != status)
return CPA_STATUS_FAIL;
if (CPA_FALSE == sessionInUse)
break;
nanotime(&current_ts);
delta = timespec_sub(current_ts, start_ts);
delta_ms = (delta.tv_sec * 1000) +
(delta.tv_nsec / NSEC_PER_MSEC);
if (delta_ms > (timeoutMS))
return CPA_STATUS_RESOURCE;
qatUtilsYield();
}
return CPA_STATUS_SUCCESS;
}
static CpaStatus
qat_ocf_session_update(struct qat_ocf_session *ocf_session,
Cpa8U *newCipher,
Cpa8U *newAuth,
Cpa32U newAADLength)
{
lac_session_desc_t *pSessionDesc = NULL;
CpaStatus status = CPA_STATUS_SUCCESS;
CpaBoolean sessionInUse = CPA_TRUE;
if (!ocf_session->sessionCtx)
return CPA_STATUS_SUCCESS;
status = cpaCySymSessionInUse(ocf_session->sessionCtx, &sessionInUse);
if (CPA_TRUE == sessionInUse)
return CPA_STATUS_RESOURCE;
pSessionDesc =
LAC_SYM_SESSION_DESC_FROM_CTX_GET(ocf_session->sessionCtx);
if (newAADLength != QAT_OCF_AAD_NOCHANGE) {
ocf_session->aadLen = newAADLength;
status =
LacAlgChain_SessionAADUpdate(pSessionDesc, newAADLength);
if (CPA_STATUS_SUCCESS != status)
return status;
}
if (newCipher) {
status =
LacAlgChain_SessionCipherKeyUpdate(pSessionDesc, newCipher);
if (CPA_STATUS_SUCCESS != status)
return status;
}
if (newAuth) {
status =
LacAlgChain_SessionAuthKeyUpdate(pSessionDesc, newAuth);
if (CPA_STATUS_SUCCESS != status)
return status;
}
return status;
}
CpaStatus
qat_ocf_handle_session_update(struct qat_ocf_dsession *ocf_dsession,
struct cryptop *crp)
{
Cpa32U newAADLength = QAT_OCF_AAD_NOCHANGE;
Cpa8U *cipherKey;
Cpa8U *authKey;
crypto_session_t cses;
const struct crypto_session_params *csp;
CpaStatus status = CPA_STATUS_SUCCESS;
if (!ocf_dsession)
return CPA_STATUS_FAIL;
cses = crp->crp_session;
if (!cses)
return CPA_STATUS_FAIL;
csp = crypto_get_params(cses);
if (!csp)
return CPA_STATUS_FAIL;
cipherKey = crp->crp_cipher_key;
authKey = crp->crp_auth_key;
if (is_sep_aad_supported(csp)) {
/* Determine if AAD has change */
if ((ocf_dsession->encSession.sessionCtx &&
ocf_dsession->encSession.aadLen != crp->crp_aad_length) ||
(ocf_dsession->decSession.sessionCtx &&
ocf_dsession->decSession.aadLen != crp->crp_aad_length)) {
newAADLength = crp->crp_aad_length;
/* Get auth and cipher keys from session if not present
* in the request. Update keys is required to update
* AAD.
*/
if (!authKey)
authKey = csp->csp_auth_key;
if (!cipherKey)
cipherKey = csp->csp_cipher_key;
}
if (!authKey)
authKey = cipherKey;
}
if (crp->crp_cipher_key || crp->crp_auth_key ||
newAADLength != QAT_OCF_AAD_NOCHANGE) {
/* Update encryption session */
status = qat_ocf_session_update(&ocf_dsession->encSession,
cipherKey,
authKey,
newAADLength);
if (CPA_STATUS_SUCCESS != status)
return status;
/* Update decryption session */
status = qat_ocf_session_update(&ocf_dsession->decSession,
cipherKey,
authKey,
newAADLength);
if (CPA_STATUS_SUCCESS != status)
return status;
}
return status;
}

116
sys/dev/qat/qat_api/common/compression/dc_buffers.c Normal file
View File

@ -0,0 +1,116 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file dc_buffers.c
*
* @defgroup Dc_DataCompression DC Data Compression
*
* @ingroup Dc_DataCompression
*
* @description
* Implementation of the buffer management operations for
* Data Compression service.
*
*****************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "cpa_dc.h"
#include "cpa_dc_bp.h"
#include "sal_types_compression.h"
#include "icp_qat_fw_comp.h"
#define CPA_DC_CEIL_DIV(x, y) (((x) + (y)-1) / (y))
#define DC_DEST_BUFF_EXTRA_DEFLATE_GEN2 (55)
CpaStatus
cpaDcBufferListGetMetaSize(const CpaInstanceHandle instanceHandle,
Cpa32U numBuffers,
Cpa32U *pSizeInBytes)
{
CpaInstanceHandle insHandle = NULL;
if (CPA_INSTANCE_HANDLE_SINGLE == instanceHandle) {
insHandle = dcGetFirstHandle();
} else {
insHandle = instanceHandle;
}
LAC_CHECK_INSTANCE_HANDLE(insHandle);
LAC_CHECK_NULL_PARAM(pSizeInBytes);
/* Ensure this is a compression instance */
SAL_CHECK_INSTANCE_TYPE(insHandle, SAL_SERVICE_TYPE_COMPRESSION);
if (0 == numBuffers) {
QAT_UTILS_LOG("Number of buffers is 0.\n");
return CPA_STATUS_INVALID_PARAM;
}
*pSizeInBytes = (sizeof(icp_buffer_list_desc_t) +
(sizeof(icp_flat_buffer_desc_t) * (numBuffers + 1)) +
ICP_DESCRIPTOR_ALIGNMENT_BYTES);
return CPA_STATUS_SUCCESS;
}
CpaStatus
cpaDcBnpBufferListGetMetaSize(const CpaInstanceHandle instanceHandle,
Cpa32U numJobs,
Cpa32U *pSizeInBytes)
{
return CPA_STATUS_UNSUPPORTED;
}
static inline CpaStatus
dcDeflateBoundGen2(CpaDcHuffType huffType, Cpa32U inputSize, Cpa32U *outputSize)
{
/* Formula for GEN2 deflate:
* ceil(9 * Total input bytes / 8) + 55 bytes.
* 55 bytes is the skid pad value for GEN2 devices.
*/
*outputSize =
CPA_DC_CEIL_DIV(9 * inputSize, 8) + DC_DEST_BUFF_EXTRA_DEFLATE_GEN2;
return CPA_STATUS_SUCCESS;
}
CpaStatus
cpaDcDeflateCompressBound(const CpaInstanceHandle dcInstance,
CpaDcHuffType huffType,
Cpa32U inputSize,
Cpa32U *outputSize)
{
CpaInstanceHandle insHandle = NULL;
if (CPA_INSTANCE_HANDLE_SINGLE == dcInstance) {
insHandle = dcGetFirstHandle();
} else {
insHandle = dcInstance;
}
LAC_CHECK_INSTANCE_HANDLE(insHandle);
LAC_CHECK_NULL_PARAM(outputSize);
/* Ensure this is a compression instance */
SAL_CHECK_INSTANCE_TYPE(insHandle, SAL_SERVICE_TYPE_COMPRESSION);
if (!inputSize) {
QAT_UTILS_LOG(
"The input size needs to be greater than zero.\n");
return CPA_STATUS_INVALID_PARAM;
}
if ((CPA_DC_HT_STATIC != huffType) &&
(CPA_DC_HT_FULL_DYNAMIC != huffType)) {
QAT_UTILS_LOG("Invalid huffType value.\n");
return CPA_STATUS_INVALID_PARAM;
}
return dcDeflateBoundGen2(huffType, inputSize, outputSize);
}

1790
sys/dev/qat/qat_api/common/compression/dc_datapath.c Normal file
View File

File diff suppressed because it is too large Load Diff

545
sys/dev/qat/qat_api/common/compression/dc_dp.c Normal file
View File

@ -0,0 +1,545 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file dc_dp.c
*
* @defgroup cpaDcDp Data Compression Data Plane API
*
* @ingroup cpaDcDp
*
* @description
* Implementation of the Data Compression DP operations.
*
*****************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "cpa_dc.h"
#include "cpa_dc_dp.h"
#include "icp_qat_fw_comp.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "dc_session.h"
#include "dc_datapath.h"
#include "lac_common.h"
#include "lac_mem.h"
#include "lac_mem_pools.h"
#include "sal_types_compression.h"
#include "lac_sal.h"
#include "lac_sync.h"
#include "sal_service_state.h"
#include "sal_qat_cmn_msg.h"
#include "icp_sal_poll.h"
/**
*****************************************************************************
* @ingroup cpaDcDp
* Check that pOpData is valid
*
* @description
* Check that all the parameters defined in the pOpData are valid
*
* @param[in] pOpData Pointer to a structure containing the
* request parameters
*
* @retval CPA_STATUS_SUCCESS Function executed successfully
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter passed in
*
*****************************************************************************/
static CpaStatus
dcDataPlaneParamCheck(const CpaDcDpOpData *pOpData)
{
sal_compression_service_t *pService = NULL;
dc_session_desc_t *pSessionDesc = NULL;
LAC_CHECK_NULL_PARAM(pOpData);
LAC_CHECK_NULL_PARAM(pOpData->dcInstance);
LAC_CHECK_NULL_PARAM(pOpData->pSessionHandle);
/* Ensure this is a compression instance */
SAL_CHECK_INSTANCE_TYPE(pOpData->dcInstance,
SAL_SERVICE_TYPE_COMPRESSION);
pService = (sal_compression_service_t *)(pOpData->dcInstance);
pSessionDesc = DC_SESSION_DESC_FROM_CTX_GET(pOpData->pSessionHandle);
if (NULL == pSessionDesc) {
QAT_UTILS_LOG("Session handle not as expected.\n");
return CPA_STATUS_INVALID_PARAM;
}
if (CPA_FALSE == pSessionDesc->isDcDp) {
QAT_UTILS_LOG("The session type should be data plane.\n");
return CPA_STATUS_INVALID_PARAM;
}
/* Compressing zero byte is not supported */
if ((CPA_DC_DIR_COMPRESS == pSessionDesc->sessDirection) &&
(0 == pOpData->bufferLenToCompress)) {
QAT_UTILS_LOG(
"The source buffer length to compress needs to be greater than zero byte.\n");
return CPA_STATUS_INVALID_PARAM;
}
if (pOpData->sessDirection > CPA_DC_DIR_DECOMPRESS) {
QAT_UTILS_LOG("Invalid direction of operation.\n");
return CPA_STATUS_INVALID_PARAM;
}
if (0 == pOpData->srcBuffer) {
QAT_UTILS_LOG("Invalid srcBuffer\n");
return CPA_STATUS_INVALID_PARAM;
}
if (0 == pOpData->destBuffer) {
QAT_UTILS_LOG("Invalid destBuffer\n");
return CPA_STATUS_INVALID_PARAM;
}
if (pOpData->srcBuffer == pOpData->destBuffer) {
QAT_UTILS_LOG("In place operation is not supported.\n");
return CPA_STATUS_INVALID_PARAM;
}
if (0 == pOpData->thisPhys) {
QAT_UTILS_LOG("Invalid thisPhys\n");
return CPA_STATUS_INVALID_PARAM;
}
if ((CPA_TRUE != pOpData->compressAndVerify) &&
(CPA_FALSE != pOpData->compressAndVerify)) {
QAT_UTILS_LOG("Invalid compressAndVerify\n");
return CPA_STATUS_INVALID_PARAM;
}
if ((CPA_TRUE == pOpData->compressAndVerify) &&
!(pService->generic_service_info.dcExtendedFeatures &
DC_CNV_EXTENDED_CAPABILITY)) {
QAT_UTILS_LOG("Invalid compressAndVerify, no CNV capability\n");
return CPA_STATUS_UNSUPPORTED;
}
if ((CPA_TRUE != pOpData->compressAndVerifyAndRecover) &&
(CPA_FALSE != pOpData->compressAndVerifyAndRecover)) {
QAT_UTILS_LOG("Invalid compressAndVerifyAndRecover\n");
return CPA_STATUS_INVALID_PARAM;
}
if ((CPA_TRUE == pOpData->compressAndVerifyAndRecover) &&
(CPA_FALSE == pOpData->compressAndVerify)) {
QAT_UTILS_LOG("CnVnR option set without setting CnV\n");
return CPA_STATUS_INVALID_PARAM;
}
if ((CPA_TRUE == pOpData->compressAndVerifyAndRecover) &&
!(pService->generic_service_info.dcExtendedFeatures &
DC_CNVNR_EXTENDED_CAPABILITY)) {
QAT_UTILS_LOG(
"Invalid CnVnR option set and no CnVnR capability.\n");
return CPA_STATUS_UNSUPPORTED;
}
if ((CPA_DP_BUFLIST == pOpData->srcBufferLen) &&
(CPA_DP_BUFLIST != pOpData->destBufferLen)) {
QAT_UTILS_LOG(
"The source and destination buffers need to be of the same type (both flat buffers or buffer lists).\n");
return CPA_STATUS_INVALID_PARAM;
}
if ((CPA_DP_BUFLIST != pOpData->srcBufferLen) &&
(CPA_DP_BUFLIST == pOpData->destBufferLen)) {
QAT_UTILS_LOG(
"The source and destination buffers need to be of the same type (both flat buffers or buffer lists).\n");
return CPA_STATUS_INVALID_PARAM;
}
if (CPA_DP_BUFLIST != pOpData->srcBufferLen) {
if (pOpData->srcBufferLen < pOpData->bufferLenToCompress) {
QAT_UTILS_LOG(
"srcBufferLen is smaller than bufferLenToCompress.\n");
return CPA_STATUS_INVALID_PARAM;
}
if (pOpData->destBufferLen < pOpData->bufferLenForData) {
QAT_UTILS_LOG(
"destBufferLen is smaller than bufferLenForData.\n");
return CPA_STATUS_INVALID_PARAM;
}
} else {
/* We are assuming that there is enough memory in the source and
* destination buffer lists. We only receive physical addresses
* of the
* buffers so we are unable to test it here */
LAC_CHECK_8_BYTE_ALIGNMENT(pOpData->srcBuffer);
LAC_CHECK_8_BYTE_ALIGNMENT(pOpData->destBuffer);
}
LAC_CHECK_8_BYTE_ALIGNMENT(pOpData->thisPhys);
if ((CPA_DC_DIR_COMPRESS == pSessionDesc->sessDirection) ||
(CPA_DC_DIR_COMBINED == pSessionDesc->sessDirection)) {
if (CPA_DC_HT_FULL_DYNAMIC == pSessionDesc->huffType) {
/* Check if Intermediate Buffer Array pointer is NULL */
if ((0 == pService->pInterBuffPtrsArrayPhyAddr) ||
(NULL == pService->pInterBuffPtrsArray)) {
QAT_UTILS_LOG(
"No intermediate buffer defined for this instance - see cpaDcStartInstance.\n");
return CPA_STATUS_INVALID_PARAM;
}
/* Ensure that the destination buffer length for data is
* greater
* or equal to 128B */
if (pOpData->bufferLenForData <
DC_DEST_BUFFER_DYN_MIN_SIZE) {
QAT_UTILS_LOG(
"Destination buffer length for data should be greater or equal to 128B.\n");
return CPA_STATUS_INVALID_PARAM;
}
} else {
/* Ensure that the destination buffer length for data is
* greater
* or equal to min output buffsize */
if (pOpData->bufferLenForData <
pService->comp_device_data.minOutputBuffSize) {
QAT_UTILS_LOG(
"Destination buffer size should be greater or equal to %d bytes.\n",
pService->comp_device_data
.minOutputBuffSize);
return CPA_STATUS_INVALID_PARAM;
}
}
}
return CPA_STATUS_SUCCESS;
}
CpaStatus
cpaDcDpGetSessionSize(CpaInstanceHandle dcInstance,
CpaDcSessionSetupData *pSessionData,
Cpa32U *pSessionSize)
{
return dcGetSessionSize(dcInstance, pSessionData, pSessionSize, NULL);
}
CpaStatus
cpaDcDpInitSession(CpaInstanceHandle dcInstance,
CpaDcSessionHandle pSessionHandle,
CpaDcSessionSetupData *pSessionData)
{
CpaStatus status = CPA_STATUS_SUCCESS;
dc_session_desc_t *pSessionDesc = NULL;
sal_compression_service_t *pService = NULL;
LAC_CHECK_INSTANCE_HANDLE(dcInstance);
SAL_CHECK_INSTANCE_TYPE(dcInstance, SAL_SERVICE_TYPE_COMPRESSION);
pService = (sal_compression_service_t *)dcInstance;
/* Check if SAL is initialised otherwise return an error */
SAL_RUNNING_CHECK(pService);
/* Stateful is not supported */
if (CPA_DC_STATELESS != pSessionData->sessState) {
QAT_UTILS_LOG("Invalid sessState value\n");
return CPA_STATUS_INVALID_PARAM;
}
status =
dcInitSession(dcInstance, pSessionHandle, pSessionData, NULL, NULL);
if (CPA_STATUS_SUCCESS == status) {
pSessionDesc = DC_SESSION_DESC_FROM_CTX_GET(pSessionHandle);
pSessionDesc->isDcDp = CPA_TRUE;
ICP_QAT_FW_COMN_PTR_TYPE_SET(
pSessionDesc->reqCacheDecomp.comn_hdr.comn_req_flags,
DC_DP_QAT_PTR_TYPE);
ICP_QAT_FW_COMN_PTR_TYPE_SET(
pSessionDesc->reqCacheComp.comn_hdr.comn_req_flags,
DC_DP_QAT_PTR_TYPE);
}
return status;
}
CpaStatus
cpaDcDpRemoveSession(const CpaInstanceHandle dcInstance,
CpaDcSessionHandle pSessionHandle)
{
return cpaDcRemoveSession(dcInstance, pSessionHandle);
}
CpaStatus
cpaDcDpRegCbFunc(const CpaInstanceHandle dcInstance,
const CpaDcDpCallbackFn pNewCb)
{
sal_compression_service_t *pService = NULL;
LAC_CHECK_NULL_PARAM(dcInstance);
SAL_CHECK_INSTANCE_TYPE(dcInstance, SAL_SERVICE_TYPE_COMPRESSION);
LAC_CHECK_NULL_PARAM(pNewCb);
/* Check if SAL is initialised otherwise return an error */
SAL_RUNNING_CHECK(dcInstance);
pService = (sal_compression_service_t *)dcInstance;
pService->pDcDpCb = pNewCb;
return CPA_STATUS_SUCCESS;
}
/**
*****************************************************************************
* @ingroup cpaDcDp
*
* @description
* Writes the message to the ring
*
* @param[in] pOpData Pointer to a structure containing the
* request parameters
* @param[in] pCurrentQatMsg Pointer to current QAT message on the ring
*
*****************************************************************************/
static void
dcDpWriteRingMsg(CpaDcDpOpData *pOpData, icp_qat_fw_comp_req_t *pCurrentQatMsg)
{
icp_qat_fw_comp_req_t *pReqCache = NULL;
dc_session_desc_t *pSessionDesc = NULL;
Cpa8U bufferFormat;
Cpa8U cnvDecompReq = ICP_QAT_FW_COMP_NO_CNV;
Cpa8U cnvnrCompReq = ICP_QAT_FW_COMP_NO_CNV_RECOVERY;
pSessionDesc = DC_SESSION_DESC_FROM_CTX_GET(pOpData->pSessionHandle);
if (CPA_DC_DIR_COMPRESS == pOpData->sessDirection) {
pReqCache = &(pSessionDesc->reqCacheComp);
/* CNV check */
if (CPA_TRUE == pOpData->compressAndVerify) {
cnvDecompReq = ICP_QAT_FW_COMP_CNV;
/* CNVNR check */
if (CPA_TRUE == pOpData->compressAndVerifyAndRecover) {
cnvnrCompReq = ICP_QAT_FW_COMP_CNV_RECOVERY;
}
}
} else {
pReqCache = &(pSessionDesc->reqCacheDecomp);
}
/* Fills in the template DC ET ring message - cached from the
* session descriptor */
memcpy((void *)pCurrentQatMsg,
(void *)(pReqCache),
(LAC_QAT_DC_REQ_SZ_LW * LAC_LONG_WORD_IN_BYTES));
if (CPA_DP_BUFLIST == pOpData->srcBufferLen) {
bufferFormat = QAT_COMN_PTR_TYPE_SGL;
} else {
bufferFormat = QAT_COMN_PTR_TYPE_FLAT;
}
pCurrentQatMsg->comp_pars.req_par_flags |=
ICP_QAT_FW_COMP_REQ_PARAM_FLAGS_BUILD(
0, 0, 0, cnvDecompReq, cnvnrCompReq, 0);
SalQatMsg_CmnMidWrite((icp_qat_fw_la_bulk_req_t *)pCurrentQatMsg,
pOpData,
bufferFormat,
pOpData->srcBuffer,
pOpData->destBuffer,
pOpData->srcBufferLen,
pOpData->destBufferLen);
pCurrentQatMsg->comp_pars.comp_len = pOpData->bufferLenToCompress;
pCurrentQatMsg->comp_pars.out_buffer_sz = pOpData->bufferLenForData;
}
CpaStatus
cpaDcDpEnqueueOp(CpaDcDpOpData *pOpData, const CpaBoolean performOpNow)
{
icp_qat_fw_comp_req_t *pCurrentQatMsg = NULL;
icp_comms_trans_handle trans_handle = NULL;
dc_session_desc_t *pSessionDesc = NULL;
CpaStatus status = CPA_STATUS_SUCCESS;
status = dcDataPlaneParamCheck(pOpData);
if (CPA_STATUS_SUCCESS != status) {
return status;
}
if ((CPA_FALSE == pOpData->compressAndVerify) &&
(CPA_DC_DIR_COMPRESS == pOpData->sessDirection)) {
return CPA_STATUS_UNSUPPORTED;
}
/* Check if SAL is initialised otherwise return an error */
SAL_RUNNING_CHECK(pOpData->dcInstance);
trans_handle = ((sal_compression_service_t *)pOpData->dcInstance)
->trans_handle_compression_tx;
pSessionDesc = DC_SESSION_DESC_FROM_CTX_GET(pOpData->pSessionHandle);
if ((CPA_DC_DIR_COMPRESS == pOpData->sessDirection) &&
(CPA_DC_DIR_DECOMPRESS == pSessionDesc->sessDirection)) {
QAT_UTILS_LOG(
"The session does not support this direction of operation.\n");
return CPA_STATUS_INVALID_PARAM;
} else if ((CPA_DC_DIR_DECOMPRESS == pOpData->sessDirection) &&
(CPA_DC_DIR_COMPRESS == pSessionDesc->sessDirection)) {
QAT_UTILS_LOG(
"The session does not support this direction of operation.\n");
return CPA_STATUS_INVALID_PARAM;
}
icp_adf_getSingleQueueAddr(trans_handle, (void **)&pCurrentQatMsg);
if (NULL == pCurrentQatMsg) {
return CPA_STATUS_RETRY;
}
dcDpWriteRingMsg(pOpData, pCurrentQatMsg);
pSessionDesc->pendingDpStatelessCbCount++;
if (CPA_TRUE == performOpNow) {
SalQatMsg_updateQueueTail(trans_handle);
}
return CPA_STATUS_SUCCESS;
}
CpaStatus
cpaDcDpEnqueueOpBatch(const Cpa32U numberRequests,
CpaDcDpOpData *pOpData[],
const CpaBoolean performOpNow)
{
icp_qat_fw_comp_req_t *pCurrentQatMsg = NULL;
icp_comms_trans_handle trans_handle = NULL;
dc_session_desc_t *pSessionDesc = NULL;
Cpa32U i = 0;
CpaStatus status = CPA_STATUS_SUCCESS;
sal_compression_service_t *pService = NULL;
LAC_CHECK_NULL_PARAM(pOpData);
LAC_CHECK_NULL_PARAM(pOpData[0]);
LAC_CHECK_NULL_PARAM(pOpData[0]->dcInstance);
pService = (sal_compression_service_t *)(pOpData[0]->dcInstance);
if ((numberRequests == 0) ||
(numberRequests > pService->maxNumCompConcurrentReq)) {
QAT_UTILS_LOG(
"The number of requests needs to be between 1 and %d.\n",
pService->maxNumCompConcurrentReq);
return CPA_STATUS_INVALID_PARAM;
}
for (i = 0; i < numberRequests; i++) {
status = dcDataPlaneParamCheck(pOpData[i]);
if (CPA_STATUS_SUCCESS != status) {
return status;
}
/* Check that all instance handles and session handles are the
* same */
if (pOpData[i]->dcInstance != pOpData[0]->dcInstance) {
QAT_UTILS_LOG(
"All instance handles should be the same in the pOpData.\n");
return CPA_STATUS_INVALID_PARAM;
}
if (pOpData[i]->pSessionHandle != pOpData[0]->pSessionHandle) {
QAT_UTILS_LOG(
"All session handles should be the same in the pOpData.\n");
return CPA_STATUS_INVALID_PARAM;
}
}
for (i = 0; i < numberRequests; i++) {
if ((CPA_FALSE == pOpData[i]->compressAndVerify) &&
(CPA_DC_DIR_COMPRESS == pOpData[i]->sessDirection)) {
return CPA_STATUS_UNSUPPORTED;
}
}
/* Check if SAL is initialised otherwise return an error */
SAL_RUNNING_CHECK(pOpData[0]->dcInstance);
trans_handle = ((sal_compression_service_t *)pOpData[0]->dcInstance)
->trans_handle_compression_tx;
pSessionDesc = DC_SESSION_DESC_FROM_CTX_GET(pOpData[0]->pSessionHandle);
for (i = 0; i < numberRequests; i++) {
if ((CPA_DC_DIR_COMPRESS == pOpData[i]->sessDirection) &&
(CPA_DC_DIR_DECOMPRESS == pSessionDesc->sessDirection)) {
QAT_UTILS_LOG(
"The session does not support this direction of operation.\n");
return CPA_STATUS_INVALID_PARAM;
} else if ((CPA_DC_DIR_DECOMPRESS ==
pOpData[i]->sessDirection) &&
(CPA_DC_DIR_COMPRESS ==
pSessionDesc->sessDirection)) {
QAT_UTILS_LOG(
"The session does not support this direction of operation.\n");
return CPA_STATUS_INVALID_PARAM;
}
}
icp_adf_getQueueMemory(trans_handle,
numberRequests,
(void **)&pCurrentQatMsg);
if (NULL == pCurrentQatMsg) {
return CPA_STATUS_RETRY;
}
for (i = 0; i < numberRequests; i++) {
dcDpWriteRingMsg(pOpData[i], pCurrentQatMsg);
icp_adf_getQueueNext(trans_handle, (void **)&pCurrentQatMsg);
}
pSessionDesc->pendingDpStatelessCbCount += numberRequests;
if (CPA_TRUE == performOpNow) {
SalQatMsg_updateQueueTail(trans_handle);
}
return CPA_STATUS_SUCCESS;
}
CpaStatus
icp_sal_DcPollDpInstance(CpaInstanceHandle dcInstance, Cpa32U responseQuota)
{
icp_comms_trans_handle trans_handle = NULL;
LAC_CHECK_INSTANCE_HANDLE(dcInstance);
SAL_CHECK_INSTANCE_TYPE(dcInstance, SAL_SERVICE_TYPE_COMPRESSION);
/* Check if SAL is initialised otherwise return an error */
SAL_RUNNING_CHECK(dcInstance);
trans_handle = ((sal_compression_service_t *)dcInstance)
->trans_handle_compression_rx;
return icp_adf_pollQueue(trans_handle, responseQuota);
}
CpaStatus
cpaDcDpPerformOpNow(CpaInstanceHandle dcInstance)
{
icp_comms_trans_handle trans_handle = NULL;
LAC_CHECK_NULL_PARAM(dcInstance);
SAL_CHECK_INSTANCE_TYPE(dcInstance, SAL_SERVICE_TYPE_COMPRESSION);
/* Check if SAL is initialised otherwise return an error */
SAL_RUNNING_CHECK(dcInstance);
trans_handle = ((sal_compression_service_t *)dcInstance)
->trans_handle_compression_tx;
if (CPA_TRUE == icp_adf_queueDataToSend(trans_handle)) {
SalQatMsg_updateQueueTail(trans_handle);
}
return CPA_STATUS_SUCCESS;
}

237
sys/dev/qat/qat_api/common/compression/dc_header_footer.c Normal file
View File

@ -0,0 +1,237 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file dc_header_footer.c
*
* @ingroup Dc_DataCompression
*
* @description
* Implementation of the Data Compression header and footer operations.
*
*****************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "cpa_dc.h"
#include "icp_adf_init.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "dc_header_footer.h"
#include "dc_session.h"
#include "dc_datapath.h"
CpaStatus
cpaDcGenerateHeader(CpaDcSessionHandle pSessionHandle,
CpaFlatBuffer *pDestBuff,
Cpa32U *count)
{
dc_session_desc_t *pSessionDesc = NULL;
LAC_CHECK_NULL_PARAM(pSessionHandle);
LAC_CHECK_NULL_PARAM(pDestBuff);
LAC_CHECK_NULL_PARAM(pDestBuff->pData);
LAC_CHECK_NULL_PARAM(count);
pSessionDesc = DC_SESSION_DESC_FROM_CTX_GET(pSessionHandle);
if (NULL == pSessionDesc) {
QAT_UTILS_LOG("Session handle not as expected\n");
return CPA_STATUS_INVALID_PARAM;
}
if (CPA_DC_DIR_DECOMPRESS == pSessionDesc->sessDirection) {
QAT_UTILS_LOG("Invalid session direction\n");
return CPA_STATUS_INVALID_PARAM;
}
if (CPA_DC_DEFLATE == pSessionDesc->compType) {
/* Adding a Gzip header */
if (CPA_DC_CRC32 == pSessionDesc->checksumType) {
Cpa8U *pDest = pDestBuff->pData;
if (pDestBuff->dataLenInBytes < DC_GZIP_HEADER_SIZE) {
QAT_UTILS_LOG(
"The dataLenInBytes of the dest buffer is too small.\n");
return CPA_STATUS_INVALID_PARAM;
}
pDest[0] = DC_GZIP_ID1; /* ID1 */
pDest[1] = DC_GZIP_ID2; /* ID2 */
pDest[2] =
0x08; /* CM = 8 denotes "deflate" compression */
pDest[3] = 0x00; /* FLG = 0 denotes "No extra fields" */
pDest[4] = 0x00;
pDest[5] = 0x00;
pDest[6] = 0x00;
pDest[7] = 0x00; /* MTIME = 0x00 means time stamp not
available */
/* XFL = 4 - compressor used fastest compression, */
/* XFL = 2 - compressor used maximum compression. */
pDest[8] = 0;
if (CPA_DC_L1 == pSessionDesc->compLevel)
pDest[8] = DC_GZIP_FAST_COMP;
else if (CPA_DC_L4 >= pSessionDesc->compLevel)
pDest[8] = DC_GZIP_MAX_COMP;
pDest[9] =
DC_GZIP_FILESYSTYPE; /* OS = 0 means FAT filesystem
(MS-DOS, OS/2, NT/Win32), 3 - Unix */
/* Set to the number of bytes added to the buffer */
*count = DC_GZIP_HEADER_SIZE;
}
/* Adding a Zlib header */
else if (CPA_DC_ADLER32 == pSessionDesc->checksumType) {
Cpa8U *pDest = pDestBuff->pData;
Cpa16U header = 0, level = 0;
if (pDestBuff->dataLenInBytes < DC_ZLIB_HEADER_SIZE) {
QAT_UTILS_LOG(
"The dataLenInBytes of the dest buffer is too small.\n");
return CPA_STATUS_INVALID_PARAM;
}
/* CMF = CM | CMINFO.
CM = 8 denotes "deflate" compression,
CMINFO = 7 indicates a 32K window size */
/* Depending on the device, at compression levels above
L1, the
window size can be 8 or 16K bytes.
The file will decompress ok if a greater window size
is specified
in the header. */
header =
(DC_ZLIB_CM_DEFLATE +
(DC_32K_WINDOW_SIZE << DC_ZLIB_WINDOWSIZE_OFFSET))
<< LAC_NUM_BITS_IN_BYTE;
switch (pSessionDesc->compLevel) {
case CPA_DC_L1:
level = DC_ZLIB_LEVEL_0;
break;
case CPA_DC_L2:
level = DC_ZLIB_LEVEL_1;
break;
case CPA_DC_L3:
level = DC_ZLIB_LEVEL_2;
break;
default:
level = DC_ZLIB_LEVEL_3;
}
/* Bits 6 - 7: FLEVEL, compression level */
header |= level << DC_ZLIB_FLEVEL_OFFSET;
/* The header has to be a multiple of 31 */
header += DC_ZLIB_HEADER_OFFSET -
(header % DC_ZLIB_HEADER_OFFSET);
pDest[0] = (Cpa8U)(header >> LAC_NUM_BITS_IN_BYTE);
pDest[1] = (Cpa8U)header;
/* Set to the number of bytes added to the buffer */
*count = DC_ZLIB_HEADER_SIZE;
}
/* If deflate but no checksum required */
else {
*count = 0;
}
} else {
/* There is no header for other compressed data */
*count = 0;
}
return CPA_STATUS_SUCCESS;
}
CpaStatus
cpaDcGenerateFooter(CpaDcSessionHandle pSessionHandle,
CpaFlatBuffer *pDestBuff,
CpaDcRqResults *pRes)
{
dc_session_desc_t *pSessionDesc = NULL;
LAC_CHECK_NULL_PARAM(pSessionHandle);
LAC_CHECK_NULL_PARAM(pDestBuff);
LAC_CHECK_NULL_PARAM(pDestBuff->pData);
LAC_CHECK_NULL_PARAM(pRes);
pSessionDesc = DC_SESSION_DESC_FROM_CTX_GET(pSessionHandle);
if (NULL == pSessionDesc) {
QAT_UTILS_LOG("Session handle not as expected\n");
return CPA_STATUS_INVALID_PARAM;
}
if (CPA_DC_DIR_DECOMPRESS == pSessionDesc->sessDirection) {
QAT_UTILS_LOG("Invalid session direction\n");
return CPA_STATUS_INVALID_PARAM;
}
if (CPA_DC_DEFLATE == pSessionDesc->compType) {
if (CPA_DC_CRC32 == pSessionDesc->checksumType) {
Cpa8U *pDest = pDestBuff->pData;
Cpa32U crc32 = pRes->checksum;
Cpa64U totalLenBeforeCompress =
pSessionDesc->cumulativeConsumedBytes;
if (pDestBuff->dataLenInBytes < DC_GZIP_FOOTER_SIZE) {
QAT_UTILS_LOG(
"The dataLenInBytes of the dest buffer is too small.\n");
return CPA_STATUS_INVALID_PARAM;
}
/* Crc32 of the uncompressed data */
pDest[0] = (Cpa8U)crc32;
pDest[1] = (Cpa8U)(crc32 >> LAC_NUM_BITS_IN_BYTE);
pDest[2] = (Cpa8U)(crc32 >> 2 * LAC_NUM_BITS_IN_BYTE);
pDest[3] = (Cpa8U)(crc32 >> 3 * LAC_NUM_BITS_IN_BYTE);
/* Length of the uncompressed data */
pDest[4] = (Cpa8U)totalLenBeforeCompress;
pDest[5] = (Cpa8U)(totalLenBeforeCompress >>
LAC_NUM_BITS_IN_BYTE);
pDest[6] = (Cpa8U)(totalLenBeforeCompress >>
2 * LAC_NUM_BITS_IN_BYTE);
pDest[7] = (Cpa8U)(totalLenBeforeCompress >>
3 * LAC_NUM_BITS_IN_BYTE);
/* Increment produced by the number of bytes added to
* the buffer */
pRes->produced += DC_GZIP_FOOTER_SIZE;
} else if (CPA_DC_ADLER32 == pSessionDesc->checksumType) {
Cpa8U *pDest = pDestBuff->pData;
Cpa32U adler32 = pRes->checksum;
if (pDestBuff->dataLenInBytes < DC_ZLIB_FOOTER_SIZE) {
QAT_UTILS_LOG(
"The dataLenInBytes of the dest buffer is too small.\n");
return CPA_STATUS_INVALID_PARAM;
}
/* Adler32 of the uncompressed data */
pDest[0] = (Cpa8U)(adler32 >> 3 * LAC_NUM_BITS_IN_BYTE);
pDest[1] = (Cpa8U)(adler32 >> 2 * LAC_NUM_BITS_IN_BYTE);
pDest[2] = (Cpa8U)(adler32 >> LAC_NUM_BITS_IN_BYTE);
pDest[3] = (Cpa8U)adler32;
/* Increment produced by the number of bytes added to
* the buffer */
pRes->produced += DC_ZLIB_FOOTER_SIZE;
}
}
return CPA_STATUS_SUCCESS;
}

957
sys/dev/qat/qat_api/common/compression/dc_session.c Normal file
View File

@ -0,0 +1,957 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file dc_session.c
*
* @ingroup Dc_DataCompression
*
* @description
* Implementation of the Data Compression session operations.
*
*****************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "cpa_dc.h"
#include "icp_qat_fw.h"
#include "icp_qat_fw_comp.h"
#include "icp_qat_hw.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "dc_session.h"
#include "dc_datapath.h"
#include "lac_mem_pools.h"
#include "sal_types_compression.h"
#include "lac_buffer_desc.h"
#include "sal_service_state.h"
#include "sal_qat_cmn_msg.h"
/**
*****************************************************************************
* @ingroup Dc_DataCompression
* Check that pSessionData is valid
*
* @description
* Check that all the parameters defined in the pSessionData are valid
*
* @param[in] pSessionData Pointer to a user instantiated structure
* containing session data
*
* @retval CPA_STATUS_SUCCESS Function executed successfully
* @retval CPA_STATUS_FAIL Function failed to find device
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter passed in
* @retval CPA_STATUS_UNSUPPORTED Unsupported algorithm/feature
*
*****************************************************************************/
static CpaStatus
dcCheckSessionData(const CpaDcSessionSetupData *pSessionData,
CpaInstanceHandle dcInstance)
{
CpaDcInstanceCapabilities instanceCapabilities = { 0 };
cpaDcQueryCapabilities(dcInstance, &instanceCapabilities);
if ((pSessionData->compLevel < CPA_DC_L1) ||
(pSessionData->compLevel > CPA_DC_L9)) {
QAT_UTILS_LOG("Invalid compLevel value\n");
return CPA_STATUS_INVALID_PARAM;
}
if ((pSessionData->autoSelectBestHuffmanTree < CPA_DC_ASB_DISABLED) ||
(pSessionData->autoSelectBestHuffmanTree >
CPA_DC_ASB_UNCOMP_STATIC_DYNAMIC_WITH_NO_HDRS)) {
QAT_UTILS_LOG("Invalid autoSelectBestHuffmanTree value\n");
return CPA_STATUS_INVALID_PARAM;
}
if (pSessionData->compType != CPA_DC_DEFLATE) {
QAT_UTILS_LOG("Invalid compType value\n");
return CPA_STATUS_INVALID_PARAM;
}
if ((pSessionData->huffType < CPA_DC_HT_STATIC) ||
(pSessionData->huffType > CPA_DC_HT_FULL_DYNAMIC) ||
(CPA_DC_HT_PRECOMP == pSessionData->huffType)) {
QAT_UTILS_LOG("Invalid huffType value\n");
return CPA_STATUS_INVALID_PARAM;
}
if ((pSessionData->sessDirection < CPA_DC_DIR_COMPRESS) ||
(pSessionData->sessDirection > CPA_DC_DIR_COMBINED)) {
QAT_UTILS_LOG("Invalid sessDirection value\n");
return CPA_STATUS_INVALID_PARAM;
}
if ((pSessionData->sessState < CPA_DC_STATEFUL) ||
(pSessionData->sessState > CPA_DC_STATELESS)) {
QAT_UTILS_LOG("Invalid sessState value\n");
return CPA_STATUS_INVALID_PARAM;
}
if ((pSessionData->checksum < CPA_DC_NONE) ||
(pSessionData->checksum > CPA_DC_ADLER32)) {
QAT_UTILS_LOG("Invalid checksum value\n");
return CPA_STATUS_INVALID_PARAM;
}
return CPA_STATUS_SUCCESS;
}
/**
*****************************************************************************
* @ingroup Dc_DataCompression
* Populate the compression hardware block
*
* @description
* This function will populate the compression hardware block and update
* the size in bytes of the block
*
* @param[in] pSessionDesc Pointer to the session descriptor
* @param[in] pCompConfig Pointer to slice config word
* @param[in] compDecomp Direction of the operation
* @param[in] enableDmm Delayed Match Mode
*
*****************************************************************************/
static void
dcCompHwBlockPopulate(dc_session_desc_t *pSessionDesc,
icp_qat_hw_compression_config_t *pCompConfig,
dc_request_dir_t compDecomp,
icp_qat_hw_compression_delayed_match_t enableDmm)
{
icp_qat_hw_compression_direction_t dir =
ICP_QAT_HW_COMPRESSION_DIR_COMPRESS;
icp_qat_hw_compression_algo_t algo =
ICP_QAT_HW_COMPRESSION_ALGO_DEFLATE;
icp_qat_hw_compression_depth_t depth = ICP_QAT_HW_COMPRESSION_DEPTH_1;
icp_qat_hw_compression_file_type_t filetype =
ICP_QAT_HW_COMPRESSION_FILE_TYPE_0;
/* Set the direction */
if (DC_COMPRESSION_REQUEST == compDecomp) {
dir = ICP_QAT_HW_COMPRESSION_DIR_COMPRESS;
} else {
dir = ICP_QAT_HW_COMPRESSION_DIR_DECOMPRESS;
}
if (CPA_DC_DEFLATE == pSessionDesc->compType) {
algo = ICP_QAT_HW_COMPRESSION_ALGO_DEFLATE;
} else {
QAT_UTILS_LOG("Algorithm not supported for Compression\n");
}
/* Set the depth */
if (DC_DECOMPRESSION_REQUEST == compDecomp) {
depth = ICP_QAT_HW_COMPRESSION_DEPTH_1;
} else {
switch (pSessionDesc->compLevel) {
case CPA_DC_L1:
depth = ICP_QAT_HW_COMPRESSION_DEPTH_1;
break;
case CPA_DC_L2:
depth = ICP_QAT_HW_COMPRESSION_DEPTH_4;
break;
case CPA_DC_L3:
depth = ICP_QAT_HW_COMPRESSION_DEPTH_8;
break;
default:
depth = ICP_QAT_HW_COMPRESSION_DEPTH_16;
}
}
/* The file type is set to ICP_QAT_HW_COMPRESSION_FILE_TYPE_0. The other
* modes will be used in the future for precompiled huffman trees */
filetype = ICP_QAT_HW_COMPRESSION_FILE_TYPE_0;
pCompConfig->val = ICP_QAT_HW_COMPRESSION_CONFIG_BUILD(
dir, enableDmm, algo, depth, filetype);
pCompConfig->reserved = 0;
}
/**
*****************************************************************************
* @ingroup Dc_DataCompression
* Populate the compression content descriptor
*
* @description
* This function will populate the compression content descriptor
*
* @param[in] pService Pointer to the service
* @param[in] pSessionDesc Pointer to the session descriptor
* @param[in] contextBufferAddrPhys Physical address of the context buffer
* @param[out] pMsg Pointer to the compression message
* @param[in] nextSlice Next slice
* @param[in] compDecomp Direction of the operation
*
*****************************************************************************/
static void
dcCompContentDescPopulate(sal_compression_service_t *pService,
dc_session_desc_t *pSessionDesc,
CpaPhysicalAddr contextBufferAddrPhys,
icp_qat_fw_comp_req_t *pMsg,
icp_qat_fw_slice_t nextSlice,
dc_request_dir_t compDecomp)
{
icp_qat_fw_comp_cd_hdr_t *pCompControlBlock = NULL;
icp_qat_hw_compression_config_t *pCompConfig = NULL;
CpaBoolean bankEnabled = CPA_FALSE;
pCompControlBlock = (icp_qat_fw_comp_cd_hdr_t *)&(pMsg->comp_cd_ctrl);
pCompConfig =
(icp_qat_hw_compression_config_t *)(pMsg->cd_pars.sl
.comp_slice_cfg_word);
ICP_QAT_FW_COMN_NEXT_ID_SET(pCompControlBlock, nextSlice);
ICP_QAT_FW_COMN_CURR_ID_SET(pCompControlBlock, ICP_QAT_FW_SLICE_COMP);
pCompControlBlock->comp_cfg_offset = 0;
if ((CPA_DC_STATEFUL == pSessionDesc->sessState) &&
(CPA_DC_DEFLATE == pSessionDesc->compType) &&
(DC_DECOMPRESSION_REQUEST == compDecomp)) {
/* Enable A, B, C, D, and E (CAMs). */
pCompControlBlock->ram_bank_flags =
ICP_QAT_FW_COMP_RAM_FLAGS_BUILD(
ICP_QAT_FW_COMP_BANK_DISABLED, /* Bank I */
ICP_QAT_FW_COMP_BANK_DISABLED, /* Bank H */
ICP_QAT_FW_COMP_BANK_DISABLED, /* Bank G */
ICP_QAT_FW_COMP_BANK_DISABLED, /* Bank F */
ICP_QAT_FW_COMP_BANK_ENABLED, /* Bank E */
ICP_QAT_FW_COMP_BANK_ENABLED, /* Bank D */
ICP_QAT_FW_COMP_BANK_ENABLED, /* Bank C */
ICP_QAT_FW_COMP_BANK_ENABLED, /* Bank B */
ICP_QAT_FW_COMP_BANK_ENABLED); /* Bank A */
bankEnabled = CPA_TRUE;
} else {
/* Disable all banks */
pCompControlBlock->ram_bank_flags =
ICP_QAT_FW_COMP_RAM_FLAGS_BUILD(
ICP_QAT_FW_COMP_BANK_DISABLED, /* Bank I */
ICP_QAT_FW_COMP_BANK_DISABLED, /* Bank H */
ICP_QAT_FW_COMP_BANK_DISABLED, /* Bank G */
ICP_QAT_FW_COMP_BANK_DISABLED, /* Bank F */
ICP_QAT_FW_COMP_BANK_DISABLED, /* Bank E */
ICP_QAT_FW_COMP_BANK_DISABLED, /* Bank D */
ICP_QAT_FW_COMP_BANK_DISABLED, /* Bank C */
ICP_QAT_FW_COMP_BANK_DISABLED, /* Bank B */
ICP_QAT_FW_COMP_BANK_DISABLED); /* Bank A */
}
if (DC_COMPRESSION_REQUEST == compDecomp) {
LAC_MEM_SHARED_WRITE_VIRT_TO_PHYS_PTR_EXTERNAL(
pService->generic_service_info,
pCompControlBlock->comp_state_addr,
pSessionDesc->stateRegistersComp);
} else {
LAC_MEM_SHARED_WRITE_VIRT_TO_PHYS_PTR_EXTERNAL(
pService->generic_service_info,
pCompControlBlock->comp_state_addr,
pSessionDesc->stateRegistersDecomp);
}
if (CPA_TRUE == bankEnabled) {
pCompControlBlock->ram_banks_addr = contextBufferAddrPhys;
} else {
pCompControlBlock->ram_banks_addr = 0;
}
pCompControlBlock->resrvd = 0;
/* Populate Compression Hardware Setup Block */
dcCompHwBlockPopulate(pSessionDesc,
pCompConfig,
compDecomp,
pService->comp_device_data.enableDmm);
}
/**
*****************************************************************************
* @ingroup Dc_DataCompression
* Populate the translator content descriptor
*
* @description
* This function will populate the translator content descriptor
*
* @param[out] pMsg Pointer to the compression message
* @param[in] nextSlice Next slice
*
*****************************************************************************/
static void
dcTransContentDescPopulate(icp_qat_fw_comp_req_t *pMsg,
icp_qat_fw_slice_t nextSlice)
{
icp_qat_fw_xlt_cd_hdr_t *pTransControlBlock = NULL;
pTransControlBlock = (icp_qat_fw_xlt_cd_hdr_t *)&(pMsg->u2.xlt_cd_ctrl);
ICP_QAT_FW_COMN_NEXT_ID_SET(pTransControlBlock, nextSlice);
ICP_QAT_FW_COMN_CURR_ID_SET(pTransControlBlock, ICP_QAT_FW_SLICE_XLAT);
pTransControlBlock->resrvd1 = 0;
pTransControlBlock->resrvd2 = 0;
pTransControlBlock->resrvd3 = 0;
}
/**
*****************************************************************************
* @ingroup Dc_DataCompression
* Get the context size and the history size
*
* @description
* This function will get the size of the context buffer and the history
* buffer. The history buffer is a subset of the context buffer and its
* size is needed for stateful compression.
* @param[in] dcInstance DC Instance Handle
*
* @param[in] pSessionData Pointer to a user instantiated
* structure containing session data
* @param[out] pContextSize Pointer to the context size
*
* @retval CPA_STATUS_SUCCESS Function executed successfully
*
*
*****************************************************************************/
static CpaStatus
dcGetContextSize(CpaInstanceHandle dcInstance,
CpaDcSessionSetupData *pSessionData,
Cpa32U *pContextSize)
{
sal_compression_service_t *pCompService = NULL;
pCompService = (sal_compression_service_t *)dcInstance;
*pContextSize = 0;
if ((CPA_DC_STATEFUL == pSessionData->sessState) &&
(CPA_DC_DEFLATE == pSessionData->compType) &&
(CPA_DC_DIR_COMPRESS != pSessionData->sessDirection)) {
*pContextSize =
pCompService->comp_device_data.inflateContextSize;
}
return CPA_STATUS_SUCCESS;
}
CpaStatus
dcInitSession(CpaInstanceHandle dcInstance,
CpaDcSessionHandle pSessionHandle,
CpaDcSessionSetupData *pSessionData,
CpaBufferList *pContextBuffer,
CpaDcCallbackFn callbackFn)
{
CpaStatus status = CPA_STATUS_SUCCESS;
sal_compression_service_t *pService = NULL;
icp_qat_fw_comp_req_t *pReqCache = NULL;
dc_session_desc_t *pSessionDesc = NULL;
CpaPhysicalAddr contextAddrPhys = 0;
CpaPhysicalAddr physAddress = 0;
CpaPhysicalAddr physAddressAligned = 0;
Cpa32U minContextSize = 0, historySize = 0;
Cpa32U rpCmdFlags = 0;
icp_qat_fw_serv_specif_flags cmdFlags = 0;
Cpa8U secureRam = ICP_QAT_FW_COMP_ENABLE_SECURE_RAM_USED_AS_INTMD_BUF;
Cpa8U sessType = ICP_QAT_FW_COMP_STATELESS_SESSION;
Cpa8U autoSelectBest = ICP_QAT_FW_COMP_NOT_AUTO_SELECT_BEST;
Cpa8U enhancedAutoSelectBest = ICP_QAT_FW_COMP_NOT_ENH_AUTO_SELECT_BEST;
Cpa8U disableType0EnhancedAutoSelectBest =
ICP_QAT_FW_COMP_NOT_DISABLE_TYPE0_ENH_AUTO_SELECT_BEST;
icp_qat_fw_la_cmd_id_t dcCmdId =
(icp_qat_fw_la_cmd_id_t)ICP_QAT_FW_COMP_CMD_STATIC;
icp_qat_fw_comn_flags cmnRequestFlags = 0;
dc_integrity_crc_fw_t *pDataIntegrityCrcs = NULL;
cmnRequestFlags =
ICP_QAT_FW_COMN_FLAGS_BUILD(DC_DEFAULT_QAT_PTR_TYPE,
QAT_COMN_CD_FLD_TYPE_16BYTE_DATA);
pService = (sal_compression_service_t *)dcInstance;
secureRam = pService->comp_device_data.useDevRam;
LAC_CHECK_NULL_PARAM(pSessionHandle);
LAC_CHECK_NULL_PARAM(pSessionData);
/* Check that the parameters defined in the pSessionData are valid for
* the
* device */
if (CPA_STATUS_SUCCESS !=
dcCheckSessionData(pSessionData, dcInstance)) {
return CPA_STATUS_INVALID_PARAM;
}
if ((CPA_DC_STATEFUL == pSessionData->sessState) &&
(CPA_DC_DIR_DECOMPRESS != pSessionData->sessDirection)) {
QAT_UTILS_LOG("Stateful sessions are not supported.\n");
return CPA_STATUS_UNSUPPORTED;
}
if (CPA_DC_HT_FULL_DYNAMIC == pSessionData->huffType) {
/* Test if DRAM is available for the intermediate buffers */
if ((NULL == pService->pInterBuffPtrsArray) &&
(0 == pService->pInterBuffPtrsArrayPhyAddr)) {
if (CPA_DC_ASB_STATIC_DYNAMIC ==
pSessionData->autoSelectBestHuffmanTree) {
/* Define the Huffman tree as static */
pSessionData->huffType = CPA_DC_HT_STATIC;
} else {
QAT_UTILS_LOG(
"No buffer defined for this instance - see cpaDcStartInstance.\n");
return CPA_STATUS_RESOURCE;
}
}
}
if ((CPA_DC_STATEFUL == pSessionData->sessState) &&
(CPA_DC_DEFLATE == pSessionData->compType)) {
/* Get the size of the context buffer */
status =
dcGetContextSize(dcInstance, pSessionData, &minContextSize);
if (CPA_STATUS_SUCCESS != status) {
QAT_UTILS_LOG(
"Unable to get the context size of the session.\n");
return CPA_STATUS_FAIL;
}
/* If the minContextSize is zero it means we will not save or
* restore
* any history */
if (0 != minContextSize) {
Cpa64U contextBuffSize = 0;
LAC_CHECK_NULL_PARAM(pContextBuffer);
if (LacBuffDesc_BufferListVerify(
pContextBuffer,
&contextBuffSize,
LAC_NO_ALIGNMENT_SHIFT) != CPA_STATUS_SUCCESS) {
return CPA_STATUS_INVALID_PARAM;
}
/* Ensure that the context buffer size is greater or
* equal
* to minContextSize */
if (contextBuffSize < minContextSize) {
QAT_UTILS_LOG(
"Context buffer size should be greater or equal to %d.\n",
minContextSize);
return CPA_STATUS_INVALID_PARAM;
}
}
}
/* Re-align the session structure to 64 byte alignment */
physAddress =
LAC_OS_VIRT_TO_PHYS_EXTERNAL(pService->generic_service_info,
(Cpa8U *)pSessionHandle +
sizeof(void *));
if (physAddress == 0) {
QAT_UTILS_LOG(
"Unable to get the physical address of the session.\n");
return CPA_STATUS_FAIL;
}
physAddressAligned =
(CpaPhysicalAddr)LAC_ALIGN_POW2_ROUNDUP(physAddress,
LAC_64BYTE_ALIGNMENT);
pSessionDesc = (dc_session_desc_t *)
/* Move the session pointer by the physical offset
between aligned and unaligned memory */
((Cpa8U *)pSessionHandle + sizeof(void *) +
(physAddressAligned - physAddress));
/* Save the aligned pointer in the first bytes (size of LAC_ARCH_UINT)
* of the session memory */
*((LAC_ARCH_UINT *)pSessionHandle) = (LAC_ARCH_UINT)pSessionDesc;
/* Zero the compression session */
LAC_OS_BZERO(pSessionDesc, sizeof(dc_session_desc_t));
/* Write the buffer descriptor for context/history */
if (0 != minContextSize) {
status = LacBuffDesc_BufferListDescWrite(
pContextBuffer,
&contextAddrPhys,
CPA_FALSE,
&(pService->generic_service_info));
if (status != CPA_STATUS_SUCCESS) {
return status;
}
pSessionDesc->pContextBuffer = pContextBuffer;
pSessionDesc->historyBuffSize = historySize;
}
pSessionDesc->cumulativeConsumedBytes = 0;
/* Initialise pSessionDesc */
pSessionDesc->requestType = DC_REQUEST_FIRST;
pSessionDesc->huffType = pSessionData->huffType;
pSessionDesc->compType = pSessionData->compType;
pSessionDesc->checksumType = pSessionData->checksum;
pSessionDesc->autoSelectBestHuffmanTree =
pSessionData->autoSelectBestHuffmanTree;
pSessionDesc->sessDirection = pSessionData->sessDirection;
pSessionDesc->sessState = pSessionData->sessState;
pSessionDesc->compLevel = pSessionData->compLevel;
pSessionDesc->isDcDp = CPA_FALSE;
pSessionDesc->minContextSize = minContextSize;
pSessionDesc->isSopForCompressionProcessed = CPA_FALSE;
pSessionDesc->isSopForDecompressionProcessed = CPA_FALSE;
if (CPA_DC_ADLER32 == pSessionDesc->checksumType) {
pSessionDesc->previousChecksum = 1;
} else {
pSessionDesc->previousChecksum = 0;
}
if (CPA_DC_STATEFUL == pSessionData->sessState) {
/* Init the spinlock used to lock the access to the number of
* stateful
* in-flight requests */
status = LAC_SPINLOCK_INIT(&(pSessionDesc->sessionLock));
if (CPA_STATUS_SUCCESS != status) {
QAT_UTILS_LOG(
"Spinlock init failed for sessionLock.\n");
return CPA_STATUS_RESOURCE;
}
}
/* For asynchronous - use the user supplied callback
* for synchronous - use the internal synchronous callback */
pSessionDesc->pCompressionCb = ((void *)NULL != (void *)callbackFn) ?
callbackFn :
LacSync_GenWakeupSyncCaller;
/* Reset the pending callback counters */
qatUtilsAtomicSet(0, &pSessionDesc->pendingStatelessCbCount);
qatUtilsAtomicSet(0, &pSessionDesc->pendingStatefulCbCount);
pSessionDesc->pendingDpStatelessCbCount = 0;
if (CPA_DC_DIR_DECOMPRESS != pSessionData->sessDirection) {
if (CPA_DC_HT_FULL_DYNAMIC == pSessionData->huffType) {
/* Populate the compression section of the content
* descriptor */
dcCompContentDescPopulate(pService,
pSessionDesc,
contextAddrPhys,
&(pSessionDesc->reqCacheComp),
ICP_QAT_FW_SLICE_XLAT,
DC_COMPRESSION_REQUEST);
/* Populate the translator section of the content
* descriptor */
dcTransContentDescPopulate(
&(pSessionDesc->reqCacheComp),
ICP_QAT_FW_SLICE_DRAM_WR);
if (0 != pService->pInterBuffPtrsArrayPhyAddr) {
pReqCache = &(pSessionDesc->reqCacheComp);
pReqCache->u1.xlt_pars.inter_buff_ptr =
pService->pInterBuffPtrsArrayPhyAddr;
}
} else {
dcCompContentDescPopulate(pService,
pSessionDesc,
contextAddrPhys,
&(pSessionDesc->reqCacheComp),
ICP_QAT_FW_SLICE_DRAM_WR,
DC_COMPRESSION_REQUEST);
}
}
/* Populate the compression section of the content descriptor for
* the decompression case or combined */
if (CPA_DC_DIR_COMPRESS != pSessionData->sessDirection) {
dcCompContentDescPopulate(pService,
pSessionDesc,
contextAddrPhys,
&(pSessionDesc->reqCacheDecomp),
ICP_QAT_FW_SLICE_DRAM_WR,
DC_DECOMPRESSION_REQUEST);
}
if (CPA_DC_STATEFUL == pSessionData->sessState) {
sessType = ICP_QAT_FW_COMP_STATEFUL_SESSION;
LAC_OS_BZERO(&pSessionDesc->stateRegistersComp,
sizeof(pSessionDesc->stateRegistersComp));
LAC_OS_BZERO(&pSessionDesc->stateRegistersDecomp,
sizeof(pSessionDesc->stateRegistersDecomp));
}
/* Get physical address of E2E CRC buffer */
pSessionDesc->physDataIntegrityCrcs = (icp_qat_addr_width_t)
LAC_OS_VIRT_TO_PHYS_EXTERNAL(pService->generic_service_info,
&pSessionDesc->dataIntegrityCrcs);
if (0 == pSessionDesc->physDataIntegrityCrcs) {
QAT_UTILS_LOG(
"Unable to get the physical address of Data Integrity buffer.\n");
return CPA_STATUS_FAIL;
}
/* Initialize default CRC parameters */
pDataIntegrityCrcs = &pSessionDesc->dataIntegrityCrcs;
pDataIntegrityCrcs->crc32 = 0;
pDataIntegrityCrcs->adler32 = 1;
pDataIntegrityCrcs->oCrc32Cpr = DC_INVALID_CRC;
pDataIntegrityCrcs->iCrc32Cpr = DC_INVALID_CRC;
pDataIntegrityCrcs->oCrc32Xlt = DC_INVALID_CRC;
pDataIntegrityCrcs->iCrc32Xlt = DC_INVALID_CRC;
pDataIntegrityCrcs->xorFlags = DC_XOR_FLAGS_DEFAULT;
pDataIntegrityCrcs->crcPoly = DC_CRC_POLY_DEFAULT;
pDataIntegrityCrcs->xorOut = DC_XOR_OUT_DEFAULT;
/* Initialise seed checksums */
pSessionDesc->seedSwCrc.swCrcI = 0;
pSessionDesc->seedSwCrc.swCrcO = 0;
/* Populate the cmdFlags */
switch (pSessionDesc->autoSelectBestHuffmanTree) {
case CPA_DC_ASB_DISABLED:
break;
case CPA_DC_ASB_STATIC_DYNAMIC:
autoSelectBest = ICP_QAT_FW_COMP_AUTO_SELECT_BEST;
break;
case CPA_DC_ASB_UNCOMP_STATIC_DYNAMIC_WITH_STORED_HDRS:
autoSelectBest = ICP_QAT_FW_COMP_AUTO_SELECT_BEST;
enhancedAutoSelectBest = ICP_QAT_FW_COMP_ENH_AUTO_SELECT_BEST;
break;
case CPA_DC_ASB_UNCOMP_STATIC_DYNAMIC_WITH_NO_HDRS:
autoSelectBest = ICP_QAT_FW_COMP_AUTO_SELECT_BEST;
enhancedAutoSelectBest = ICP_QAT_FW_COMP_ENH_AUTO_SELECT_BEST;
disableType0EnhancedAutoSelectBest =
ICP_QAT_FW_COMP_DISABLE_TYPE0_ENH_AUTO_SELECT_BEST;
break;
default:
break;
}
rpCmdFlags = ICP_QAT_FW_COMP_REQ_PARAM_FLAGS_BUILD(
ICP_QAT_FW_COMP_SOP,
ICP_QAT_FW_COMP_EOP,
ICP_QAT_FW_COMP_BFINAL,
ICP_QAT_FW_COMP_NO_CNV,
ICP_QAT_FW_COMP_NO_CNV_RECOVERY,
ICP_QAT_FW_COMP_CRC_MODE_LEGACY);
cmdFlags =
ICP_QAT_FW_COMP_FLAGS_BUILD(sessType,
autoSelectBest,
enhancedAutoSelectBest,
disableType0EnhancedAutoSelectBest,
secureRam);
if (CPA_DC_DIR_DECOMPRESS != pSessionData->sessDirection) {
if (CPA_DC_HT_FULL_DYNAMIC == pSessionDesc->huffType) {
dcCmdId = (icp_qat_fw_la_cmd_id_t)(
ICP_QAT_FW_COMP_CMD_DYNAMIC);
}
pReqCache = &(pSessionDesc->reqCacheComp);
pReqCache->comp_pars.req_par_flags = rpCmdFlags;
pReqCache->comp_pars.crc.legacy.initial_adler = 1;
pReqCache->comp_pars.crc.legacy.initial_crc32 = 0;
/* Populate header of the common request message */
SalQatMsg_CmnHdrWrite((icp_qat_fw_comn_req_t *)pReqCache,
ICP_QAT_FW_COMN_REQ_CPM_FW_COMP,
(uint8_t)dcCmdId,
cmnRequestFlags,
cmdFlags);
}
if (CPA_DC_DIR_COMPRESS != pSessionData->sessDirection) {
dcCmdId =
(icp_qat_fw_la_cmd_id_t)(ICP_QAT_FW_COMP_CMD_DECOMPRESS);
pReqCache = &(pSessionDesc->reqCacheDecomp);
pReqCache->comp_pars.req_par_flags = rpCmdFlags;
pReqCache->comp_pars.crc.legacy.initial_adler = 1;
pReqCache->comp_pars.crc.legacy.initial_crc32 = 0;
/* Populate header of the common request message */
SalQatMsg_CmnHdrWrite((icp_qat_fw_comn_req_t *)pReqCache,
ICP_QAT_FW_COMN_REQ_CPM_FW_COMP,
(uint8_t)dcCmdId,
cmnRequestFlags,
cmdFlags);
}
return status;
}
CpaStatus
cpaDcInitSession(CpaInstanceHandle dcInstance,
CpaDcSessionHandle pSessionHandle,
CpaDcSessionSetupData *pSessionData,
CpaBufferList *pContextBuffer,
CpaDcCallbackFn callbackFn)
{
CpaInstanceHandle insHandle = NULL;
sal_compression_service_t *pService = NULL;
if (CPA_INSTANCE_HANDLE_SINGLE == dcInstance) {
insHandle = dcGetFirstHandle();
} else {
insHandle = dcInstance;
}
LAC_CHECK_INSTANCE_HANDLE(insHandle);
SAL_CHECK_INSTANCE_TYPE(insHandle, SAL_SERVICE_TYPE_COMPRESSION);
pService = (sal_compression_service_t *)insHandle;
/* Check if SAL is initialised otherwise return an error */
SAL_RUNNING_CHECK(pService);
return dcInitSession(insHandle,
pSessionHandle,
pSessionData,
pContextBuffer,
callbackFn);
}
CpaStatus
cpaDcResetSession(const CpaInstanceHandle dcInstance,
CpaDcSessionHandle pSessionHandle)
{
CpaStatus status = CPA_STATUS_SUCCESS;
CpaInstanceHandle insHandle = NULL;
dc_session_desc_t *pSessionDesc = NULL;
Cpa64U numPendingStateless = 0;
Cpa64U numPendingStateful = 0;
icp_comms_trans_handle trans_handle = NULL;
LAC_CHECK_NULL_PARAM(pSessionHandle);
pSessionDesc = DC_SESSION_DESC_FROM_CTX_GET(pSessionHandle);
LAC_CHECK_NULL_PARAM(pSessionDesc);
if (CPA_TRUE == pSessionDesc->isDcDp) {
insHandle = dcInstance;
} else {
if (CPA_INSTANCE_HANDLE_SINGLE == dcInstance) {
insHandle = dcGetFirstHandle();
} else {
insHandle = dcInstance;
}
}
LAC_CHECK_NULL_PARAM(insHandle);
SAL_CHECK_INSTANCE_TYPE(insHandle, SAL_SERVICE_TYPE_COMPRESSION);
/* Check if SAL is running otherwise return an error */
SAL_RUNNING_CHECK(insHandle);
if (CPA_TRUE == pSessionDesc->isDcDp) {
trans_handle = ((sal_compression_service_t *)dcInstance)
->trans_handle_compression_tx;
if (CPA_TRUE == icp_adf_queueDataToSend(trans_handle)) {
/* Process the remaining messages on the ring */
SalQatMsg_updateQueueTail(trans_handle);
QAT_UTILS_LOG(
"There are remaining messages on the ring\n");
return CPA_STATUS_RETRY;
}
/* Check if there are stateless pending requests */
if (0 != pSessionDesc->pendingDpStatelessCbCount) {
QAT_UTILS_LOG(
"There are %llu stateless DP requests pending.\n",
(unsigned long long)
pSessionDesc->pendingDpStatelessCbCount);
return CPA_STATUS_RETRY;
}
} else {
numPendingStateless =
qatUtilsAtomicGet(&(pSessionDesc->pendingStatelessCbCount));
numPendingStateful =
qatUtilsAtomicGet(&(pSessionDesc->pendingStatefulCbCount));
/* Check if there are stateless pending requests */
if (0 != numPendingStateless) {
QAT_UTILS_LOG(
"There are %llu stateless requests pending.\n",
(unsigned long long)numPendingStateless);
return CPA_STATUS_RETRY;
}
/* Check if there are stateful pending requests */
if (0 != numPendingStateful) {
QAT_UTILS_LOG(
"There are %llu stateful requests pending.\n",
(unsigned long long)numPendingStateful);
return CPA_STATUS_RETRY;
}
/* Reset pSessionDesc */
pSessionDesc->requestType = DC_REQUEST_FIRST;
pSessionDesc->cumulativeConsumedBytes = 0;
if (CPA_DC_ADLER32 == pSessionDesc->checksumType) {
pSessionDesc->previousChecksum = 1;
} else {
pSessionDesc->previousChecksum = 0;
}
}
/* Reset the pending callback counters */
qatUtilsAtomicSet(0, &pSessionDesc->pendingStatelessCbCount);
qatUtilsAtomicSet(0, &pSessionDesc->pendingStatefulCbCount);
pSessionDesc->pendingDpStatelessCbCount = 0;
if (CPA_DC_STATEFUL == pSessionDesc->sessState) {
LAC_OS_BZERO(&pSessionDesc->stateRegistersComp,
sizeof(pSessionDesc->stateRegistersComp));
LAC_OS_BZERO(&pSessionDesc->stateRegistersDecomp,
sizeof(pSessionDesc->stateRegistersDecomp));
}
return status;
}
CpaStatus
cpaDcRemoveSession(const CpaInstanceHandle dcInstance,
CpaDcSessionHandle pSessionHandle)
{
CpaStatus status = CPA_STATUS_SUCCESS;
CpaInstanceHandle insHandle = NULL;
dc_session_desc_t *pSessionDesc = NULL;
Cpa64U numPendingStateless = 0;
Cpa64U numPendingStateful = 0;
icp_comms_trans_handle trans_handle = NULL;
LAC_CHECK_NULL_PARAM(pSessionHandle);
pSessionDesc = DC_SESSION_DESC_FROM_CTX_GET(pSessionHandle);
LAC_CHECK_NULL_PARAM(pSessionDesc);
if (CPA_TRUE == pSessionDesc->isDcDp) {
insHandle = dcInstance;
} else {
if (CPA_INSTANCE_HANDLE_SINGLE == dcInstance) {
insHandle = dcGetFirstHandle();
} else {
insHandle = dcInstance;
}
}
LAC_CHECK_NULL_PARAM(insHandle);
SAL_CHECK_INSTANCE_TYPE(insHandle, SAL_SERVICE_TYPE_COMPRESSION);
/* Check if SAL is running otherwise return an error */
SAL_RUNNING_CHECK(insHandle);
if (CPA_TRUE == pSessionDesc->isDcDp) {
trans_handle = ((sal_compression_service_t *)insHandle)
->trans_handle_compression_tx;
if (CPA_TRUE == icp_adf_queueDataToSend(trans_handle)) {
/* Process the remaining messages on the ring */
SalQatMsg_updateQueueTail(trans_handle);
QAT_UTILS_LOG(
"There are remaining messages on the ring.\n");
return CPA_STATUS_RETRY;
}
/* Check if there are stateless pending requests */
if (0 != pSessionDesc->pendingDpStatelessCbCount) {
QAT_UTILS_LOG(
"There are %llu stateless DP requests pending.\n",
(unsigned long long)
pSessionDesc->pendingDpStatelessCbCount);
return CPA_STATUS_RETRY;
}
} else {
numPendingStateless =
qatUtilsAtomicGet(&(pSessionDesc->pendingStatelessCbCount));
numPendingStateful =
qatUtilsAtomicGet(&(pSessionDesc->pendingStatefulCbCount));
/* Check if there are stateless pending requests */
if (0 != numPendingStateless) {
QAT_UTILS_LOG(
"There are %llu stateless requests pending.\n",
(unsigned long long)numPendingStateless);
status = CPA_STATUS_RETRY;
}
/* Check if there are stateful pending requests */
if (0 != numPendingStateful) {
QAT_UTILS_LOG(
"There are %llu stateful requests pending.\n",
(unsigned long long)numPendingStateful);
status = CPA_STATUS_RETRY;
}
if ((CPA_DC_STATEFUL == pSessionDesc->sessState) &&
(CPA_STATUS_SUCCESS == status)) {
if (CPA_STATUS_SUCCESS !=
LAC_SPINLOCK_DESTROY(
&(pSessionDesc->sessionLock))) {
QAT_UTILS_LOG(
"Failed to destory session lock.\n");
}
}
}
return status;
}
CpaStatus
dcGetSessionSize(CpaInstanceHandle dcInstance,
CpaDcSessionSetupData *pSessionData,
Cpa32U *pSessionSize,
Cpa32U *pContextSize)
{
CpaStatus status = CPA_STATUS_SUCCESS;
CpaInstanceHandle insHandle = NULL;
if (CPA_INSTANCE_HANDLE_SINGLE == dcInstance) {
insHandle = dcGetFirstHandle();
} else {
insHandle = dcInstance;
}
/* Check parameters */
LAC_CHECK_NULL_PARAM(insHandle);
LAC_CHECK_NULL_PARAM(pSessionData);
LAC_CHECK_NULL_PARAM(pSessionSize);
if (dcCheckSessionData(pSessionData, insHandle) != CPA_STATUS_SUCCESS) {
return CPA_STATUS_INVALID_PARAM;
}
/* Get session size for session data */
*pSessionSize = sizeof(dc_session_desc_t) + LAC_64BYTE_ALIGNMENT +
sizeof(LAC_ARCH_UINT);
if (NULL != pContextSize) {
status =
dcGetContextSize(insHandle, pSessionData, pContextSize);
if (CPA_STATUS_SUCCESS != status) {
QAT_UTILS_LOG(
"Unable to get the context size of the session.\n");
return CPA_STATUS_FAIL;
}
}
return CPA_STATUS_SUCCESS;
}
CpaStatus
cpaDcGetSessionSize(CpaInstanceHandle dcInstance,
CpaDcSessionSetupData *pSessionData,
Cpa32U *pSessionSize,
Cpa32U *pContextSize)
{
LAC_CHECK_NULL_PARAM(pContextSize);
return dcGetSessionSize(dcInstance,
pSessionData,
pSessionSize,
pContextSize);
}

90
sys/dev/qat/qat_api/common/compression/dc_stats.c Normal file
View File

@ -0,0 +1,90 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file dc_stats.c
*
* @ingroup Dc_DataCompression
*
* @description
* Implementation of the Data Compression stats operations.
*
*****************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "cpa_dc.h"
#include "icp_accel_devices.h"
#include "icp_adf_debug.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "lac_common.h"
#include "icp_accel_devices.h"
#include "sal_statistics.h"
#include "dc_session.h"
#include "dc_datapath.h"
#include "lac_mem_pools.h"
#include "sal_service_state.h"
#include "sal_types_compression.h"
#include "dc_stats.h"
CpaStatus
dcStatsInit(sal_compression_service_t *pService)
{
CpaStatus status = CPA_STATUS_SUCCESS;
pService->pCompStatsArr =
LAC_OS_MALLOC(COMPRESSION_NUM_STATS * sizeof(QatUtilsAtomic));
if (pService->pCompStatsArr == NULL) {
status = CPA_STATUS_RESOURCE;
}
if (CPA_STATUS_SUCCESS == status) {
COMPRESSION_STATS_RESET(pService);
}
return status;
}
void
dcStatsFree(sal_compression_service_t *pService)
{
if (NULL != pService->pCompStatsArr) {
LAC_OS_FREE(pService->pCompStatsArr);
}
}
CpaStatus
cpaDcGetStats(CpaInstanceHandle dcInstance, CpaDcStats *pStatistics)
{
sal_compression_service_t *pService = NULL;
CpaInstanceHandle insHandle = NULL;
if (CPA_INSTANCE_HANDLE_SINGLE == dcInstance) {
insHandle = dcGetFirstHandle();
} else {
insHandle = dcInstance;
}
pService = (sal_compression_service_t *)insHandle;
LAC_CHECK_NULL_PARAM(insHandle);
LAC_CHECK_NULL_PARAM(pStatistics);
SAL_RUNNING_CHECK(insHandle);
SAL_CHECK_INSTANCE_TYPE(insHandle, SAL_SERVICE_TYPE_COMPRESSION);
/* Retrieves the statistics for compression */
COMPRESSION_STATS_GET(pStatistics, pService);
return CPA_STATUS_SUCCESS;
}

33
sys/dev/qat/qat_api/common/compression/icp_sal_dc_err.c Normal file
View File

@ -0,0 +1,33 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file icp_sal_dc_err.c
*
* @defgroup SalCommon
*
* @ingroup SalCommon
*
*****************************************************************************/
/*
******************************************************************************
* Include public/global header files
******************************************************************************
*/
#include "cpa.h"
#include "icp_sal.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "dc_error_counter.h"
Cpa64U
icp_sal_get_dc_error(Cpa8S dcError)
{
return getDcErrorCounter(dcError);
}

186
sys/dev/qat/qat_api/common/compression/include/dc_datapath.h Normal file
View File

@ -0,0 +1,186 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file dc_datapath.h
*
* @ingroup Dc_DataCompression
*
* @description
* Definition of the Data Compression datapath parameters.
*
*******************
* **********************************************************/
#ifndef DC_DATAPATH_H_
#define DC_DATAPATH_H_
#define LAC_QAT_DC_REQ_SZ_LW 32
#define LAC_QAT_DC_RESP_SZ_LW 8
/* Restriction on the source buffer size for compression due to the firmware
* processing */
#define DC_SRC_BUFFER_MIN_SIZE (15)
/* Restriction on the destination buffer size for compression due to
* the management of skid buffers in the firmware */
#define DC_DEST_BUFFER_DYN_MIN_SIZE (128)
#define DC_DEST_BUFFER_STA_MIN_SIZE (64)
/* C62x and C3xxx pcie rev0 devices require an additional 32bytes */
#define DC_DEST_BUFFER_STA_ADDITIONAL_SIZE (32)
/* C4xxx device only requires 47 bytes */
#define DC_DEST_BUFFER_MIN_SIZE (47)
/* Minimum destination buffer size for decompression */
#define DC_DEST_BUFFER_DEC_MIN_SIZE (1)
/* Restriction on the source and destination buffer sizes for compression due
* to the firmware taking 32 bits parameters. The max size is 2^32-1 */
#define DC_BUFFER_MAX_SIZE (0xFFFFFFFF)
/* DC Source & Destination buffer type (FLAT/SGL) */
#define DC_DEFAULT_QAT_PTR_TYPE QAT_COMN_PTR_TYPE_SGL
#define DC_DP_QAT_PTR_TYPE QAT_COMN_PTR_TYPE_FLAT
/* Offset to first byte of Input Byte Counter (IBC) in state register */
#define DC_STATE_IBC_OFFSET (8)
/* Size in bytes of input byte counter (IBC) in state register */
#define DC_IBC_SIZE_IN_BYTES (4)
/* Offset to first byte to CRC32 in state register */
#define DC_STATE_CRC32_OFFSET (40)
/* Offset to first byte to output CRC32 in state register */
#define DC_STATE_OUTPUT_CRC32_OFFSET (48)
/* Offset to first byte to input CRC32 in state register */
#define DC_STATE_INPUT_CRC32_OFFSET (52)
/* Offset to first byte of ADLER32 in state register */
#define DC_STATE_ADLER32_OFFSET (48)
/* 8 bit mask value */
#define DC_8_BIT_MASK (0xff)
/* 8 bit shift position */
#define DC_8_BIT_SHIFT_POS (8)
/* Size in bytes of checksum */
#define DC_CHECKSUM_SIZE_IN_BYTES (4)
/* Mask used to set the most significant bit to zero */
#define DC_STATE_REGISTER_ZERO_MSB_MASK (0x7F)
/* Mask used to keep only the most significant bit and set the others to zero */
#define DC_STATE_REGISTER_KEEP_MSB_MASK (0x80)
/* Compression state register word containing the parity bit */
#define DC_STATE_REGISTER_PARITY_BIT_WORD (5)
/* Location of the parity bit within the compression state register word */
#define DC_STATE_REGISTER_PARITY_BIT (7)
/* size which needs to be reserved before the results field to
* align the results field with the API struct */
#define DC_API_ALIGNMENT_OFFSET (offsetof(CpaDcDpOpData, results))
/* Mask used to check the CompressAndVerify capability bit */
#define DC_CNV_EXTENDED_CAPABILITY (0x01)
/* Mask used to check the CompressAndVerifyAndRecover capability bit */
#define DC_CNVNR_EXTENDED_CAPABILITY (0x100)
/* Default values for CNV integrity checks,
* those are used to inform hardware of specifying CRC parameters to be used
* when calculating CRCs */
#define DC_CRC_POLY_DEFAULT 0x04c11db7
#define DC_XOR_FLAGS_DEFAULT 0xe0000
#define DC_XOR_OUT_DEFAULT 0xffffffff
#define DC_INVALID_CRC 0x0
/**
*******************************************************************************
* @ingroup cpaDc Data Compression
* Compression cookie
* @description
* This cookie stores information for a particular compression perform op.
* This includes various user-supplied parameters for the operation which
* will be needed in our callback function.
* A pointer to this cookie is stored in the opaque data field of the QAT
* message so that it can be accessed in the asynchronous callback.
* @note
* The order of the parameters within this structure is important. It needs
* to match the order of the parameters in CpaDcDpOpData up to the
* pSessionHandle. This allows the correct processing of the callback.
*****************************************************************************/
typedef struct dc_compression_cookie_s {
Cpa8U dcReqParamsBuffer[DC_API_ALIGNMENT_OFFSET];
/**< Memory block - was previously reserved for request parameters.
* Now size maintained so following members align with API struct,
* but no longer used for request parameters */
CpaDcRqResults reserved;
/**< This is reserved for results to correctly align the structure
* to match the one from the data plane API */
CpaInstanceHandle dcInstance;
/**< Compression instance handle */
CpaDcSessionHandle pSessionHandle;
/**< Pointer to the session handle */
icp_qat_fw_comp_req_t request;
/**< Compression request */
void *callbackTag;
/**< Opaque data supplied by the client */
dc_session_desc_t *pSessionDesc;
/**< Pointer to the session descriptor */
CpaDcFlush flushFlag;
/**< Flush flag */
CpaDcOpData *pDcOpData;
/**< struct containing flags and CRC related data for this session */
CpaDcRqResults *pResults;
/**< Pointer to result buffer holding consumed and produced data */
Cpa32U srcTotalDataLenInBytes;
/**< Total length of the source data */
Cpa32U dstTotalDataLenInBytes;
/**< Total length of the destination data */
dc_request_dir_t compDecomp;
/**< Used to know whether the request is compression or decompression.
* Useful when defining the session as combined */
CpaBufferList *pUserSrcBuff;
/**< virtual userspace ptr to source SGL */
CpaBufferList *pUserDestBuff;
/**< virtual userspace ptr to destination SGL */
} dc_compression_cookie_t;
/**
*****************************************************************************
* @ingroup Dc_DataCompression
* Callback function called for compression and decompression requests in
* asynchronous mode
*
* @description
* Called to process compression and decompression response messages. This
* callback will check for errors, update the statistics and will call the
* user callback
*
* @param[in] pRespMsg Response message
*
*****************************************************************************/
void dcCompression_ProcessCallback(void *pRespMsg);
/**
*****************************************************************************
* @ingroup Dc_DataCompression
* Describes CNV and CNVNR modes
*
* @description
* This enum is used to indicate the CNV modes.
*
*****************************************************************************/
typedef enum dc_cnv_mode_s {
DC_NO_CNV = 0,
/* CNV = FALSE, CNVNR = FALSE */
DC_CNV,
/* CNV = TRUE, CNVNR = FALSE */
DC_CNVNR,
/* CNV = TRUE, CNVNR = TRUE */
} dc_cnv_mode_t;
#endif /* DC_DATAPATH_H_ */

25
sys/dev/qat/qat_api/common/compression/include/dc_error_counter.h Normal file
View File

@ -0,0 +1,25 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file dc_error_counter.h
*
* @ingroup Dc_DataCompression
*
* @description
* Definition of the Data Compression Error Counter parameters.
*
*****************************************************************************/
#ifndef DC_ERROR_COUNTER_H
#define DC_ERROR_COUNTER_H
#include "cpa_types.h"
#include "cpa_dc.h"
#define MAX_DC_ERROR_TYPE 20
void dcErrorLog(CpaDcReqStatus dcError);
Cpa64U getDcErrorCounter(CpaDcReqStatus dcError);
#endif /* DC_ERROR_COUNTER_H */

44
sys/dev/qat/qat_api/common/compression/include/dc_header_footer.h Normal file
View File

@ -0,0 +1,44 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file dc_header_footer.h
*
* @ingroup Dc_DataCompression
*
* @description
* Definition of the Data Compression header and footer parameters.
*
*****************************************************************************/
#ifndef DC_HEADER_FOOTER_H_
#define DC_HEADER_FOOTER_H_
/* Header and footer sizes for Zlib and Gzip */
#define DC_ZLIB_HEADER_SIZE (2)
#define DC_GZIP_HEADER_SIZE (10)
#define DC_ZLIB_FOOTER_SIZE (4)
#define DC_GZIP_FOOTER_SIZE (8)
/* Values used to build the headers for Zlib and Gzip */
#define DC_GZIP_ID1 (0x1f)
#define DC_GZIP_ID2 (0x8b)
#define DC_GZIP_FILESYSTYPE (0x03)
#define DC_ZLIB_WINDOWSIZE_OFFSET (4)
#define DC_ZLIB_FLEVEL_OFFSET (6)
#define DC_ZLIB_HEADER_OFFSET (31)
/* Compression level for Zlib */
#define DC_ZLIB_LEVEL_0 (0)
#define DC_ZLIB_LEVEL_1 (1)
#define DC_ZLIB_LEVEL_2 (2)
#define DC_ZLIB_LEVEL_3 (3)
/* CM parameter for Zlib */
#define DC_ZLIB_CM_DEFLATE (8)
/* Type of Gzip compression */
#define DC_GZIP_FAST_COMP (4)
#define DC_GZIP_MAX_COMP (2)
#endif /* DC_HEADER_FOOTER_H_ */

278
sys/dev/qat/qat_api/common/compression/include/dc_session.h Normal file
View File

@ -0,0 +1,278 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file dc_session.h
*
* @ingroup Dc_DataCompression
*
* @description
* Definition of the Data Compression session parameters.
*
*****************************************************************************/
#ifndef DC_SESSION_H
#define DC_SESSION_H
#include "cpa_dc_dp.h"
#include "icp_qat_fw_comp.h"
#include "sal_qat_cmn_msg.h"
/* Maximum number of intermediate buffers SGLs for devices
* with a maximum of 6 compression slices */
#define DC_QAT_MAX_NUM_INTER_BUFFERS_6COMP_SLICES (12)
/* Maximum number of intermediate buffers SGLs for devices
* with a maximum of 10 max compression slices */
#define DC_QAT_MAX_NUM_INTER_BUFFERS_10COMP_SLICES (20)
/* Maximum number of intermediate buffers SGLs for devices
* with a maximum of 24 max compression slices and 32 MEs */
#define DC_QAT_MAX_NUM_INTER_BUFFERS_24COMP_SLICES (64)
/* Maximum size of the state registers 64 bytes */
#define DC_QAT_STATE_REGISTERS_MAX_SIZE (64)
/* Size of the history window.
* Base 2 logarithm of maximum window size minus 8 */
#define DC_8K_WINDOW_SIZE (5)
#define DC_16K_WINDOW_SIZE (6)
#define DC_32K_WINDOW_SIZE (7)
/* Context size */
#define DC_DEFLATE_MAX_CONTEXT_SIZE (49152)
#define DC_INFLATE_CONTEXT_SIZE (36864)
#define DC_DEFLATE_EH_MAX_CONTEXT_SIZE (65536)
#define DC_DEFLATE_EH_MIN_CONTEXT_SIZE (49152)
#define DC_INFLATE_EH_CONTEXT_SIZE (34032)
/* Retrieve the session descriptor pointer from the session context structure
* that the user allocates. The pointer to the internally realigned address
* is stored at the start of the session context that the user allocates */
#define DC_SESSION_DESC_FROM_CTX_GET(pSession) \
(dc_session_desc_t *)(*(LAC_ARCH_UINT *)pSession)
/* Maximum size for the compression part of the content descriptor */
#define DC_QAT_COMP_CONTENT_DESC_SIZE sizeof(icp_qat_fw_comp_cd_hdr_t)
/* Maximum size for the translator part of the content descriptor */
#define DC_QAT_TRANS_CONTENT_DESC_SIZE \
(sizeof(icp_qat_fw_xlt_cd_hdr_t) + DC_QAT_MAX_TRANS_SETUP_BLK_SZ)
/* Maximum size of the decompression content descriptor */
#define DC_QAT_CONTENT_DESC_DECOMP_MAX_SIZE \
LAC_ALIGN_POW2_ROUNDUP(DC_QAT_COMP_CONTENT_DESC_SIZE, \
(1 << LAC_64BYTE_ALIGNMENT_SHIFT))
/* Maximum size of the compression content descriptor */
#define DC_QAT_CONTENT_DESC_COMP_MAX_SIZE \
LAC_ALIGN_POW2_ROUNDUP(DC_QAT_COMP_CONTENT_DESC_SIZE + \
DC_QAT_TRANS_CONTENT_DESC_SIZE, \
(1 << LAC_64BYTE_ALIGNMENT_SHIFT))
/* Direction of the request */
typedef enum dc_request_dir_e {
DC_COMPRESSION_REQUEST = 1,
DC_DECOMPRESSION_REQUEST
} dc_request_dir_t;
/* Type of the compression request */
typedef enum dc_request_type_e {
DC_REQUEST_FIRST = 1,
DC_REQUEST_SUBSEQUENT
} dc_request_type_t;
typedef enum dc_block_type_e {
DC_CLEARTEXT_TYPE = 0,
DC_STATIC_TYPE,
DC_DYNAMIC_TYPE
} dc_block_type_t;
/* Internal data structure supporting end to end data integrity checks. */
typedef struct dc_integrity_crc_fw_s {
Cpa32U crc32;
/* CRC32 checksum returned for compressed data */
Cpa32U adler32;
/* ADLER32 checksum returned for compressed data */
Cpa32U oCrc32Cpr;
/* CRC32 checksum returned for data output by compression accelerator */
Cpa32U iCrc32Cpr;
/* CRC32 checksum returned for input data to compression accelerator */
Cpa32U oCrc32Xlt;
/* CRC32 checksum returned for data output by translator accelerator */
Cpa32U iCrc32Xlt;
/* CRC32 checksum returned for input data to translator accelerator */
Cpa32U xorFlags;
/* Initialise transactor pCRC controls in state register */
Cpa32U crcPoly;
/* CRC32 polynomial used by hardware */
Cpa32U xorOut;
/* CRC32 from XOR stage (Input CRC is xor'ed with value in the state) */
Cpa32U deflateBlockType;
/* Bit 1 - Bit 0
* 0 0 -> RAW DATA + Deflate header.
* This will not produced any CRC check because
* the output will not come from the slices.
* It will be a simple copy from input to output
* buffers list.
* 0 1 -> Static deflate block type
* 1 0 -> Dynamic deflate block type
* 1 1 -> Invalid type */
} dc_integrity_crc_fw_t;
typedef struct dc_sw_checksums_s {
Cpa32U swCrcI;
Cpa32U swCrcO;
} dc_sw_checksums_t;
/* Session descriptor structure for compression */
typedef struct dc_session_desc_s {
Cpa8U stateRegistersComp[DC_QAT_STATE_REGISTERS_MAX_SIZE];
/**< State registers for compression */
Cpa8U stateRegistersDecomp[DC_QAT_STATE_REGISTERS_MAX_SIZE];
/**< State registers for decompression */
icp_qat_fw_comp_req_t reqCacheComp;
/**< Cache as much as possible of the compression request in a pre-built
* request */
icp_qat_fw_comp_req_t reqCacheDecomp;
/**< Cache as much as possible of the decompression request in a
* pre-built
* request */
dc_request_type_t requestType;
/**< Type of the compression request. As stateful mode do not support
* more
* than one in-flight request there is no need to use spinlocks */
dc_request_type_t previousRequestType;
/**< Type of the previous compression request. Used in cases where there
* the
* stateful operation needs to be resubmitted */
CpaDcHuffType huffType;
/**< Huffman tree type */
CpaDcCompType compType;
/**< Compression type */
CpaDcChecksum checksumType;
/**< Type of checksum */
CpaDcAutoSelectBest autoSelectBestHuffmanTree;
/**< Indicates if the implementation selects the best Huffman encoding
*/
CpaDcSessionDir sessDirection;
/**< Session direction */
CpaDcSessionState sessState;
/**< Session state */
Cpa32U deflateWindowSize;
/**< Window size */
CpaDcCompLvl compLevel;
/**< Compression level */
CpaDcCallbackFn pCompressionCb;
/**< Callback function defined for the traditional compression session
*/
QatUtilsAtomic pendingStatelessCbCount;
/**< Keeps track of number of pending requests on stateless session */
QatUtilsAtomic pendingStatefulCbCount;
/**< Keeps track of number of pending requests on stateful session */
Cpa64U pendingDpStatelessCbCount;
/**< Keeps track of number of data plane pending requests on stateless
* session */
struct mtx sessionLock;
/**< Lock used to provide exclusive access for number of stateful
* in-flight
* requests update */
CpaBoolean isDcDp;
/**< Indicates if the data plane API is used */
Cpa32U minContextSize;
/**< Indicates the minimum size required to allocate the context buffer
*/
CpaBufferList *pContextBuffer;
/**< Context buffer */
Cpa32U historyBuffSize;
/**< Size of the history buffer */
Cpa64U cumulativeConsumedBytes;
/**< Cumulative amount of consumed bytes. Used to build the footer in
* the
* stateful case */
Cpa32U previousChecksum;
/**< Save the previous value of the checksum. Used to process zero byte
* stateful compression or decompression requests */
CpaBoolean isSopForCompressionProcessed;
/**< Indicates whether a Compression Request is received in this session
*/
CpaBoolean isSopForDecompressionProcessed;
/**< Indicates whether a Decompression Request is received in this
* session
*/
/**< Data integrity table */
dc_integrity_crc_fw_t dataIntegrityCrcs;
/**< Physical address of Data integrity buffer */
CpaPhysicalAddr physDataIntegrityCrcs;
/* Seed checksums structure used to calculate software calculated
* checksums.
*/
dc_sw_checksums_t seedSwCrc;
/* Driver calculated integrity software CRC */
dc_sw_checksums_t integritySwCrc;
} dc_session_desc_t;
/**
*****************************************************************************
* @ingroup Dc_DataCompression
* Initialise a compression session
*
* @description
* This function will initialise a compression session
*
* @param[in] dcInstance Instance handle derived from discovery
* functions
* @param[in,out] pSessionHandle Pointer to a session handle
* @param[in,out] pSessionData Pointer to a user instantiated structure
* containing session data
* @param[in] pContextBuffer Pointer to context buffer
*
* @param[in] callbackFn For synchronous operation this callback
* shall be a null pointer
*
* @retval CPA_STATUS_SUCCESS Function executed successfully
* @retval CPA_STATUS_FAIL Function failed
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter passed in
* @retval CPA_STATUS_RESOURCE Error related to system resources
*****************************************************************************/
CpaStatus dcInitSession(CpaInstanceHandle dcInstance,
CpaDcSessionHandle pSessionHandle,
CpaDcSessionSetupData *pSessionData,
CpaBufferList *pContextBuffer,
CpaDcCallbackFn callbackFn);
/**
*****************************************************************************
* @ingroup Dc_DataCompression
* Get the size of the memory required to hold the session information
*
* @description
* This function will get the size of the memory required to hold the
* session information
*
* @param[in] dcInstance Instance handle derived from discovery
* functions
* @param[in] pSessionData Pointer to a user instantiated structure
* containing session data
* @param[out] pSessionSize On return, this parameter will be the size
* of the memory that will be
* required by cpaDcInitSession() for session
* data.
* @param[out] pContextSize On return, this parameter will be the size
* of the memory that will be required
* for context data. Context data is
* save/restore data including history and
* any implementation specific data that is
* required for a save/restore operation.
*
* @retval CPA_STATUS_SUCCESS Function executed successfully
* @retval CPA_STATUS_FAIL Function failed
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter passed in
*****************************************************************************/
CpaStatus dcGetSessionSize(CpaInstanceHandle dcInstance,
CpaDcSessionSetupData *pSessionData,
Cpa32U *pSessionSize,
Cpa32U *pContextSize);
#endif /* DC_SESSION_H */

81
sys/dev/qat/qat_api/common/compression/include/dc_stats.h Normal file
View File

@ -0,0 +1,81 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file dc_stats.h
*
* @ingroup Dc_DataCompression
*
* @description
* Definition of the Data Compression stats parameters.
*
*****************************************************************************/
#ifndef DC_STATS_H_
#define DC_STATS_H_
/* Number of Compression statistics */
#define COMPRESSION_NUM_STATS (sizeof(CpaDcStats) / sizeof(Cpa64U))
#define COMPRESSION_STAT_INC(statistic, pService) \
do { \
if (CPA_TRUE == \
pService->generic_service_info.stats->bDcStatsEnabled) { \
qatUtilsAtomicInc( \
&pService->pCompStatsArr[offsetof(CpaDcStats, \
statistic) / \
sizeof(Cpa64U)]); \
} \
} while (0)
/* Macro to get all Compression stats (from internal array of atomics) */
#define COMPRESSION_STATS_GET(compStats, pService) \
do { \
int i; \
for (i = 0; i < COMPRESSION_NUM_STATS; i++) { \
((Cpa64U *)compStats)[i] = \
qatUtilsAtomicGet(&pService->pCompStatsArr[i]); \
} \
} while (0)
/* Macro to reset all Compression stats */
#define COMPRESSION_STATS_RESET(pService) \
do { \
int i; \
for (i = 0; i < COMPRESSION_NUM_STATS; i++) { \
qatUtilsAtomicSet(0, &pService->pCompStatsArr[i]); \
} \
} while (0)
/**
*******************************************************************************
* @ingroup Dc_DataCompression
* Initialises the compression stats
*
* @description
* This function allocates and initialises the stats array to 0
*
* @param[in] pService Pointer to a compression service structure
*
* @retval CPA_STATUS_SUCCESS initialisation successful
* @retval CPA_STATUS_RESOURCE array allocation failed
*
*****************************************************************************/
CpaStatus dcStatsInit(sal_compression_service_t *pService);
/**
*******************************************************************************
* @ingroup Dc_DataCompression
* Frees the compression stats
*
* @description
* This function frees the stats array
*
* @param[in] pService Pointer to a compression service structure
*
* @retval None
*
*****************************************************************************/
void dcStatsFree(sal_compression_service_t *pService);
#endif /* DC_STATS_H_ */

622
sys/dev/qat/qat_api/common/crypto/sym/include/lac_session.h Normal file
View File

@ -0,0 +1,622 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file lac_session.h
*
* @defgroup LacSym_Session Session
*
* @ingroup LacSym
*
* Definition of symmetric session descriptor structure
*
* @lld_start
*
* @lld_overview
* A session is required for each symmetric operation. The session descriptor
* holds information about the session from when the session is initialised to
* when the session is removed. The session descriptor is used in the
* subsequent perform operations in the paths for both sending the request and
* receiving the response. The session descriptor and any other state
* information required for processing responses from the QAT are stored in an
* internal cookie. A pointer to this cookie is stored in the opaque data
* field of the QAT request.
*
* The user allocates the memory for the session using the size returned from
* \ref cpaCySymSessionCtxGetSize(). Internally this memory is re-aligned on a
* 64 byte boundary for use by the QAT engine. The aligned pointer is saved in
* the first bytes (size of void *) of the session memory. This address
* is then dereferenced in subsequent performs to get access to the session
* descriptor.
*
* <b>LAC Session Init</b>\n The session descriptor is re-aligned and
* populated. This includes populating the content descriptor which contains
* the hardware setup for the QAT engine. The content descriptor is a read
* only structure after session init and a pointer to it is sent to the QAT
* for each perform operation.
*
* <b>LAC Perform </b>\n
* The address for the session descriptor is got by dereferencing the first
* bytes of the session memory (size of void *). For each successful
* request put on the ring, the pendingCbCount for the session is incremented.
*
* <b>LAC Callback </b>\n
* For each successful response the pendingCbCount for the session is
* decremented. See \ref LacSymCb_ProcessCallbackInternal()
*
* <b>LAC Session Remove </b>\n
* The address for the session descriptor is got by dereferencing the first
* bytes of the session memory (size of void *).
* The pendingCbCount for the session is checked to see if it is 0. If it is
* non 0 then there are requests in flight. An error is returned to the user.
*
* <b>Concurrency</b>\n
* A reference count is used to prevent the descriptor being removed
* while there are requests in flight.
*
* <b>Reference Count</b>\n
* - The perform funcion increments the reference count for the session.
* - The callback function decrements the reference count for the session.
* - The Remove function checks the reference count to ensure that it is 0.
*
* @lld_dependencies
* - \ref LacMem "Memory" - Inline memory functions
* - QatUtils: logging, locking & virt to phys translations.
*
* @lld_initialisation
*
* @lld_module_algorithms
*
* @lld_process_context
*
* @lld_end
*
*****************************************************************************/
/***************************************************************************/
#ifndef LAC_SYM_SESSION_H
#define LAC_SYM_SESSION_H
/*
* Common alignment attributes to ensure
* hashStatePrefixBuffer is 64-byte aligned
*/
#define ALIGN_START(x)
#define ALIGN_END(x) __attribute__((__aligned__(x)))
/*
******************************************************************************
* Include public/global header files
******************************************************************************
*/
#include "cpa.h"
#include "icp_accel_devices.h"
#include "lac_list.h"
#include "lac_sal_types.h"
#include "sal_qat_cmn_msg.h"
#include "lac_sym_cipher_defs.h"
#include "lac_sym.h"
#include "lac_sym_hash_defs.h"
#include "lac_sym_qat_hash.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
/**
*******************************************************************************
* @ingroup LacSym_Session
* Symmetric session descriptor
* @description
* This structure stores information about a session
* Note: struct types lac_session_d1_s and lac_session_d2_s are subsets of
* this structure. Elements in all three should retain the same order
* Only this structure is used in the session init call. The other two are
* for determining the size of memory to allocate.
* The comments section of each of the other two structures below show
* the conditions that determine which session context memory size to use.
*****************************************************************************/
typedef struct lac_session_desc_s {
Cpa8U contentDescriptor[LAC_SYM_QAT_CONTENT_DESC_MAX_SIZE];
/**< QAT Content Descriptor for this session.
* NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
Cpa8U contentDescriptorOptimised[LAC_SYM_OPTIMISED_CD_SIZE];
/**< QAT Optimised Content Descriptor for this session.
* NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
CpaCySymOp symOperation;
/**< type of command to be performed */
sal_qat_content_desc_info_t contentDescInfo;
/**< info on the content descriptor */
sal_qat_content_desc_info_t contentDescOptimisedInfo;
/**< info on the optimised content descriptor */
icp_qat_fw_la_cmd_id_t laCmdId;
/**<Command Id for the QAT FW */
lac_sym_qat_hash_state_buffer_info_t hashStateBufferInfo;
/**< info on the hash state prefix buffer */
CpaCySymHashAlgorithm hashAlgorithm;
/**< hash algorithm */
Cpa32U authKeyLenInBytes;
/**< Authentication key length in bytes */
CpaCySymHashMode hashMode;
/**< Mode of the hash operation. plain, auth or nested */
Cpa32U hashResultSize;
/**< size of the digest produced/compared in bytes */
CpaCySymCipherAlgorithm cipherAlgorithm;
/**< Cipher algorithm and mode */
Cpa32U cipherKeyLenInBytes;
/**< Cipher key length in bytes */
CpaCySymCipherDirection cipherDirection;
/**< This parameter determines if the cipher operation is an encrypt or
* a decrypt operation. */
CpaCySymPacketType partialState;
/**< state of the partial packet. This can be written to by the perform
* because the SpinLock pPartialInFlightSpinlock guarantees that the
* state is accessible in only one place at a time. */
icp_qat_la_bulk_req_hdr_t reqCacheHdr;
icp_qat_fw_la_key_gen_common_t reqCacheMid;
icp_qat_la_bulk_req_ftr_t reqCacheFtr;
/**< Cache as much as possible of the bulk request in a pre built
* request (header, mid & footer). */
CpaCySymCbFunc pSymCb;
/**< symmetric function callback pointer */
union {
QatUtilsAtomic pendingCbCount;
/**< Keeps track of number of pending requests. */
QatUtilsAtomic pendingDpCbCount;
/**< Keeps track of number of pending DP requests (not thread
* safe)*/
} u;
struct lac_sym_bulk_cookie_s *pRequestQueueHead;
/**< A fifo list of queued QAT requests. Head points to first queue
* entry */
struct lac_sym_bulk_cookie_s *pRequestQueueTail;
/**< A fifo list of queued QAT requests. Tail points to last queue entry
*/
struct mtx requestQueueLock;
/**< A lock to protect accesses to the above request queue */
CpaInstanceHandle pInstance;
/**< Pointer to Crypto instance running this session. */
CpaBoolean isAuthEncryptOp : 1;
/**< if the algorithm chaining operation is auth encrypt */
CpaBoolean nonBlockingOpsInProgress : 1;
/**< Flag is set if a non blocking operation is in progress for a
* session.
* If set to false, new requests will be queued until the condition is
* cleared.
* ASSUMPTION: Only one blocking condition per session can exist at any
* time
*/
CpaBoolean internalSession : 1;
/**< Flag which is set if the session was set up internally for DRBG */
CpaBoolean isDPSession : 1;
/**< Flag which is set if the session was set up for Data Plane */
CpaBoolean digestVerify : 1;
/**< Session digest verify for data plane and for CCM/GCM for trad
* api. For other cases on trad api this flag is set in each performOp
*/
CpaBoolean digestIsAppended : 1;
/**< Flag indicating whether the digest is appended immediately
* following
* the region over which the digest is computed */
CpaBoolean isCipher : 1;
/**< Flag indicating whether symOperation includes a cipher operation */
CpaBoolean isAuth : 1;
/**< Flag indicating whether symOperation includes an auth operation */
CpaBoolean useSymConstantsTable : 1;
/**< Flag indicating whether the SymConstantsTable can be used or not */
CpaBoolean useOptimisedContentDesc : 1;
/**< Flag indicating whether to use the optimised CD or not */
icp_qat_la_bulk_req_hdr_t shramReqCacheHdr;
icp_qat_fw_la_key_gen_common_t shramReqCacheMid;
icp_qat_la_bulk_req_ftr_t shramReqCacheFtr;
/**< Alternative pre-built request (header, mid & footer)
* for use with symConstantsTable. */
CpaBoolean isPartialSupported : 1;
/**< Flag indicating whether symOperation support partial packet */
CpaBoolean isSinglePass : 1;
/**< Flag indicating whether symOperation is single pass operation */
icp_qat_fw_serv_specif_flags laCmdFlags;
/**< Common request - Service specific flags type */
icp_qat_fw_comn_flags cmnRequestFlags;
/**< Common request flags type */
icp_qat_fw_ext_serv_specif_flags laExtCmdFlags;
/**< Common request - Service specific flags type */
icp_qat_la_bulk_req_hdr_t reqSpcCacheHdr;
icp_qat_la_bulk_req_ftr_t reqSpcCacheFtr;
/**< request (header & footer)for use with Single Pass. */
icp_qat_hw_auth_mode_t qatHashMode;
/**< Hash Mode for the qat slices. Not to be confused with QA-API
* hashMode
*/
void *writeRingMsgFunc;
/**< function which will be called to write ring message */
Cpa32U aadLenInBytes;
/**< For CCM,GCM and Snow3G cases, this parameter holds the AAD size,
* otherwise it is set to zero */
ALIGN_START(64)
Cpa8U hashStatePrefixBuffer[LAC_MAX_AAD_SIZE_BYTES] ALIGN_END(64);
/**< hash state prefix buffer used for hash operations - AAD only
* NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
Cpa8U hashStatePrefixBufferExt[LAC_MAX_HASH_STATE_BUFFER_SIZE_BYTES -
LAC_MAX_AAD_SIZE_BYTES];
/**< hash state prefix buffer used for hash operations - Remainder of
* array.
* NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
Cpa8U cipherPartialOpState[LAC_CIPHER_STATE_SIZE_MAX];
/**< Buffer to hold the cipher state for the session (for partial ops).
* NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
Cpa8U cipherARC4InitialState[LAC_CIPHER_ARC4_STATE_LEN_BYTES];
/**< Buffer to hold the initial ARC4 cipher state for the session, which
* is derived from the user-supplied base key during session
* registration.
* NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
CpaPhysicalAddr cipherARC4InitialStatePhysAddr;
/**< The physical address of the ARC4 initial state, set at init
** session time .
*/
} lac_session_desc_t;
/**
*******************************************************************************
* @ingroup LacSym_Session
* Symmetric session descriptor - d1
* @description
* This structure stores information about a specific session which
* assumes the following:
* - cipher algorithm is not ARC4 or Snow3G
* - partials not used
* - not AuthEncrypt operation
* - hash mode not Auth or Nested
* - no hashStatePrefixBuffer required
* It is therefore a subset of the standard symmetric session descriptor,
* with a smaller memory footprint
*****************************************************************************/
typedef struct lac_session_desc_d1_s {
Cpa8U contentDescriptor[LAC_SYM_QAT_CONTENT_DESC_MAX_SIZE];
/**< QAT Content Descriptor for this session.
* NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
Cpa8U contentDescriptorOptimised[LAC_SYM_OPTIMISED_CD_SIZE];
/**< QAT Optimised Content Descriptor for this session.
* NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
CpaCySymOp symOperation;
/**< type of command to be performed */
sal_qat_content_desc_info_t contentDescInfo;
/**< info on the content descriptor */
sal_qat_content_desc_info_t contentDescOptimisedInfo;
/**< info on the optimised content descriptor */
icp_qat_fw_la_cmd_id_t laCmdId;
/**<Command Id for the QAT FW */
lac_sym_qat_hash_state_buffer_info_t hashStateBufferInfo;
/**< info on the hash state prefix buffer */
CpaCySymHashAlgorithm hashAlgorithm;
/**< hash algorithm */
Cpa32U authKeyLenInBytes;
/**< Authentication key length in bytes */
CpaCySymHashMode hashMode;
/**< Mode of the hash operation. plain, auth or nested */
Cpa32U hashResultSize;
/**< size of the digest produced/compared in bytes */
CpaCySymCipherAlgorithm cipherAlgorithm;
/**< Cipher algorithm and mode */
Cpa32U cipherKeyLenInBytes;
/**< Cipher key length in bytes */
CpaCySymCipherDirection cipherDirection;
/**< This parameter determines if the cipher operation is an encrypt or
* a decrypt operation. */
CpaCySymPacketType partialState;
/**< state of the partial packet. This can be written to by the perform
* because the SpinLock pPartialInFlightSpinlock guarantees that that
* the
* state is accessible in only one place at a time. */
icp_qat_la_bulk_req_hdr_t reqCacheHdr;
icp_qat_fw_la_key_gen_common_t reqCacheMid;
icp_qat_la_bulk_req_ftr_t reqCacheFtr;
/**< Cache as much as possible of the bulk request in a pre built
* request (header, mid & footer). */
CpaCySymCbFunc pSymCb;
/**< symmetric function callback pointer */
union {
QatUtilsAtomic pendingCbCount;
/**< Keeps track of number of pending requests. */
Cpa64U pendingDpCbCount;
/**< Keeps track of number of pending DP requests (not thread
* safe)*/
} u;
struct lac_sym_bulk_cookie_s *pRequestQueueHead;
/**< A fifo list of queued QAT requests. Head points to first queue
* entry */
struct lac_sym_bulk_cookie_s *pRequestQueueTail;
/**< A fifo list of queued QAT requests. Tail points to last queue entry
*/
struct mtx requestQueueLock;
/**< A lock to protect accesses to the above request queue */
CpaInstanceHandle pInstance;
/**< Pointer to Crypto instance running this session. */
CpaBoolean isAuthEncryptOp : 1;
/**< if the algorithm chaining operation is auth encrypt */
CpaBoolean nonBlockingOpsInProgress : 1;
/**< Flag is set if a non blocking operation is in progress for a
* session.
* If set to false, new requests will be queued until the condition is
* cleared.
* ASSUMPTION: Only one blocking condition per session can exist at any
* time
*/
CpaBoolean internalSession : 1;
/**< Flag which is set if the session was set up internally for DRBG */
CpaBoolean isDPSession : 1;
/**< Flag which is set if the session was set up for Data Plane */
CpaBoolean digestVerify : 1;
/**< Session digest verify for data plane and for CCM/GCM for trad
* api. For other cases on trad api this flag is set in each performOp
*/
CpaBoolean digestIsAppended : 1;
/**< Flag indicating whether the digest is appended immediately
* following
* the region over which the digest is computed */
CpaBoolean isCipher : 1;
/**< Flag indicating whether symOperation includes a cipher operation */
CpaBoolean isAuth : 1;
/**< Flag indicating whether symOperation includes an auth operation */
CpaBoolean useSymConstantsTable : 1;
/**< Flag indicating whether the SymConstantsTable can be used or not */
CpaBoolean useOptimisedContentDesc : 1;
/**< Flag indicating whether to use the optimised CD or not */
icp_qat_la_bulk_req_hdr_t shramReqCacheHdr;
icp_qat_fw_la_key_gen_common_t shramReqCacheMid;
icp_qat_la_bulk_req_ftr_t shramReqCacheFtr;
/**< Alternative pre-built request (header, mid & footer)
* for use with symConstantsTable. */
CpaBoolean isPartialSupported : 1;
/**< Flag indicating whether symOperation support partial packet */
CpaBoolean isSinglePass : 1;
/**< Flag indicating whether symOperation is single pass operation */
icp_qat_fw_serv_specif_flags laCmdFlags;
/**< Common request - Service specific flags type */
icp_qat_fw_comn_flags cmnRequestFlags;
/**< Common request flags type */
icp_qat_fw_ext_serv_specif_flags laExtCmdFlags;
/**< Common request - Service specific flags type */
icp_qat_la_bulk_req_hdr_t reqSpcCacheHdr;
icp_qat_la_bulk_req_ftr_t reqSpcCacheFtr;
/**< request (header & footer)for use with Single Pass. */
icp_qat_hw_auth_mode_t qatHashMode;
/**< Hash Mode for the qat slices. Not to be confused with QA-API
* hashMode
*/
void *writeRingMsgFunc;
/**< function which will be called to write ring message */
} lac_session_desc_d1_t;
/**
*******************************************************************************
* @ingroup LacSym_Session
* Symmetric session descriptor - d2
* @description
* This structure stores information about a specific session which
* assumes the following:
* - authEncrypt only
* - partials not used
* - hasStatePrefixBuffer just contains AAD
* It is therefore a subset of the standard symmetric session descriptor,
* with a smaller memory footprint
*****************************************************************************/
typedef struct lac_session_desc_d2_s {
Cpa8U contentDescriptor[LAC_SYM_QAT_CONTENT_DESC_MAX_SIZE];
/**< QAT Content Descriptor for this session.
* NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
Cpa8U contentDescriptorOptimised[LAC_SYM_OPTIMISED_CD_SIZE];
/**< QAT Optimised Content Descriptor for this session.
* NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
CpaCySymOp symOperation;
/**< type of command to be performed */
sal_qat_content_desc_info_t contentDescInfo;
/**< info on the content descriptor */
sal_qat_content_desc_info_t contentDescOptimisedInfo;
/**< info on the optimised content descriptor */
icp_qat_fw_la_cmd_id_t laCmdId;
/**<Command Id for the QAT FW */
lac_sym_qat_hash_state_buffer_info_t hashStateBufferInfo;
/**< info on the hash state prefix buffer */
CpaCySymHashAlgorithm hashAlgorithm;
/**< hash algorithm */
Cpa32U authKeyLenInBytes;
/**< Authentication key length in bytes */
CpaCySymHashMode hashMode;
/**< Mode of the hash operation. plain, auth or nested */
Cpa32U hashResultSize;
/**< size of the digest produced/compared in bytes */
CpaCySymCipherAlgorithm cipherAlgorithm;
/**< Cipher algorithm and mode */
Cpa32U cipherKeyLenInBytes;
/**< Cipher key length in bytes */
CpaCySymCipherDirection cipherDirection;
/**< This parameter determines if the cipher operation is an encrypt or
* a decrypt operation. */
CpaCySymPacketType partialState;
/**< state of the partial packet. This can be written to by the perform
* because the SpinLock pPartialInFlightSpinlock guarantees that that
* the
* state is accessible in only one place at a time. */
icp_qat_la_bulk_req_hdr_t reqCacheHdr;
icp_qat_fw_la_key_gen_common_t reqCacheMid;
icp_qat_la_bulk_req_ftr_t reqCacheFtr;
/**< Cache as much as possible of the bulk request in a pre built
* request (header. mid & footer). */
CpaCySymCbFunc pSymCb;
/**< symmetric function callback pointer */
union {
QatUtilsAtomic pendingCbCount;
/**< Keeps track of number of pending requests. */
Cpa64U pendingDpCbCount;
/**< Keeps track of number of pending DP requests (not thread
* safe)*/
} u;
struct lac_sym_bulk_cookie_s *pRequestQueueHead;
/**< A fifo list of queued QAT requests. Head points to first queue
* entry */
struct lac_sym_bulk_cookie_s *pRequestQueueTail;
/**< A fifo list of queued QAT requests. Tail points to last queue entry
*/
struct mtx requestQueueLock;
/**< A lock to protect accesses to the above request queue */
CpaInstanceHandle pInstance;
/**< Pointer to Crypto instance running this session. */
CpaBoolean isAuthEncryptOp : 1;
/**< if the algorithm chaining operation is auth encrypt */
CpaBoolean nonBlockingOpsInProgress : 1;
/**< Flag is set if a non blocking operation is in progress for a
* session.
* If set to false, new requests will be queued until the condition is
* cleared.
* ASSUMPTION: Only one blocking condition per session can exist at any
* time
*/
CpaBoolean internalSession : 1;
/**< Flag which is set if the session was set up internally for DRBG */
CpaBoolean isDPSession : 1;
/**< Flag which is set if the session was set up for Data Plane */
CpaBoolean digestVerify : 1;
/**< Session digest verify for data plane and for CCM/GCM for trad
* api. For other cases on trad api this flag is set in each performOp
*/
CpaBoolean digestIsAppended : 1;
/**< Flag indicating whether the digest is appended immediately
* following
* the region over which the digest is computed */
CpaBoolean isCipher : 1;
/**< Flag indicating whether symOperation includes a cipher operation */
CpaBoolean isAuth : 1;
/**< Flag indicating whether symOperation includes an auth operation */
CpaBoolean useSymConstantsTable : 1;
/**< Flag indicating whether the SymConstantsTable can be used or not */
CpaBoolean useOptimisedContentDesc : 1;
/**< Flag indicating whether to use the optimised CD or not */
icp_qat_la_bulk_req_hdr_t shramReqCacheHdr;
icp_qat_fw_la_key_gen_common_t shramReqCacheMid;
icp_qat_la_bulk_req_ftr_t shramReqCacheFtr;
/**< Alternative pre-built request (header. mid & footer)
* for use with symConstantsTable. */
CpaBoolean isPartialSupported : 1;
/**< Flag indicating whether symOperation support partial packet */
CpaBoolean isSinglePass : 1;
/**< Flag indicating whether symOperation is single pass operation */
icp_qat_fw_serv_specif_flags laCmdFlags;
/**< Common request - Service specific flags type */
icp_qat_fw_comn_flags cmnRequestFlags;
/**< Common request flags type */
icp_qat_fw_ext_serv_specif_flags laExtCmdFlags;
/**< Common request - Service specific flags type */
icp_qat_la_bulk_req_hdr_t reqSpcCacheHdr;
icp_qat_la_bulk_req_ftr_t reqSpcCacheFtr;
/**< request (header & footer)for use with Single Pass. */
icp_qat_hw_auth_mode_t qatHashMode;
/**< Hash Mode for the qat slices. Not to be confused with QA-API
* hashMode
*/
void *writeRingMsgFunc;
/**< function which will be called to write ring message */
Cpa32U aadLenInBytes;
/**< For CCM,GCM and Snow3G cases, this parameter holds the AAD size,
* otherwise it is set to zero */
ALIGN_START(64)
Cpa8U hashStatePrefixBuffer[LAC_MAX_AAD_SIZE_BYTES] ALIGN_END(64);
/**< hash state prefix buffer used for hash operations - AAD only
* NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
} lac_session_desc_d2_t;
#define LAC_SYM_SESSION_SIZE \
(sizeof(lac_session_desc_t) + LAC_64BYTE_ALIGNMENT + \
sizeof(LAC_ARCH_UINT))
/**< @ingroup LacSym_Session
* Size of the memory that the client has to allocate for a session. Extra
* memory is needed to internally re-align the data. The pointer to the algined
* data is stored at the start of the user allocated memory hence the extra
* space for an LAC_ARCH_UINT */
#define LAC_SYM_SESSION_D1_SIZE \
(sizeof(lac_session_desc_d1_t) + LAC_64BYTE_ALIGNMENT + \
sizeof(LAC_ARCH_UINT))
/**< @ingroup LacSym_Session
** Size of the memory that the client has to allocate for a session where :
* - cipher algorithm not ARC4 or Snow3G, no Partials, nonAuthEncrypt.
* Extra memory is needed to internally re-align the data. The pointer to the
* aligned data is stored at the start of the user allocated memory hence the
* extra space for an LAC_ARCH_UINT */
#define LAC_SYM_SESSION_D2_SIZE \
(sizeof(lac_session_desc_d2_t) + LAC_64BYTE_ALIGNMENT + \
sizeof(LAC_ARCH_UINT))
/**< @ingroup LacSym_Session
** Size of the memory that the client has to allocate for a session where :
* - authEncrypt, no Partials - so hashStatePrefixBuffer is only AAD
* Extra memory is needed to internally re-align the data. The pointer to the
* aligned data is stored at the start of the user allocated memory hence the
* extra space for an LAC_ARCH_UINT */
#define LAC_SYM_SESSION_DESC_FROM_CTX_GET(pSession) \
(lac_session_desc_t *)(*(LAC_ARCH_UINT *)pSession)
/**< @ingroup LacSym_Session
* Retrieve the session descriptor pointer from the session context structure
* that the user allocates. The pointer to the internally realigned address
* is stored at the start of the session context that the user allocates */
/**
*******************************************************************************
* @ingroup LacSym_Session
* This function initializes a session
*
* @description
* This function is called from the LAC session register API functions.
* It validates all input parameters. If an invalid parameter is passed,
* an error is returned to the calling function. If all parameters are valid
* a symmetric session is initialized
*
* @param[in] instanceHandle_in Instance Handle
* @param[in] pSymCb callback function
* @param[in] pSessionSetupData pointer to the strucutre containing the setup
*data
* @param[in] isDpSession CPA_TRUE for a data plane session
* @param[out] pSessionCtx Pointer to session context
*
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_FAIL Function failed.
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter passed in.
* @retval CPA_STATUS_RESOURCE Error related to system resources.
*
*/
CpaStatus LacSym_InitSession(const CpaInstanceHandle instanceHandle_in,
const CpaCySymCbFunc pSymCb,
const CpaCySymSessionSetupData *pSessionSetupData,
const CpaBoolean isDpSession,
CpaCySymSessionCtx pSessionCtx);
#endif /* LAC_SYM_SESSION_H */

356
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym.h Normal file
View File

@ -0,0 +1,356 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym.h
*
* @defgroup LacSym Symmetric
*
* @ingroup Lac
*
* Symmetric component includes cipher, Hash, chained cipher & hash,
* authenticated encryption and key generation.
*
* @lld_start
* @lld_overview
*
* The symmetric component demuliplexes the following crypto operations to
* the appropriate sub-components: cipher, hash, algorithm chaining and
* authentication encryption. It is a common layer between the above
* mentioned components where common resources are allocated and paramater
* checks are done. The operation specific resource allocation and parameter
* checks are done in the sub-component itself.
*
* The symmetric component demultiplexes the session register/deregister
* and perform functions to the appropriate subcomponents.
*
* @lld_dependencies
* - \ref LacSymPartial "Partial Packet Code": This code manages the partial
* packet state for a session.
* - \ref LacBufferDesc "Common Buffer Code" : This code traverses a buffer
* chain to ensure it is valid.
* - \ref LacSymStats "Statistics": Manages statistics for symmetric
* - \ref LacSymQat "Symmetric QAT": The symmetric qat component is
* initialiased by the symmetric component.
* - \ref LacCipher "Cipher" : demultiplex cipher opertions to this component.
* - \ref LacHash "Hash" : demultiplex hash opertions to this component.
* to this component.
* - \ref LacAlgChain "Algorithm Chaining": The algorithm chaining component
* - OSAL : Memory allocation, Mutex's, atomics
*
* @lld_initialisation
* This component is initialied during the LAC initialisation sequence. It
* initialises the session table, statistics, symmetric QAT, initialises the
* hash definitions lookup table, the hash alg supported lookup table and
* registers a callback function with the symmetric response handler to process
* response messages for Cipher, Hash and Algorithm-Chaining requests.
*
* @lld_module_algorithms
*
* @lld_process_context
* Refer to \ref LacHash "Hash" and \ref LacCipher "Cipher" for sequence
* diagrams from the symmetric component through the sub components.
*
* @lld_end
*
***************************************************************************/
/***************************************************************************/
#ifndef LAC_SYM_H
#define LAC_SYM_H
#include "cpa.h"
#include "cpa_cy_sym.h"
#include "cpa_cy_sym_dp.h"
#include "lac_common.h"
#include "lac_mem_pools.h"
#include "lac_sym_cipher_defs.h"
#include "icp_qat_fw_la.h"
#define LAC_SYM_KEY_TLS_PREFIX_SIZE 128
/**< Hash Prefix size in bytes for TLS (128 = MAX = SHA2 (384, 512)*/
#define LAC_SYM_OPTIMISED_CD_SIZE 64
/**< The size of the optimised content desc in DRAM*/
#define LAC_SYM_KEY_MAX_HASH_STATE_BUFFER (LAC_SYM_KEY_TLS_PREFIX_SIZE * 2)
/**< hash state prefix buffer structure that holds the maximum sized secret */
#define LAC_SYM_HASH_BUFFER_LEN 64
/**< Buffer length to hold 16 byte MD5 key and 20 byte SHA1 key */
/* The ARC4 key will not be stored in the content descriptor so we only need to
* reserve enough space for the next biggest cipher setup block.
* Kasumi needs to store 2 keys and to have the size of 2 blocks for fw*/
#define LAC_SYM_QAT_MAX_CIPHER_SETUP_BLK_SZ \
(sizeof(icp_qat_hw_cipher_config_t) + 2 * ICP_QAT_HW_KASUMI_KEY_SZ + \
2 * ICP_QAT_HW_KASUMI_BLK_SZ)
/**< @ingroup LacSymQat
* Maximum size for the cipher setup block of the content descriptor */
#define LAC_SYM_QAT_MAX_HASH_SETUP_BLK_SZ sizeof(icp_qat_hw_auth_algo_blk_t)
/**< @ingroup LacSymQat
* Maximum size for the hash setup block of the content descriptor */
#define LAC_SYM_QAT_CONTENT_DESC_MAX_SIZE \
LAC_ALIGN_POW2_ROUNDUP(LAC_SYM_QAT_MAX_CIPHER_SETUP_BLK_SZ + \
LAC_SYM_QAT_MAX_HASH_SETUP_BLK_SZ, \
(1 << LAC_64BYTE_ALIGNMENT_SHIFT))
/**< @ingroup LacSymQat
* Maximum size of content descriptor. This is incremented to the next multiple
* of 64 so that it can be 64 byte aligned */
#define LAC_SYM_QAT_API_ALIGN_COOKIE_OFFSET \
(offsetof(CpaCySymDpOpData, instanceHandle))
/**< @ingroup LacSymQat
* Size which needs to be reserved before the instanceHandle field of
* lac_sym_bulk_cookie_s to align it to the correspondent instanceHandle
* in CpaCySymDpOpData */
#define LAC_SIZE_OF_CACHE_HDR_IN_LW 6
/**< Size of Header part of reqCache/shramReqCache */
#define LAC_SIZE_OF_CACHE_MID_IN_LW 2
/**< Size of Mid part (LW14/15) of reqCache/shramReqCache */
#define LAC_SIZE_OF_CACHE_FTR_IN_LW 6
/**< Size of Footer part of reqCache/shramReqCache */
#define LAC_SIZE_OF_CACHE_TO_CLEAR_IN_LW 20
/**< Size of dummy reqCache/shramReqCache to clear */
#define LAC_START_OF_CACHE_MID_IN_LW 14
/**< Starting LW of reqCache/shramReqCache Mid */
#define LAC_START_OF_CACHE_FTR_IN_LW 26
/**< Starting LW of reqCache/shramReqCache Footer */
/**
*******************************************************************************
* @ingroup LacSym
* Symmetric cookie
*
* @description
* This cookie stores information for a particular symmetric perform op.
* This includes the request params, re-aligned Cipher IV, the request
* message sent to the QAT engine, and various user-supplied parameters
* for the operation which will be needed in our callback function.
* A pointer to this cookie is stored in the opaque data field of the QAT
* message so that it can be accessed in the asynchronous callback.
* Cookies for multiple operations on a given session can be linked
* together to allow queuing of requests using the pNext field.
*
* The parameters are placed in order to match the CpaCySymDpOpData
*structure
*****************************************************************************/
typedef struct lac_sym_bulk_cookie_s {
/* CpaCySymDpOpData struct so need to keep this here for correct
* alignment*/
Cpa8U reserved[LAC_SYM_QAT_API_ALIGN_COOKIE_OFFSET];
/** NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
CpaInstanceHandle instanceHandle;
/**< Instance handle for the operation */
CpaCySymSessionCtx sessionCtx;
/**< Session context */
void *pCallbackTag;
/**< correlator supplied by the client */
icp_qat_fw_la_bulk_req_t qatMsg;
/**< QAT request message */
const CpaCySymOpData *pOpData;
/**< pointer to the op data structure that the user supplied in the
* perform
* operation. The op data is modified in the process callback function
* and the pointer is returned to the user in their callback function */
CpaBoolean updateSessionIvOnSend;
/**< Boolean flag to indicate if the session cipher IV buffer should be
* updated prior to sending the request */
CpaBoolean updateUserIvOnRecieve;
/**< Boolean flag to indicate if the user's cipher IV buffer should be
* updated after receiving the response from the QAT */
CpaBoolean updateKeySizeOnRecieve;
/**< Boolean flag to indicate if the cipher key size should be
* updated after receiving the response from the QAT */
CpaBufferList *pDstBuffer;
/**< Pointer to destination buffer to hold the data output */
struct lac_sym_bulk_cookie_s *pNext;
/**< Pointer to next node in linked list (if request is queued) */
} lac_sym_bulk_cookie_t;
/**
*******************************************************************************
* @ingroup LacSymKey
* symmetric Key cookie
* @description
* This cookie stores information for a particular keygen perform op.
* This includes a hash content descriptor, request params, hash state
* buffer, and various user-supplied parameters for the operation which
* will be needed in our callback function.
* A pointer to this cookie is stored in the opaque data field of the QAT
* message so that it can be accessed in the asynchronous callback.
*****************************************************************************/
typedef struct lac_sym_key_cookie_s {
CpaInstanceHandle instanceHandle;
/**< QAT device id supplied by the client */
void *pCallbackTag;
/**< Mechanism used. TLS, SSL or MGF */
Cpa8U contentDesc[LAC_SYM_QAT_MAX_HASH_SETUP_BLK_SZ];
/**< Content descriptor.
**< NOTE: Field must be correctly aligned in memory for access by QAT
* engine */
union {
icp_qat_fw_la_ssl_key_material_input_t sslKeyInput;
/**< SSL key material input structure */
icp_qat_fw_la_tls_key_material_input_t tlsKeyInput;
/**< TLS key material input structure */
icp_qat_fw_la_hkdf_key_material_input_t tlsHKDFKeyInput;
/**< TLS HHKDF key material input structure */
} u;
/**< NOTE: Field must be correctly aligned in memory for access by QAT
* engine */
Cpa8U hashStateBuffer[LAC_SYM_KEY_MAX_HASH_STATE_BUFFER];
/**< hash state prefix buffer
* NOTE: Field must be correctly aligned in memory for access by QAT
* engine
*/
CpaCyGenFlatBufCbFunc pKeyGenCb;
/**< callback function supplied by the client */
void *pKeyGenOpData;
/**< pointer to the (SSL/TLS) or MGF op data structure that the user
* supplied in the perform operation */
CpaFlatBuffer *pKeyGenOutputData;
/**< Output data pointer supplied by the client */
Cpa8U hashKeyBuffer[LAC_SYM_HASH_BUFFER_LEN];
/**< 36 byte buffer to store MD5 key and SHA1 key */
} lac_sym_key_cookie_t;
/**
*******************************************************************************
* @ingroup LacSymNrbg
* symmetric NRBG cookie
* @description
* This cookie stores information for a particular NRBG operation.
* This includes various user-supplied parameters for the operation which
* will be needed in our callback function.
* A pointer to this cookie is stored in the opaque data field of the QAT
* message so that it can be accessed in the asynchronous callback.
*****************************************************************************/
typedef struct lac_sym_nrbg_cookie_s {
CpaInstanceHandle instanceHandle;
/**< QAT device id supplied by the client */
void *pCallbackTag;
/**< Opaque data supplied by the client */
icp_qat_fw_la_trng_test_result_t trngHTResult;
/**< TRNG health test result
**< NOTE: Field must be correctly aligned in memory for access by QAT
* engine */
icp_qat_fw_la_trng_req_t trngReq;
/**< TRNG request message */
CpaCyGenFlatBufCbFunc pCb;
/**< Callback function supplied by the client */
void *pOpData;
/**< Op data pointer supplied by the client */
CpaFlatBuffer *pOutputData;
/**< Output data pointer supplied by the client */
} lac_sym_nrbg_cookie_t;
/**
*******************************************************************************
* @ingroup LacSym
* symmetric cookie
* @description
* used to determine the amount of memory to allocate for the symmetric
* cookie pool. As symmetric, random and key generation shared the same
* pool
*****************************************************************************/
typedef struct lac_sym_cookie_s {
union {
lac_sym_bulk_cookie_t bulkCookie;
/**< symmetric bulk cookie */
lac_sym_key_cookie_t keyCookie;
/**< symmetric key cookie */
lac_sym_nrbg_cookie_t nrbgCookie;
/**< symmetric NRBG cookie */
} u;
Cpa64U keyContentDescPhyAddr;
Cpa64U keyHashStateBufferPhyAddr;
Cpa64U keySslKeyInputPhyAddr;
Cpa64U keyTlsKeyInputPhyAddr;
} lac_sym_cookie_t;
typedef struct icp_qat_la_auth_req_params_s {
/** equivalent of LW26 of icp_qat_fw_la_auth_req_params_s */
union {
uint8_t inner_prefix_sz;
/**< Size in bytes of the inner prefix data */
uint8_t aad_sz;
/**< Size in bytes of padded AAD data to prefix to the packet
* for CCM
* or GCM processing */
} u2;
uint8_t resrvd1;
/**< reserved */
uint8_t hash_state_sz;
/**< Number of quad words of inner and outer hash prefix data to process
* Maximum size is 240 */
uint8_t auth_res_sz;
/**< Size in bytes of the authentication result */
} icp_qat_la_auth_req_params_t;
/* Header (LW's 0 - 5) of struct icp_qat_fw_la_bulk_req_s */
typedef struct icp_qat_la_bulk_req_hdr_s {
/**< LWs 0-1 */
icp_qat_fw_comn_req_hdr_t comn_hdr;
/**< Common request header - for Service Command Id,
* use service-specific Crypto Command Id.
* Service Specific Flags - use Symmetric Crypto Command Flags
* (all of cipher, auth, SSL3, TLS and MGF,
* excluding TRNG - field unused) */
/**< LWs 2-5 */
icp_qat_fw_comn_req_hdr_cd_pars_t cd_pars;
/**< Common Request content descriptor field which points either to a
* content descriptor
* parameter block or contains the service-specific data itself. */
} icp_qat_la_bulk_req_hdr_t;
/** Footer (LW's 26 - 31) of struct icp_qat_fw_la_bulk_req_s */
typedef struct icp_qat_la_bulk_req_ftr_s {
/**< LW 0 - equivalent to LW26 of icp_qat_fw_la_bulk_req_t */
icp_qat_la_auth_req_params_t serv_specif_rqpars;
/**< Common request service-specific parameter field */
/**< LW's 1-5, equivalent to LWs 27-31 of icp_qat_fw_la_bulk_req_s */
icp_qat_fw_comn_req_cd_ctrl_t cd_ctrl;
/**< Common request content descriptor control block -
* this field is service-specific */
} icp_qat_la_bulk_req_ftr_t;
/**
***
*******************************************************************************
* @ingroup LacSym
* Compile time check of lac_sym_bulk_cookie_t
*
* @description
* Performs a compile time check of lac_sym_bulk_cookie_t to ensure IA
* assumptions are valid.
*
*****************************************************************************/
void LacSym_CompileTimeAssertions(void);
void LacDp_WriteRingMsgFull(CpaCySymDpOpData *pRequest,
icp_qat_fw_la_bulk_req_t *pCurrentQatMsg);
void LacDp_WriteRingMsgOpt(CpaCySymDpOpData *pRequest,
icp_qat_fw_la_bulk_req_t *pCurrentQatMsg);
#endif /* LAC_SYM_H */

294
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_alg_chain.h Normal file
View File

@ -0,0 +1,294 @@
/***************************************************************************
*
* <COPYRIGHT_TAG>
*
***************************************************************************/
/**
*****************************************************************************
* @file lac_sym_alg_chain.h
*
* @defgroup LacAlgChain Algorithm Chaining
*
* @ingroup LacSym
*
* Interfaces exposed by the Algorithm Chaining Component
*
* @lld_start
*
* @lld_overview
* This is the LAC Algorithm-Chaining feature component. This component
* implements session registration and cleanup functions, and a perform
* function. Statistics are maintained to track requests issued and completed,
* errors incurred, and authentication verification failures. For each
* function the parameters supplied by the client are checked, and then the
* function proceeds if all the parameters are valid. This component also
* incorporates support for Authenticated-Encryption (CCM and GCM) which
* essentially comprises of a cipher operation and a hash operation combined.
*
* This component can combine a cipher operation with a hash operation or just
* simply create a hash only or cipher only operation and is called from the
* LAC Symmetric API component. In turn it calls the LAC Cipher, LAC Hash, and
* LAC Symmetric QAT components. The goal here is to duplicate as little code
* as possible from the Cipher and Hash components.
*
* The cipher and hash operations can be combined in either order, i.e. cipher
* first then hash or hash first then cipher. The client specifies this via
* the algChainOrder field in the session context. This ordering choice is
* stored as part of the session descriptor, so that it is known when a
* perform request is issued. In the case of Authenticated-Encryption, the
* ordering is an implicit part of the CCM or GCM protocol.
*
* When building a content descriptor, as part of session registration, this
* component asks the Cipher and Hash components to build their respective
* parts of the session descriptor. The key aspect here is to provide the
* correct offsets to the Cipher and Hash components for where in the content
* descriptor to write their Config and Hardware Setup blocks. Also the
* Config block in each case must specify the appropriate next slice.
*
* When building request parameters, as part of a perform operation, this
* component asks the Cipher and Hash components to build their respective
* parts of the request parameters block. Again the key aspect here is to
* provide the correct offsets to the Cipher and Hash components for where in
* the request parameters block to write their parameters. Also the request
* parameters block in each case must specify the appropriate next slice.
*
* Parameter checking for session registration and for operation perform is
* mostly delegated to the Cipher and Hash components. There are a few
* extra checks that this component must perform: check the algChainOrder
* parameter, ensure that CCM/GCM are specified for hash/cipher algorithms
* as appropriate, and ensure that requests are for full packets (partial
* packets are not supported for Algorithm-Chaining).
*
* The perform operation allocates a cookie to capture information required
* in the request callback. This cookie is then freed in the callback.
*
* @lld_dependencies
* - \ref LacCipher "Cipher" : For taking care of the cipher aspects of
* session registration and operation perform
* - \ref LacHash "Hash" : For taking care of the hash aspects of session
* registration and operation perform
* - \ref LacSymCommon "Symmetric Common" : statistics.
* - \ref LacSymQat "Symmetric QAT": To build the QAT request message,
* request param structure, and populate the content descriptor. Also
* for registering a callback function to process the QAT response.
* - \ref QatComms "QAT Comms" : For sending messages to the QAT, and for
* setting the response callback
* - \ref LacMem "Mem" : For memory allocation and freeing, virtual/physical
* address translation, and translating between scalar and pointer types
* - OSAL : For atomics and locking
*
* @lld_module_algorithms
* This component builds up a chain of slices at session init time
* and stores it in the session descriptor. This is used for building up the
* content descriptor at session init time and the request parameters structure
* in the perform operation.
*
* The offsets for the first slice are updated so that the second slice adds
* its configuration information after that of the first slice. The first
* slice also configures the next slice appropriately.
*
* This component is very much hard-coded to just support cipher+hash or
* hash+cipher. It should be quite possible to extend this idea to support
* an arbitrary chain of commands, by building up a command chain that can
* be traversed in order to build up the appropriate configuration for the
* QAT. This notion should be looked at in the future if other forms of
* Algorithm-Chaining are desired.
*
* @lld_process_context
*
* @lld_end
*
*****************************************************************************/
/*****************************************************************************/
#ifndef LAC_SYM_ALG_CHAIN_H
#define LAC_SYM_ALG_CHAIN_H
/*
******************************************************************************
* Include public/global header files
******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
#include "lac_session.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
/* Macro for checking if zero length buffer are supported
* only for cipher is AES-GCM and hash are AES-GCM/AES-GMAC */
#define IS_ZERO_LENGTH_BUFFER_SUPPORTED(cipherAlgo, hashAlgo) \
(CPA_CY_SYM_CIPHER_AES_GCM == cipherAlgo && \
(CPA_CY_SYM_HASH_AES_GMAC == hashAlgo || \
CPA_CY_SYM_HASH_AES_GCM == hashAlgo))
/**
*******************************************************************************
* @ingroup LacAlgChain
* This function registers a session for an Algorithm-Chaining operation.
*
* @description
* This function is called from the LAC session register API function for
* Algorithm-Chaining operations. It validates all input parameters. If
* an invalid parameter is passed, an error is returned to the calling
* function. If all parameters are valid an Algorithm-Chaining session is
* registered.
*
* @param[in] instanceHandle Instance Handle
*
* @param[in] pSessionCtx Pointer to session context which contains
* parameters which are static for a given
* cryptographic session such as operation type,
* mechanisms, and keys for cipher and/or digest
* operations.
* @param[out] pSessionDesc Pointer to session descriptor
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_FAIL Function failed.
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter passed in.
* @retval CPA_STATUS_RESOURCE Error related to system resources.
*
* @see cpaCySymInitSession()
*
*****************************************************************************/
CpaStatus LacAlgChain_SessionInit(const CpaInstanceHandle instanceHandle,
const CpaCySymSessionSetupData *pSessionCtx,
lac_session_desc_t *pSessionDesc);
/**
*******************************************************************************
* @ingroup LacAlgChain
* Data path function for the Algorithm-Chaining component
*
* @description
* This function gets called from cpaCySymPerformOp() which is the
* symmetric LAC API function. It is the data path function for the
* Algorithm-Chaining component. It does the parameter checking on the
* client supplied parameters and if the parameters are valid, the
* operation is performed and a request sent to the QAT, otherwise an
* error is returned to the client.
*
* @param[in] instanceHandle Instance Handle
*
* @param[in] pSessionDesc Pointer to session descriptor
* @param[in] pCallbackTag The application's context for this call
* @param[in] pOpData Pointer to a structure containing request
* parameters. The client code allocates the memory for
* this structure. This component takes ownership of
* the memory until it is returned in the callback.
*
* @param[in] pSrcBuffer Source Buffer List
* @param[out] pDstBuffer Destination Buffer List
* @param[out] pVerifyResult Verify Result
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_FAIL Function failed.
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter passed in.
* @retval CPA_STATUS_RESOURCE Error related to system resource.
*
* @see cpaCySymPerformOp()
*
*****************************************************************************/
CpaStatus LacAlgChain_Perform(const CpaInstanceHandle instanceHandle,
lac_session_desc_t *pSessionDesc,
void *pCallbackTag,
const CpaCySymOpData *pOpData,
const CpaBufferList *pSrcBuffer,
CpaBufferList *pDstBuffer,
CpaBoolean *pVerifyResult);
/**
*******************************************************************************
* @ingroup LacAlgChain
* This function is used to update cipher key, as specified in provided
* input.
*
* @description
* This function is called from the LAC session register API function for
* Algorithm-Chaining operations. It validates all input parameters. If
* an invalid parameter is passed, an error is returned to the calling
* function. If all parameters are valid an Algorithm-Chaining session is
* updated.
*
* @threadSafe
* No
*
* @param[in] pSessionDesc Pointer to session descriptor
* @param[in] pCipherKey Pointer to new cipher key.
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_FAIL Function failed.
* @retval CPA_STATUS_RETRY Resubmit the request.
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter passed in.
* @retval CPA_STATUS_UNSUPPORTED Function is not supported.
*
*****************************************************************************/
CpaStatus LacAlgChain_SessionCipherKeyUpdate(lac_session_desc_t *pSessionDesc,
Cpa8U *pCipherKey);
/**
*******************************************************************************
* @ingroup LacAlgChain
* This function is used to update authentication key, as specified in
* provided input.
*
* @description
* This function is called from the LAC session register API function for
* Algorithm-Chaining operations. It validates all input parameters. If
* an invalid parameter is passed, an error is returned to the calling
* function. If all parameters are valid an Algorithm-Chaining session is
* updated.
*
* @threadSafe
* No
*
* @param[in] pSessionDesc Pointer to session descriptor
* @param[in] pCipherKey Pointer to new authentication key.
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_FAIL Function failed.
* @retval CPA_STATUS_RETRY Resubmit the request.
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter passed in.
* @retval CPA_STATUS_UNSUPPORTED Function is not supported.
*
*****************************************************************************/
CpaStatus LacAlgChain_SessionAuthKeyUpdate(lac_session_desc_t *pSessionDesc,
Cpa8U *pAuthKey);
/**
*******************************************************************************
* @ingroup LacAlgChain
* This function is used to update AAD length as specified in provided
* input.
*
* @description
* This function is called from the LAC session register API function for
* Algorithm-Chaining operations. It validates all input parameters. If
* an invalid parameter is passed, an error is returned to the calling
* function. If all parameters are valid an Algorithm-Chaining session is
* updated.
*
* @threadSafe
* No
*
* @param[in] pSessionDesc Pointer to session descriptor
* @param[in] newAADLength New AAD length.
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_FAIL Function failed.
* @retval CPA_STATUS_RETRY Resubmit the request.
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter passed in.
* @retval CPA_STATUS_UNSUPPORTED Function is not supported.
*
*****************************************************************************/
CpaStatus LacAlgChain_SessionAADUpdate(lac_session_desc_t *pSessionDesc,
Cpa32U newAADLength);
#endif /* LAC_SYM_ALG_CHAIN_H */

87
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_auth_enc.h Normal file
View File

@ -0,0 +1,87 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file lac_sym_auth_enc.h
*
* @defgroup LacAuthEnc Authenticated Encryption
*
* @ingroup LacSym
*
* @description
* Authenticated encryption specific functionality.
* For CCM related code NIST SP 800-38C is followed.
* For GCM related code NIST SP 800-38D is followed.
*
***************************************************************************/
#ifndef LAC_SYM_AUTH_ENC_H_
#define LAC_SYM_AUTH_ENC_H_
/* This define for CCM describes constant sum of n and q */
#define LAC_ALG_CHAIN_CCM_NQ_CONST 15
/* These defines for CCM describe maximum and minimum
* length of nonce in bytes*/
#define LAC_ALG_CHAIN_CCM_N_LEN_IN_BYTES_MAX 13
#define LAC_ALG_CHAIN_CCM_N_LEN_IN_BYTES_MIN 7
/**
* @ingroup LacAuthEnc
* This function applies any necessary padding to additional authentication data
* pointed by pAdditionalAuthData field of pOpData as described in
* NIST SP 800-38D
*
* @param[in] pSessionDesc Pointer to the session descriptor
* @param[in,out] pAdditionalAuthData Pointer to AAD
*
* @retval CPA_STATUS_SUCCESS Operation finished successfully
*
* @pre pAdditionalAuthData has been param checked
*
*/
void LacSymAlgChain_PrepareGCMData(lac_session_desc_t *pSessionDesc,
Cpa8U *pAdditionalAuthData);
/**
* @ingroup LacAuthEnc
* This function prepares param checks iv and aad for CCM
*
* @param[in,out] pAdditionalAuthData Pointer to AAD
* @param[in,out] pIv Pointer to IV
* @param[in] messageLenToCipherInBytes Size of the message to cipher
* @param[in] ivLenInBytes Size of the IV
*
* @retval CPA_STATUS_SUCCESS Operation finished successfully
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter passed
*
*/
CpaStatus LacSymAlgChain_CheckCCMData(Cpa8U *pAdditionalAuthData,
Cpa8U *pIv,
Cpa32U messageLenToCipherInBytes,
Cpa32U ivLenInBytes);
/**
* @ingroup LacAuthEnc
* This function prepares Ctr0 and B0-Bn blocks for CCM algorithm as described
* in NIST SP 800-38C. Ctr0 block is placed in pIv field of pOpData and B0-BN
* blocks are placed in pAdditionalAuthData.
*
* @param[in] pSessionDesc Pointer to the session descriptor
* @param[in,out] pAdditionalAuthData Pointer to AAD
* @param[in,out] pIv Pointer to IV
* @param[in] messageLenToCipherInBytes Size of the message to cipher
* @param[in] ivLenInBytes Size of the IV
*
* @retval none
*
* @pre parameters have been checked using LacSymAlgChain_CheckCCMData()
*/
void LacSymAlgChain_PrepareCCMData(lac_session_desc_t *pSessionDesc,
Cpa8U *pAdditionalAuthData,
Cpa8U *pIv,
Cpa32U messageLenToCipherInBytes,
Cpa32U ivLenInBytes);
#endif /* LAC_SYM_AUTH_ENC_H_ */

55
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_cb.h Normal file
View File

@ -0,0 +1,55 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_cb.h
*
* @defgroup LacSymCb Symmetric callback functions
*
* @ingroup LacSym
*
* Functions to assist with callback processing for the symmetric component
***************************************************************************/
#ifndef LAC_SYM_CB_H
#define LAC_SYM_CB_H
/**
*****************************************************************************
* @ingroup LacSym
* Dequeue pending requests
* @description
* This function is called by a callback function of a blocking
* operation (either a partial packet or a hash precompute operaion)
* in softIRQ context. It dequeues requests for the following reasons:
* 1. All pre-computes that happened when initialising a session
* have completed. Dequeue any requests that were queued on the
* session while waiting for the precompute operations to complete.
* 2. A partial packet request has completed. Dequeue any partials
* that were queued for this session while waiting for a previous
* partial to complete.
*
* @param[in] pSessionDesc Pointer to the session descriptor
*
* @return CpaStatus
*
****************************************************************************/
CpaStatus LacSymCb_PendingReqsDequeue(lac_session_desc_t *pSessionDesc);
/**
*****************************************************************************
* @ingroup LacSym
* Register symmetric callback funcion handlers
*
* @description
* This function registers the symmetric callback handler functions with
* the main symmetric callback handler function
*
* @return None
*
****************************************************************************/
void LacSymCb_CallbacksRegister(void);
#endif /* LAC_SYM_CB_H */

312
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_cipher.h Normal file
View File

@ -0,0 +1,312 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file lac_sym_cipher.h
*
* @defgroup LacCipher Cipher
*
* @ingroup LacSym
*
* API functions of the cipher component
*
* @lld_start
* @lld_overview
* There is a single \ref icp_LacSym "Symmetric LAC API" for hash, cipher,
* auth encryption and algorithm chaining. This API is implemented by the
* \ref LacSym "Symmetric" module. It demultiplexes calls to this API into
* their basic operation and does some common parameter checking and deals
* with accesses to the session table.
*
* The cipher component supports data encryption/decryption using the AES, DES,
* and Triple-DES cipher algorithms, in ECB, CBC and CTR modes. The ARC4 stream
* cipher algorithm is also supported. Data may be provided as a full packet,
* or as a sequence of partial packets. The result of the operation can be
* written back to the source buffer (in-place) or to a seperate output buffer
* (out-of-place). Data must be encapsulated in ICP buffers.
*
* The cipher component is responsible for implementing the cipher-specific
* functionality for registering and de-registering a session, for the perform
* operation and for processing the QAT responses to cipher requests. Statistics
* are maintained for cipher in the symmetric \ref CpaCySymStats64 "stats"
* structure. This module has been seperated out into two. The cipher QAT module
* deals entirely with QAT data structures. The cipher module itself has minimal
* exposure to the QAT data structures.
*
* @lld_dependencies
* - \ref LacCommon
* - \ref LacSymQat "Symmetric QAT": Hash uses the lookup table provided by
* this module to validate user input. Hash also uses this module to build
* the hash QAT request message, request param structure, populate the
* content descriptor, allocate and populate the hash state prefix buffer.
* Hash also registers its function to process the QAT response with this
* module.
* - OSAL : For memory functions, atomics and locking
*
* @lld_module_algorithms
* In general, all the cipher algorithms supported by this component are
* implemented entirely by the QAT. However, in the case of the ARC4 algorithm,
* it was deemed more efficient to carry out some processing on IA. During
* session registration, an initial state is derived from the base key provided
* by the user, using a simple ARC4 Key Scheduling Algorithm (KSA). Then the
* base key is discarded, but the state is maintained for the duration of the
* session.
*
* The ARC4 key scheduling algorithm (KSA) is specified as follows
* (taken from http://en.wikipedia.org/wiki/RC4_(cipher)):
* \code
* for i from 0 to 255
* S[i] := i
* endfor
* j := 0
* for i from 0 to 255
* j := (j + S[i] + key[i mod keylength]) mod 256
* swap(S[i],S[j])
* endfor
* \endcode
*
* On registration of a new ARC4 session, the user provides a base key of any
* length from 1 to 256 bytes. This algorithm produces the initial ARC4 state
* (key matrix + i & j index values) from that base key. This ARC4 state is
* used as input for each ARC4 cipher operation in that session, and is updated
* by the QAT after each operation. The ARC4 state is stored in a session
* descriptor, and it's memory is freed when the session is deregistered.
*
* <b>Block Vs. Stream Ciphers</b>\n
* Block ciphers are treated slightly differently than Stream ciphers by this
* cipher component. Supported stream ciphers consist of AES and
* TripleDES algorithms in CTR mode, and ARC4. The 2 primary differences are:
* - Data buffers for block ciphers are required to be a multiple of the
* block size defined for the algorithm (e.g. 8 bytes for DES). For stream
* ciphers, there is no such restriction.
* - For stream ciphers, decryption is performed by setting the QAT hardware
* to encryption mode.
*
* <b>Memory address alignment of data buffers </b>\n
* The QAT requires that most data buffers are aligned on an 8-byte memory
* address boundary (64-byte boundary for optimum performance). For Cipher,
* this applies to the cipher key buffer passed in the Content Descriptor,
* and the IV/State buffer passed in the Request Parameters block in each
* request. Both of these buffers are provided by the user. It does not
* apply to the cipher source/destination data buffers.
* Alignment of the key buffer is ensured because the key is always copied
* from the user provided buffer into a new (aligned) buffer for the QAT
* (the hardware setup block, which configures the QAT slice). This is done
* once only during session registration, and the user's key buffer can be
* effectively discarded after that.
* The IV/State buffer is provided per-request by the user, so it is recommended
* to the user to provide aligned buffers for optimal performance. In the case
* where an unaligned buffer is provided, a new temporary buffer is allocated
* and the user's IV/State data is copied into this buffer. The aligned buffer
* is then passed to the QAT in the request. In the response callback, if the
* IV was updated by the QAT, the contents are copied back to the user's buffer
* and the temporary buffer is freed.
*
* @lld_process_context
*
* Session Register Sequence Diagram: For ARC4 cipher algorithm
* \msc
* APP [label="Application"], SYM [label="Symmetric LAC"],
* Achain [label="Alg chain"], Cipher, SQAT [label="Symmetric QAT"];
*
* APP=>SYM [ label = "cpaCySymInitSession(cbFunc)",
* URL="\ref cpaCySymInitSession()"] ;
* SYM=>SYM [ label = "LacSymSession_ParamCheck()",
* URL="\ref LacSymSession_ParamCheck()"];
* SYM=>Achain [ label = "LacAlgChain_SessionInit()",
* URL="\ref LacAlgChain_SessionInit()"];
* Achain=>Cipher [ label = "LacCipher_SessionSetupDataCheck()",
* URL="\ref LacCipher_SessionSetupDataCheck()"];
* Achain<<Cipher [ label="return"];
* Achain=>SQAT [ label = "LacSymQat_CipherContentDescPopulate()",
* URL="\ref LacSymQat_CipherContentDescPopulate()"];
* Achain<<SQAT [ label="return"];
* Achain=>SQAT [ label = "LacSymQat_CipherArc4StateInit()",
* URL="\ref LacSymQat_CipherArc4StateInit()"];
* Achain<<SQAT [ label="return"];
* SYM<<Achain [ label = "status" ];
* SYM=>SYM [label = "LAC_SYM_STAT_INC", URL="\ref LAC_SYM_STAT_INC"];
* APP<<SYM [label = "status"];
* \endmsc
*
* Perform Sequence Diagram: TripleDES CBC-mode encryption, in-place full
*packet, asynchronous mode \msc APP [label="Application"], SYM
*[label="Symmetric LAC"], SC [label="Symmetric Common"], Achain [label="Alg
*chain"], Cipher, SQAT [label="Symmetric QAT"], BUF [label="LAC Buffer Desc"],
*SYMQ [label="Symmetric Queue"], SYMCB [label="Symmetric Callback"], LMP
*[label="LAC Mem Pool"], QATCOMMS [label="QAT Comms"];
*
* APP=>SYM [ label = "cpaCySymPerformOp()",
* URL="\ref cpaCySymPerformOp()"] ;
* SYM=>SYM [ label = "LacSym_Perform()",
* URL="\ref LacSym_Perform()"];
* SYM=>SYM [ label = "LacSymPerform_BufferParamCheck()",
* URL="\ref LacSymPerform_BufferParamCheck()"];
* SYM<<SYM [ label = "status"];
* SYM=>Achain [ label = "LacAlgChain_Perform()",
* URL="\ref LacCipher()"];
* Achain=>Cipher [ label = "LacCipher_PerformParamCheck()",
* URL="\ref LacCipher_PerformParamCheck()"];
* Achain<<Cipher [ label="status"];
* Achain=>LMP [label="Lac_MemPoolEntryAlloc()",
* URL="\ref Lac_MemPoolEntryAlloc()"];
* Achain<<LMP [label="return"];
* Achain=>Cipher [ label = "LacCipher_PerformIvCheckAndAlign()",
* URL="\ref LacCipher_PerformIvCheckAndAlign()"];
* Achain<<Cipher [ label="status"];
* Achain=>SQAT [ label = "LacSymQat_CipherRequestParamsPopulate()",
* URL="\ref LacSymQat_CipherRequestParamsPopulate()"];
* Achain<<SQAT [ label="return"];
* Achain=>BUF [ label = "LacBuffDesc_BufferListDescWrite()",
* URL = "\ref LacBuffDesc_BufferListDescWrite()"];
* Achain<<BUF [ label="return"];
* Achain=>SQAT [ label = "SalQatMsg_CmnMsgAndReqParamsPopulate()",
* URL="\ref SalQatMsg_CmnMsgAndReqParamsPopulate()"];
* Achain<<SQAT [ label="return"];
* Achain=>SYMQ [ label = "LacSymQueue_RequestSend()",
* URL="\ref LacSymQueue_RequestSend()"];
* SYMQ=>QATCOMMS [ label = "QatComms_MsgSend()",
* URL="\ref QatComms_MsgSend()"];
* SYMQ<<QATCOMMS [ label="status"];
* Achain<<SYMQ [ label="status"];
* SYM<<Achain[ label="status"];
* SYM=>SYM [ label = "LacSym_PartialPacketStateUpdate()",
* URL="\ref LacSym_PartialPacketStateUpdate()"];
* SYM<<SYM [ label = "return"];
* SYM=>SC [label = "LAC_SYM_STAT_INC", URL="\ref LAC_SYM_STAT_INC"];
* SYM<<SC [ label="return"];
* SYM<<SYM [ label = "status"];
* APP<<SYM [label = "status"];
* ... [label = "QAT processing the request and generates response"];
* ...;
* QATCOMMS=>QATCOMMS [label ="QatComms_ResponseMsgHandler()",
* URL="\ref QatComms_ResponseMsgHandler()"];
* QATCOMMS=>SQAT [label ="LacSymQat_SymRespHandler()",
* URL="\ref LacSymQat_SymRespHandler()"];
* SQAT=>SYMCB [label="LacSymCb_ProcessCallback()",
* URL="\ref LacSymCb_ProcessCallback()"];
* SYMCB=>SYMCB [label="LacSymCb_ProcessCallbackInternal()",
* URL="\ref LacSymCb_ProcessCallbackInternal()"];
* SYMCB=>LMP [label="Lac_MemPoolEntryFree()",
* URL="\ref Lac_MemPoolEntryFree()"];
* SYMCB<<LMP [label="return"];
* SYMCB=>SC [label = "LAC_SYM_STAT_INC", URL="\ref LAC_SYM_STAT_INC"];
* SYMCB<<SC [label = "return"];
* SYMCB=>APP [label="cbFunc"];
* SYMCB<<APP [label="return"];
* SQAT<<SYMCB [label="return"];
* QATCOMMS<<SQAT [label="return"];
* \endmsc
*
* #See the sequence diagram for cpaCySymInitSession()
*
* @lld_end
*
*****************************************************************************/
/***************************************************************************/
#ifndef LAC_SYM_CIPHER_H
#define LAC_SYM_CIPHER_H
/*
******************************************************************************
* Include public/global header files
******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "lac_session.h"
#include "lac_sym.h"
/*
* WARNING: There are no checks done on the parameters of the functions in
* this file. The expected values of the parameters are documented and it is
* up to the caller to provide valid values.
*/
/***************************************************************************/
/**
*****************************************************************************
* @ingroup LacCipher
* Cipher session setup data check
*
* @description
* This function will check any algorithm-specific fields
* in the session cipher setup data structure
*
* @param[in] pCipherSetupData Pointer to session cipher context
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_FAIL Function failed.
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter.
*
*****************************************************************************/
CpaStatus LacCipher_SessionSetupDataCheck(
const CpaCySymCipherSetupData *pCipherSetupData);
/**
*******************************************************************************
* @ingroup LacCipher
* Function that checks the perform common parameters for cipher
*
* @description
* This function checks the perform parameters for cipher operations
*
* @param[in] cipherAlgorithm read only pointer to cipher context structure
*
* @param[in] pOpData read only pointer to user-supplied data for this
* cipher operation
* @param[in] packetLen read only length of data in buffer
*
* @retval CPA_STATUS_SUCCESS Success
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter
*
*****************************************************************************/
CpaStatus LacCipher_PerformParamCheck(CpaCySymCipherAlgorithm cipherAlgorithm,
const CpaCySymOpData *pOpData,
const Cpa64U packetLen);
/**
*****************************************************************************
* @ingroup LacCipher
* Cipher perform IV check
*
* @description
* This function will perform algorithm-specific checks on the
* cipher Initialisation Vector data provided by the user.
*
* @param[in] pCbCookie Pointer to struct containing internal cookie
* data for the operation
* @param[in] qatPacketType QAT partial packet type (start/mid/end/none)
* @param[out] ppIvBuffer Returns a pointer to an IV buffer.
*
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_FAIL Function failed.
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter.
*
* @see LacCipher_Perform(), LacCipher_IvBufferRestore()
*
* @note LacCipher_IvBufferRestore() must be called when the request is
* completed to update the users IV buffer, only in the case of partial
* packet requests
*
*****************************************************************************/
CpaStatus LacCipher_PerformIvCheck(sal_service_t *pService,
lac_sym_bulk_cookie_t *pCbCookie,
Cpa32U qatPacketType,
Cpa8U **ppIvBuffer);
#endif /* LAC_SYM_CIPHER_H */

182
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_cipher_defs.h Normal file
View File

@ -0,0 +1,182 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file lac_sym_cipher_defs.h
*
* @ingroup LacCipher
*
* @description
* This file defines constants for the cipher operations.
*
*****************************************************************************/
/***************************************************************************/
#ifndef LAC_SYM_CIPHER_DEFS_H
#define LAC_SYM_CIPHER_DEFS_H
/*
******************************************************************************
* Include public/global header files
******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
/***************************************************************************/
/*
* Constants value for ARC4 algorithm
*/
/* ARC4 algorithm block size */
#define LAC_CIPHER_ARC4_BLOCK_LEN_BYTES 8
/* ARC4 key matrix size (bytes) */
#define LAC_CIPHER_ARC4_KEY_MATRIX_LEN_BYTES 256
/* ARC4 256 bytes for key matrix, 2 for i and j and 6 bytes for padding */
#define LAC_CIPHER_ARC4_STATE_LEN_BYTES 264
#define LAC_SYM_SNOW3G_CIPHER_CONFIG_FOR_HASH_SZ 40
/* Snow3g cipher config required for performing a Snow3g hash operation.
* It contains 8 Bytes of config for hardware, 16 Bytes of Key and requires
* 16 Bytes for the IV.
*/
/* Key Modifier (KM) 4 bytes used in Kasumi algorithm in F8 mode to XOR
* Cipher Key (CK) */
#define LAC_CIPHER_KASUMI_F8_KEY_MODIFIER_4_BYTES 0x55555555
/* The IV length for Kasumi Kgcore is 8 bytes */
#define LAC_CIPHER_KASUMI_F8_IV_LENGTH 8
/* The Counter length for Kasumi Kgcore is 8 bytes */
#define LAC_CIPHER_KASUMI_F8_COUNTER_LENGTH 8
/* The IV length for AES F8 is 16 bytes */
#define LAC_CIPHER_AES_F8_IV_LENGTH 16
/* For Snow3G UEA2, need to make sure last 8 Bytes of IV buffer are
* zero. */
#define LAC_CIPHER_SNOW3G_UEA2_IV_BUFFER_ZERO_LENGTH 8
/* Reserve enough space for max length cipher state
* (can be IV , counter or ARC4 state) */
#define LAC_CIPHER_STATE_SIZE_MAX LAC_CIPHER_ARC4_STATE_LEN_BYTES
/* Reserve enough space for max length cipher IV
* (can be A value for Kasumi(passed in as IV), IV or counter,
* but not ARC4 state) */
#define LAC_CIPHER_IV_SIZE_MAX ICP_QAT_HW_SNOW_3G_UEA2_IV_SZ
/* 96-bit case of IV for GCM algorithm */
#define LAC_CIPHER_IV_SIZE_GCM_12 12
/* 96-bit case of IV for CCP/GCM single pass algorithm */
#define LAC_CIPHER_SPC_IV_SIZE 12
/*
* Constants value for NULL algorithm
*/
/* NULL algorithm block size */
#define LAC_CIPHER_NULL_BLOCK_LEN_BYTES 8
/* Macro to check if the Algorithm is SM4 */
#define LAC_CIPHER_IS_SM4(algo) \
((algo == CPA_CY_SYM_CIPHER_SM4_ECB) || \
(algo == CPA_CY_SYM_CIPHER_SM4_CBC) || \
(algo == CPA_CY_SYM_CIPHER_SM4_CTR))
/* Macro to check if the Algorithm is CHACHA */
#define LAC_CIPHER_IS_CHACHA(algo) (algo == CPA_CY_SYM_CIPHER_CHACHA)
/* Macro to check if the Algorithm is AES */
#define LAC_CIPHER_IS_AES(algo) \
((algo == CPA_CY_SYM_CIPHER_AES_ECB) || \
(algo == CPA_CY_SYM_CIPHER_AES_CBC) || \
(algo == CPA_CY_SYM_CIPHER_AES_CTR) || \
(algo == CPA_CY_SYM_CIPHER_AES_CCM) || \
(algo == CPA_CY_SYM_CIPHER_AES_GCM) || \
(algo == CPA_CY_SYM_CIPHER_AES_XTS))
/* Macro to check if the Algorithm is DES */
#define LAC_CIPHER_IS_DES(algo) \
((algo == CPA_CY_SYM_CIPHER_DES_ECB) || \
(algo == CPA_CY_SYM_CIPHER_DES_CBC))
/* Macro to check if the Algorithm is Triple DES */
#define LAC_CIPHER_IS_TRIPLE_DES(algo) \
((algo == CPA_CY_SYM_CIPHER_3DES_ECB) || \
(algo == CPA_CY_SYM_CIPHER_3DES_CBC) || \
(algo == CPA_CY_SYM_CIPHER_3DES_CTR))
/* Macro to check if the Algorithm is Kasumi */
#define LAC_CIPHER_IS_KASUMI(algo) (algo == CPA_CY_SYM_CIPHER_KASUMI_F8)
/* Macro to check if the Algorithm is Snow3G UEA2 */
#define LAC_CIPHER_IS_SNOW3G_UEA2(algo) (algo == CPA_CY_SYM_CIPHER_SNOW3G_UEA2)
/* Macro to check if the Algorithm is ARC4 */
#define LAC_CIPHER_IS_ARC4(algo) (algo == CPA_CY_SYM_CIPHER_ARC4)
/* Macro to check if the Algorithm is ZUC EEA3 */
#define LAC_CIPHER_IS_ZUC_EEA3(algo) (algo == CPA_CY_SYM_CIPHER_ZUC_EEA3)
/* Macro to check if the Algorithm is NULL */
#define LAC_CIPHER_IS_NULL(algo) (algo == CPA_CY_SYM_CIPHER_NULL)
/* Macro to check if the Mode is CTR */
#define LAC_CIPHER_IS_CTR_MODE(algo) \
((algo == CPA_CY_SYM_CIPHER_AES_CTR) || \
(algo == CPA_CY_SYM_CIPHER_3DES_CTR) || (LAC_CIPHER_IS_CCM(algo)) || \
(LAC_CIPHER_IS_GCM(algo)) || (LAC_CIPHER_IS_CHACHA(algo)) || \
(algo == CPA_CY_SYM_CIPHER_SM4_CTR))
/* Macro to check if the Algorithm is ECB */
#define LAC_CIPHER_IS_ECB_MODE(algo) \
((algo == CPA_CY_SYM_CIPHER_AES_ECB) || \
(algo == CPA_CY_SYM_CIPHER_DES_ECB) || \
(algo == CPA_CY_SYM_CIPHER_3DES_ECB) || \
(algo == CPA_CY_SYM_CIPHER_NULL) || \
(algo == CPA_CY_SYM_CIPHER_SNOW3G_UEA2) || \
(algo == CPA_CY_SYM_CIPHER_SM4_ECB))
/* Macro to check if the Algorithm Mode is F8 */
#define LAC_CIPHER_IS_F8_MODE(algo) \
((algo == CPA_CY_SYM_CIPHER_KASUMI_F8) || \
(algo == CPA_CY_SYM_CIPHER_AES_F8))
/* Macro to check if the Algorithm is CBC */
#define LAC_CIPHER_IS_CBC_MODE(algo) \
((algo == CPA_CY_SYM_CIPHER_AES_CBC) || \
(algo == CPA_CY_SYM_CIPHER_DES_CBC) || \
(algo == CPA_CY_SYM_CIPHER_3DES_CBC) || \
(algo == CPA_CY_SYM_CIPHER_SM4_CBC))
/* Macro to check if the Algorithm is CCM */
#define LAC_CIPHER_IS_CCM(algo) (algo == CPA_CY_SYM_CIPHER_AES_CCM)
/* Macro to check if the Algorithm is GCM */
#define LAC_CIPHER_IS_GCM(algo) (algo == CPA_CY_SYM_CIPHER_AES_GCM)
/* Macro to check if the Algorithm is AES-F8 */
#define LAC_CIPHER_IS_AES_F8(algo) (algo == CPA_CY_SYM_CIPHER_AES_F8)
/* Macro to check if the Algorithm Mode is XTS */
#define LAC_CIPHER_IS_XTS_MODE(algo) (algo == CPA_CY_SYM_CIPHER_AES_XTS)
/* Macro to check if the Algorithm is single pass */
#define LAC_CIPHER_IS_SPC(cipher, hash, mask) \
((LAC_CIPHER_IS_CHACHA(cipher) && (CPA_CY_SYM_HASH_POLY == hash) && \
((mask)&ICP_ACCEL_CAPABILITIES_CHACHA_POLY)) || \
(LAC_CIPHER_IS_GCM(cipher) && ((CPA_CY_SYM_HASH_AES_GCM == hash) || \
(CPA_CY_SYM_HASH_AES_GMAC == hash)) && \
((mask)&ICP_ACCEL_CAPABILITIES_AESGCM_SPC)))
#endif /* LAC_CIPHER_DEFS_H */

559
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_hash.h Normal file
View File

@ -0,0 +1,559 @@
/***************************************************************************
*
* <COPYRIGHT_TAG>
*
***************************************************************************/
/**
*****************************************************************************
* @file lac_sym_hash.h
*
* @defgroup LacHash Hash
*
* @ingroup LacSym
*
* API functions of the Hash component
*
* @lld_start
* @lld_overview
* There is a single \ref cpaCySym "Symmetric LAC API" for hash, cipher,
* auth encryption and algorithm chaining. This API is implemented by the
* \ref LacSym "Symmetric" module. It demultiplexes calls to this API into
* their basic operation and does some common parameter checking and deals
* with accesses to the session table.
*
* The hash component supports hashing in 3 modes. PLAIN, AUTH and NESTED.
* Plain mode is used to provide data integrity while auth mode is used to
* provide integrity as well as its authenticity. Nested mode is inteded
* for use by non standard HMAC like algorithms such as for the SSL master
* key secret. Partial packets is supported for both plain and auth modes.
* In-place and out-of-place processing is supported for all modes. The
* verify operation is supported for PLAIN and AUTH modes only.
*
* The hash component is responsible for implementing the hash specific
* functionality for initialising a session and for a perform operation.
* Statistics are maintained in the symmetric \ref CpaCySymStats64 "stats"
* structure. This module has been seperated out into two. The hash QAT module
* deals entirely with QAT data structures. The hash module itself has minimal
* exposure to the QAT data structures.
*
* @lld_dependencies
* - \ref LacCommon
* - \ref LacSymQat "Symmetric QAT": Hash uses the lookup table provided by
* this module to validate user input. Hash also uses this module to build
* the hash QAT request message, request param structure, populate the
* content descriptor, allocate and populate the hash state prefix buffer.
* Hash also registers its function to process the QAT response with this
* module.
* - OSAL : For memory functions, atomics and locking
*
* @lld_module_algorithms
* <b>a. HMAC Precomputes</b>\n
* HMAC algorithm is specified as follows:
* \f$ HMAC(msg) = hash((key \oplus opad) \parallel
* hash((key \oplus ipad) \parallel msg ))\f$.
* The key is fixed per session, and is padded up to the block size of the
* algorithm if necessary and xored with the ipad/opad. The following portion
* of the operation can be precomputed: \f$ hash(key \oplus ipad) \f$ as the
* output of this intermediate hash will be the same for every perform
* operation. This intermediate state is the intermediate state of a partial
* partial packet. It is used as the initialiser state to \f$ hash(msg) \f$.
* The same applies to \f$ hash(key \oplus ipad) \f$. There is a saving in
* the data path by the length of time it takes to do two hashes on a block
* size of data. Note: a partial packet operation generates an intermediate
* state. The final operation on a partial packet or when a full packet is
* used applies padding and gives the final hash result. Esentially for the
* inner hash, a partial packet final is issued on the data, using the
* precomputed intermediate state and returns the digest.
*
* For the HMAC precomputes, \ref LacSymHash_HmacPreCompute(), there are two
* hash operations done using a internal content descriptor to configure the
* QAT. A first partial packet is specified as the packet type for the
* pre-computes as we need the state that uses the initialiser constants
* specific to the algorithm. The resulting output is copied from the hash
* state prefix buffer into the QAT content descriptor for the session being
* initialised. The state is used each perform operation as the initialiser
* to the algorithm
*
* <b>b. AES XCBC Precomputes</b>\n
* A similar technique to HMAC will be used to generate the precomputes for
* AES XCBC. In this case a cipher operation will be used to generate the
* precomputed result. The Pre-compute operation involves deriving 3 128-bit
* keys (K1, K2 and K3) from the 128-bit secret key K.
*
* - K1 = 0x01010101010101010101010101010101 encrypted with Key K
* - K2 = 0x02020202020202020202020202020202 encrypted with Key K
* - K3 = 0x03030303030303030303030303030303 encrypted with Key K
*
* A content descriptor is created with the cipher algorithm set to AES
* in ECB mode and with the keysize set to 128 bits. The 3 constants, 16 bytes
* each, are copied into the src buffer and an in-place cipher operation is
* performed on the 48 bytes. ECB mode does not maintain the state, therefore
* the 3 keys can be encrypted in one perform. The encrypted result is used by
* the state2 field in the hash setup block of the content descriptor.
*
* The precompute operations use a different lac command ID and thus have a
* different route in the response path to the symmetric code. In this
* precompute callback function the output of the precompute operation is
* copied into the content descriptor for the session being registered.
*
* <b>c. AES CCM Precomputes</b>\n
* The precomputes for AES CCM are trivial, i.e. there is no need to perform
* a cipher or a digest operation. Instead, the key is stored directly in
* the state2 field.
*
* <b>d. AES GCM Precomputes</b>\n
* As with AES XCBC precomputes, a cipher operation will be used to generate
* the precomputed result for AES GCM. In this case the Galois Hash
* Multiplier (H) must be derived and stored in the state2 field. H is
* derived by encrypting a 16-byte block of zeroes with the
* cipher/authentication key, using AES in ECB mode.
*
* <b>Key size for Auth algorithms</b>\n
* <i>Min Size</i>\n
* RFC 2104 states "The key for HMAC can be of any length. However, less than
* L bytes is strongly discouraged as it would decrease the security strength
* of the function."
*
* FIPS 198a states "The size of the key, K, shall be equal to or greater than
* L/2, where L is the size of the hash function output."
*
* RFC 4434 states "If the key has fewer than 128 bits, lengthen it to exactly
* 128 bits by padding it on the right with zero bits.
*
* A key length of 0 upwards is accepted. It is up to the client to pass in a
* key that complies with the standard they wish to support.
*
* <i>Max Size</i>\n
* RFC 2104 section 2 states : "Applications that use keys longer than B bytes
* will first hash the key using H and then use the resultant L byte string
* as the actual key to HMAC
*
* RFC 4434 section 2 states:
* "If the key is 129 bits or longer, shorten it to exactly 128 bits
* by performing the steps in AES-XCBC-PRF-128 (that is, the
* algorithm described in this document). In that re-application of
* this algorithm, the key is 128 zero bits; the message is the
* too-long current key."
*
* We push this up to the client. They need to do the hash operation through
* the LAC API if the key is greater than the block size of the algorithm. This
* will reduce the key size to the digest size of the algorithm.
*
* RFC 3566 section 4 states:
* AES-XCBC-MAC-96 is a secret key algorithm. For use with either ESP or
* AH a fixed key length of 128-bits MUST be supported. Key lengths
* other than 128-bits MUST NOT be supported (i.e., only 128-bit keys are
* to be used by AES-XCBC-MAC-96).
*
* In this case it is up to the client to provide a key that complies with
* the standards. i.e. exactly 128 bits in size.
*
*
* <b>HMAC-MD5-96 and HMAC-SHA1-96</b>\n
* HMAC-MD5-96 and HMAC-SHA1-96 are defined as requirements by Look Aside
* IPsec. The differences between HMAC-SHA1 and HMAC-SHA1-96 are that the
* digest produced is truncated and there are strict requirements on the
* size of the key that is used.
*
* They are supported in LAC by HMAC-MD5 and HMAC-SHA1. The field
* \ref CpaCySymHashSetupData::digestResultLenInBytes in the LAC API in
* bytes needs to be set to 12 bytes. There are also requirements regarding
* the keysize. It is up to the client to ensure the key size meets the
* requirements of the standards they are using.
*
* RFC 2403: HMAC-MD5-96 Key lengths other than 128-bits MUST NOT be supported.
* HMAC-MD5-96 produces a 128-bit authenticator value. For use with either
* ESP or AH, a truncated value using the first 96 bits MUST be supported.
*
* RFC2404: HMAC-SHA1-96 Key lengths other than 160- bits MUST NOT be supported
* HMAC-SHA-1-96 produces a 160-bit authenticator value. For use with either
* ESP or AH, a truncated value using the first 96 bits MUST be supported.
*
* <b>Out of place operations</b>
* When verify is disabled, the digest will be written to the destination
* buffer. When verify is enabled, the digest calculated is compared to the
* digest stored in the source buffer.
*
* <b>Partial Packets</b>
* Partial packets are handled in the \ref LacSym "Symmetric" component for
* the request. The hash callback function handles the update of the state
* in the callback.
*
*
* @lld_process_context
*
* Session Register Sequence Diagram: For hash modes plain and nested.
* \msc
* APP [label="Application"], SYM [label="Symmetric LAC"],
* Achain [label="Alg chain"], Hash, SQAT [label="Symmetric QAT"];
*
* APP=>SYM [ label = "cpaCySymInitSession(cbFunc)",
* URL="\ref cpaCySymInitSession()"] ;
* SYM=>SYM [ label = "LacSymSession_ParamCheck()",
* URL="\ref LacSymSession_ParamCheck()"];
* SYM=>Achain [ label = "LacAlgChain_SessionInit()",
* URL="\ref LacAlgChain_SessionInit()"];
* Achain=>Hash [ label = "LacHash_HashContextCheck()",
* URL="\ref LacHash_HashContextCheck()"];
* Achain<<Hash [ label="return"];
* Achain=>SQAT [ label = "LacSymQat_HashContentDescInit()",
* URL="\ref LacSymQat_HashContentDescInit()"];
* Achain<<SQAT [ label="return"];
* Achain=>Hash [ label = "LacHash_StatePrefixAadBufferInit()",
* URL="\ref LacHash_StatePrefixAadBufferInit()"];
* Hash=>SQAT [ label = "LacSymQat_HashStatePrefixAadBufferSizeGet()",
* URL="\ref LacSymQat_HashStatePrefixAadBufferSizeGet()"];
* Hash<<SQAT [ label="return"];
* Hash=>SQAT [ label = "LacSymQat_HashStatePrefixAadBufferPopulate()",
* URL="\ref LacSymQat_HashStatePrefixAadBufferPopulate()"];
* Hash<<SQAT [ label="return"];
* Achain<<Hash [ label="return"];
* SYM<<Achain [ label = "status" ];
* SYM=>SYM [label = "LAC_SYM_STAT_INC", URL="\ref LAC_SYM_STAT_INC"];
* APP<<SYM [label = "status"];
* \endmsc
*
* Perform Sequence Diagram: For all 3 modes, full packets and in-place.
* \msc
* APP [label="Application"], SYM [label="Symmetric LAC"],
* Achain [label="Alg chain"], Hash, SQAT [label="Symmetric QAT"],
* QATCOMMS [label="QAT Comms"];
*
* APP=>SYM [ label = "cpaCySymPerformOp()",
* URL="\ref cpaCySymPerformOp()"] ;
* SYM=>SYM [ label = "LacSymPerform_BufferParamCheck()",
* URL="\ref LacSymPerform_BufferParamCheck()"];
* SYM=>Achain [ label = "LacAlgChain_Perform()",
* URL="\ref LacAlgChain_Perform()"];
* Achain=>Achain [ label = "Lac_MemPoolEntryAlloc()",
* URL="\ref Lac_MemPoolEntryAlloc()"];
* Achain=>SQAT [ label = "LacSymQat_packetTypeGet()",
* URL="\ref LacSymQat_packetTypeGet()"];
* Achain<<SQAT [ label="return"];
* Achain=>Achain [ label = "LacBuffDesc_BufferListTotalSizeGet()",
* URL="\ref LacBuffDesc_BufferListTotalSizeGet()"];
* Achain=>Hash [ label = "LacHash_PerformParamCheck()",
* URL = "\ref LacHash_PerformParamCheck()"];
* Achain<<Hash [ label="status"];
* Achain=>SQAT [ label = "LacSymQat_HashRequestParamsPopulate()",
* URL="\ref LacSymQat_HashRequestParamsPopulate()"];
* Achain<<SQAT [ label="return"];
* Achain<<SQAT [ label="cmdFlags"];
*
* Achain=>Achain [ label = "LacBuffDesc_BufferListDescWrite()",
* URL="\ref LacBuffDesc_BufferListDescWrite()"];
* Achain=>SQAT [ label = "SalQatMsg_CmnMsgAndReqParamsPopulate()",
* URL="\ref SalQatMsg_CmnMsgAndReqParamsPopulate()"];
* Achain<<SQAT [ label="return"];
* Achain=>SYM [ label = "LacSymQueue_RequestSend()",
* URL="\ref LacSymQueue_RequestSend()"];
* SYM=>QATCOMMS [ label = "QatComms_MsgSend()",
* URL="\ref QatComms_MsgSend()"];
* SYM<<QATCOMMS [ label="status"];
* Achain<<SYM [ label="status"];
* SYM<<Achain [ label="status"];
* SYM=>SYM [label = "LAC_SYM_STAT_INC", URL="\ref LAC_SYM_STAT_INC"];
* APP<<SYM [label = "status"];
* ... [label = "QAT processing the request and generates response.
* Callback in Bottom Half Context"];
* ...;
* QATCOMMS=>QATCOMMS [label ="QatComms_ResponseMsgHandler()",
* URL="\ref QatComms_ResponseMsgHandler()"];
* QATCOMMS=>SQAT [label ="LacSymQat_SymRespHandler()",
* URL="\ref LacSymQat_SymRespHandler()"];
* SQAT=>SYM [label="LacSymCb_ProcessCallback()",
* URL="\ref LacSymCb_ProcessCallback()"];
* SYM=>SYM [label = "LacSymCb_ProcessCallbackInternal()",
* URL="\ref LacSymCb_ProcessCallbackInternal()"];
* SYM=>SYM [label = "Lac_MemPoolEntryFree()",
* URL="\ref Lac_MemPoolEntryFree()"];
* SYM=>SYM [label = "LAC_SYM_STAT_INC", URL="\ref LAC_SYM_STAT_INC"];
* SYM=>APP [label="cbFunc"];
* APP>>SYM [label="return"];
* SYM>>SQAT [label="return"];
* SQAT>>QATCOMMS [label="return"];
* \endmsc
*
* @lld_end
*
*****************************************************************************/
/*****************************************************************************/
#ifndef LAC_SYM_HASH_H
#define LAC_SYM_HASH_H
/*
******************************************************************************
* Include public/global header files
******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "lac_session.h"
#include "lac_buffer_desc.h"
/**
*****************************************************************************
* @ingroup LacHash
* Definition of callback function.
*
* @description
* This is the callback function prototype. The callback function is
* invoked when a hash precompute operation completes.
*
* @param[in] pCallbackTag Opaque value provided by user while making
* individual function call.
*
* @retval
* None
*****************************************************************************/
typedef void (*lac_hash_precompute_done_cb_t)(void *pCallbackTag);
/*
* WARNING: There are no checks done on the parameters of the functions in
* this file. The expected values of the parameters are documented and it is
* up to the caller to provide valid values.
*/
/**
*******************************************************************************
* @ingroup LacHash
* validate the hash context
*
* @description
* The client populates the hash context in the session context structure
* This is passed as parameter to the session register API function and
* needs to be validated.
*
* @param[in] pHashSetupData pointer to hash context structure
*
* @retval CPA_STATUS_SUCCESS Success
* @retval CPA_STATUS_INVALID_PARAM Invalid parameter
*
*****************************************************************************/
CpaStatus LacHash_HashContextCheck(CpaInstanceHandle instanceHandle,
const CpaCySymHashSetupData *pHashSetupData);
/**
******************************************************************************
* @ingroup LacHash
* Populate the hash pre-compute data.
*
* @description
* This function populates the state1 and state2 fields with the hash
* pre-computes. This is only done for authentication. The state1
* and state2 pointers must be set to point to the correct locations
* in the content descriptor where the precompute result(s) will be
* written, before this function is called.
*
* @param[in] instanceHandle Instance Handle
* @param[in] pSessionSetup pointer to session setup data
* @param[in] callbackFn Callback function which is invoked when
* the precompute operation is completed
* @param[in] pCallbackTag Opaque data which is passed back to the user
* as a parameter in the callback function
* @param[out] pWorkingBuffer Pointer to working buffer, sufficient memory
* must be allocated by the caller for this.
* Assumption that this is 8 byte aligned.
* @param[out] pState1 pointer to State 1 in content descriptor
* @param[out] pState2 pointer to State 2 in content descriptor
*
* @retval CPA_STATUS_SUCCESS Success
* @retval CPA_STATUS_RETRY Retry the operation.
* @retval CPA_STATUS_RESOURCE Error Allocating memory
* @retval CPA_STATUS_FAIL Operation Failed
*
*****************************************************************************/
CpaStatus LacHash_PrecomputeDataCreate(const CpaInstanceHandle instanceHandle,
CpaCySymSessionSetupData *pSessionSetup,
lac_hash_precompute_done_cb_t callbackFn,
void *pCallbackTag,
Cpa8U *pWorkingBuffer,
Cpa8U *pState1,
Cpa8U *pState2);
/**
******************************************************************************
* @ingroup LacHash
* populate the hash state prefix aad buffer.
*
* @description
* This function populates the hash state prefix aad buffer. This function
* is not called for CCM/GCM operations as the AAD data varies per request
* and is stored in the cookie as opposed to the session descriptor.
*
* @param[in] pHashSetupData pointer to hash setup structure
* @param[in] pHashControlBlock pointer to hash control block
* @param[in] qatHashMode QAT Mode for hash
* @param[in] pHashStateBuffer pointer to hash state prefix aad buffer
* @param[in] pHashStateBufferInfo Pointer to hash state prefix buffer info
*
* @retval CPA_STATUS_SUCCESS Success
* @retval CPA_STATUS_FAIL Operation Failed
*
*****************************************************************************/
CpaStatus LacHash_StatePrefixAadBufferInit(
sal_service_t *pService,
const CpaCySymHashSetupData *pHashSetupData,
icp_qat_la_bulk_req_ftr_t *pHashControlBlock,
icp_qat_hw_auth_mode_t qatHashMode,
Cpa8U *pHashStateBuffer,
lac_sym_qat_hash_state_buffer_info_t *pHashStateBufferInfo);
/**
*******************************************************************************
* @ingroup LacHash
* Check parameters for a hash perform operation
*
* @description
* This function checks the parameters for a hash perform operation.
*
* @param[in] pSessionDesc Pointer to session descriptor.
* @param[in] pOpData Pointer to request parameters.
* @param[in] srcPktSize Total size of the Buffer List
* @param[in] pVerifyResult Pointer to user flag
*
* @retval CPA_STATUS_SUCCESS Success
* @retval CPA_STATUS_INVALID_PARAM Invalid Parameter
*
*****************************************************************************/
CpaStatus LacHash_PerformParamCheck(CpaInstanceHandle instanceHandle,
lac_session_desc_t *pSessionDesc,
const CpaCySymOpData *pOpData,
Cpa64U srcPktSize,
const CpaBoolean *pVerifyResult);
/**
*******************************************************************************
* @ingroup LacHash
* Perform hash precompute operation for HMAC
*
* @description
* This function sends 2 requests to the CPM for the hmac precompute
* operations. The results of the ipad and opad state calculation
* is copied into pState1 and pState2 (e.g. these may be the state1 and
* state2 buffers in a hash content desciptor) and when
* the final operation has completed the condition passed as a param to
* this function is set to true.
*
* This function performs the XORing of the IPAD and OPAD constants to
* the key (which was padded to the block size of the algorithm)
*
* @param[in] instanceHandle Instance Handle
* @param[in] hashAlgorithm Hash Algorithm
* @param[in] authKeyLenInBytes Length of Auth Key
* @param[in] pAuthKey Pointer to Auth Key
* @param[out] pWorkingMemory Pointer to working memory that is carved
* up and used in the pre-compute operations.
* Assumption that this is 8 byte aligned.
* @param[out] pState1 Pointer to State 1 in content descriptor
* @param[out] pState2 Pointer to State 2 in content descriptor
* @param[in] callbackFn Callback function which is invoked when
* the precompute operation is completed
* @param[in] pCallbackTag Opaque data which is passed back to the user
* as a parameter in the callback function
*
* @retval CPA_STATUS_SUCCESS Success
* @retval CPA_STATUS_RETRY Retry the operation.
* @retval CPA_STATUS_FAIL Operation Failed
*
*****************************************************************************/
CpaStatus LacSymHash_HmacPreComputes(CpaInstanceHandle instanceHandle,
CpaCySymHashAlgorithm hashAlgorithm,
Cpa32U authKeyLenInBytes,
Cpa8U *pAuthKey,
Cpa8U *pWorkingMemory,
Cpa8U *pState1,
Cpa8U *pState2,
lac_hash_precompute_done_cb_t callbackFn,
void *pCallbackTag);
/**
*******************************************************************************
* @ingroup LacHash
* Perform hash precompute operation for XCBC MAC and GCM
*
* @description
* This function sends 1 request to the CPM for the precompute operation
* based on an AES ECB cipher. The results of the calculation is copied
* into pState (this may be a pointer to the State 2 buffer in a Hash
* content descriptor) and when the operation has completed the condition
* passed as a param to this function is set to true.
*
* @param[in] instanceHandle Instance Handle
* @param[in] hashAlgorithm Hash Algorithm
* @param[in] authKeyLenInBytes Length of Auth Key
* @param[in] pAuthKey Auth Key
* @param[out] pWorkingMemory Pointer to working memory that is carved
* up and used in the pre-compute operations.
* Assumption that this is 8 byte aligned.
* @param[out] pState Pointer to output state
* @param[in] callbackFn Callback function which is invoked when
* the precompute operation is completed
* @param[in] pCallbackTag Opaque data which is passed back to the user
* as a parameter in the callback function
*
* @retval CPA_STATUS_SUCCESS Success
* @retval CPA_STATUS_RETRY Retry the operation.
* @retval CPA_STATUS_FAIL Operation Failed
*
*****************************************************************************/
CpaStatus LacSymHash_AesECBPreCompute(CpaInstanceHandle instanceHandle,
CpaCySymHashAlgorithm hashAlgorithm,
Cpa32U authKeyLenInBytes,
Cpa8U *pAuthKey,
Cpa8U *pWorkingMemory,
Cpa8U *pState,
lac_hash_precompute_done_cb_t callbackFn,
void *pCallbackTag);
/**
*******************************************************************************
* @ingroup LacHash
* initialise data structures for the hash precompute operations
*
* @description
* This function registers the precompute callback handler function, which
* is different to the default one used by symmetric. Content desciptors
* are preallocted for the hmac precomputes as they are constant for these
* operations.
*
* @retval CPA_STATUS_SUCCESS Success
* @retval CPA_STATUS_RESOURCE Error allocating memory
*
*****************************************************************************/
CpaStatus LacSymHash_HmacPrecompInit(CpaInstanceHandle instanceHandle);
/**
*******************************************************************************
* @ingroup LacHash
* free resources allocated for the precompute operations
*
* @description
* free up the memory allocated on init time for the content descriptors
* that were allocated for the HMAC precompute operations.
*
* @return none
*
*****************************************************************************/
void LacSymHash_HmacPrecompShutdown(CpaInstanceHandle instanceHandle);
void LacSync_GenBufListVerifyCb(void *pCallbackTag,
CpaStatus status,
CpaCySymOp operationType,
void *pOpData,
CpaBufferList *pDstBuffer,
CpaBoolean opResult);
#endif /* LAC_SYM_HASH_H */

344
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_hash_defs.h Normal file
View File

@ -0,0 +1,344 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_hash_defs.h
*
* @defgroup LacHashDefs Hash Definitions
*
* @ingroup LacHash
*
* Constants for hash algorithms
*
***************************************************************************/
#ifndef LAC_SYM_HASH_DEFS_H
#define LAC_SYM_HASH_DEFS_H
/* Constant for MD5 algorithm */
#define LAC_HASH_MD5_BLOCK_SIZE 64
/**< @ingroup LacHashDefs
* MD5 block size in bytes */
#define LAC_HASH_MD5_DIGEST_SIZE 16
/**< @ingroup LacHashDefs
* MD5 digest length in bytes */
#define LAC_HASH_MD5_STATE_SIZE 16
/**< @ingroup LacHashDefs
* MD5 state size */
/* Constants for SHA1 algorithm */
#define LAC_HASH_SHA1_BLOCK_SIZE 64
/**< @ingroup LacHashDefs
* SHA1 Block size in bytes */
#define LAC_HASH_SHA1_DIGEST_SIZE 20
/**< @ingroup LacHashDefs
* SHA1 digest length in bytes */
#define LAC_HASH_SHA1_STATE_SIZE 20
/**< @ingroup LacHashDefs
* SHA1 state size */
/* Constants for SHA224 algorithm */
#define LAC_HASH_SHA224_BLOCK_SIZE 64
/**< @ingroup LacHashDefs
* SHA224 block size in bytes */
#define LAC_HASH_SHA224_DIGEST_SIZE 28
/**< @ingroup LacHashDefs
* SHA224 digest length in bytes */
#define LAC_HASH_SHA224_STATE_SIZE 32
/**< @ingroup LacHashDefs
* SHA224 state size */
/* Constants for SHA256 algorithm */
#define LAC_HASH_SHA256_BLOCK_SIZE 64
/**< @ingroup LacHashDefs
* SHA256 block size in bytes */
#define LAC_HASH_SHA256_DIGEST_SIZE 32
/**< @ingroup LacHashDefs
* SHA256 digest length */
#define LAC_HASH_SHA256_STATE_SIZE 32
/**< @ingroup LacHashDefs
* SHA256 state size */
/* Constants for SHA384 algorithm */
#define LAC_HASH_SHA384_BLOCK_SIZE 128
/**< @ingroup LacHashDefs
* SHA384 block size in bytes */
#define LAC_HASH_SHA384_DIGEST_SIZE 48
/**< @ingroup LacHashDefs
* SHA384 digest length in bytes */
#define LAC_HASH_SHA384_STATE_SIZE 64
/**< @ingroup LacHashDefs
* SHA384 state size */
/* Constants for SHA512 algorithm */
#define LAC_HASH_SHA512_BLOCK_SIZE 128
/**< @ingroup LacHashDefs
* SHA512 block size in bytes */
#define LAC_HASH_SHA512_DIGEST_SIZE 64
/**< @ingroup LacHashDefs
* SHA512 digest length in bytes */
#define LAC_HASH_SHA512_STATE_SIZE 64
/**< @ingroup LacHashDefs
* SHA512 state size */
/* Constants for SHA3_224 algorithm */
#define LAC_HASH_SHA3_224_BLOCK_SIZE 144
/**< @ingroup LacHashDefs
* SHA3_224 block size in bytes */
#define LAC_HASH_SHA3_224_DIGEST_SIZE 28
/**< @ingroup LacHashDefs
* SHA3_224 digest length in bytes */
#define LAC_HASH_SHA3_224_STATE_SIZE 28
/**< @ingroup LacHashDefs
* SHA3_224 state size */
/* Constants for SHA3_256 algorithm */
#define LAC_HASH_SHA3_256_BLOCK_SIZE 136
/**< @ingroup LacHashDefs
* SHA3_256 block size in bytes */
#define LAC_HASH_SHA3_256_DIGEST_SIZE 32
/**< @ingroup LacHashDefs
* SHA3_256 digest length in bytes */
#define LAC_HASH_SHA3_256_STATE_SIZE 32
/**< @ingroup LacHashDefs
* SHA3_256 state size */
/* Constants for SHA3_384 algorithm */
#define LAC_HASH_SHA3_384_BLOCK_SIZE 104
/**< @ingroup LacHashDefs
* * SHA3_384 block size in bytes */
#define LAC_HASH_SHA3_384_DIGEST_SIZE 48
/**< @ingroup LacHashDefs
* * SHA3_384 digest length in bytes */
#define LAC_HASH_SHA3_384_STATE_SIZE 48
/**< @ingroup LacHashDefs
* * SHA3_384 state size */
/* Constants for SHA3_512 algorithm */
#define LAC_HASH_SHA3_512_BLOCK_SIZE 72
/**< @ingroup LacHashDefs
* * * SHA3_512 block size in bytes */
#define LAC_HASH_SHA3_512_DIGEST_SIZE 64
/**< @ingroup LacHashDefs
* * * SHA3_512 digest length in bytes */
#define LAC_HASH_SHA3_512_STATE_SIZE 64
/**< @ingroup LacHashDefs
* * * SHA3_512 state size */
/* Constants for SHAKE_128 algorithm */
#define LAC_HASH_SHAKE_128_BLOCK_SIZE 168
/**< @ingroup LacHashDefs
* * * SHAKE_128 block size in bytes */
#define LAC_HASH_SHAKE_128_DIGEST_SIZE 0xFFFFFFFF
/**< @ingroup LacHashDefs
* * * SHAKE_128 digest length in bytes ((2^32)-1)*/
/* Constants for SHAKE_256 algorithm */
#define LAC_HASH_SHAKE_256_BLOCK_SIZE 136
/**< @ingroup LacHashDefs
* * * SHAKE_256 block size in bytes */
#define LAC_HASH_SHAKE_256_DIGEST_SIZE 0xFFFFFFFF
/**< @ingroup LacHashDefs
* * * SHAKE_256 digest length in bytes ((2^ 32)-1)*/
/* Constants for POLY algorithm */
#define LAC_HASH_POLY_BLOCK_SIZE 64
/**< @ingroup LacHashDefs
* POLY block size in bytes */
#define LAC_HASH_POLY_DIGEST_SIZE 16
/**< @ingroup LacHashDefs
* POLY digest length */
#define LAC_HASH_POLY_STATE_SIZE 0
/**< @ingroup LacHashDefs
* POLY state size */
/* Constants for SM3 algorithm */
#define LAC_HASH_SM3_BLOCK_SIZE 64
/**< @ingroup LacHashDefs
* SM3 block size in bytes */
#define LAC_HASH_SM3_DIGEST_SIZE 32
/**< @ingroup LacHashDefs
* SM3 digest length */
#define LAC_HASH_SM3_STATE_SIZE 32
/**< @ingroup LacHashDefs
* SM3 state size */
/* Constants for XCBC precompute algorithm */
#define LAC_HASH_XCBC_PRECOMP_KEY_NUM 3
/**< @ingroup LacHashDefs
* The Pre-compute operation involves deriving 3 128-bit
* keys (K1, K2 and K3) */
/* Constants for XCBC MAC algorithm */
#define LAC_HASH_XCBC_MAC_BLOCK_SIZE 16
/**< @ingroup LacHashDefs
* XCBC_MAC block size in bytes */
#define LAC_HASH_XCBC_MAC_128_DIGEST_SIZE 16
/**< @ingroup LacHashDefs
* XCBC_MAC_PRF_128 digest length in bytes */
/* Constants for AES CMAC algorithm */
#define LAC_HASH_CMAC_BLOCK_SIZE 16
/**< @ingroup LacHashDefs
* AES CMAC block size in bytes */
#define LAC_HASH_CMAC_128_DIGEST_SIZE 16
/**< @ingroup LacHashDefs
* AES CMAC digest length in bytes */
/* constants for AES CCM */
#define LAC_HASH_AES_CCM_BLOCK_SIZE 16
/**< @ingroup LacHashDefs
* block size for CBC-MAC part of CCM */
#define LAC_HASH_AES_CCM_DIGEST_SIZE 16
/**< @ingroup LacHashDefs
* untruncated size of authentication field */
/* constants for AES GCM */
#define LAC_HASH_AES_GCM_BLOCK_SIZE 16
/**< @ingroup LacHashDefs
* block size for Galois Hash 128 part of CCM */
#define LAC_HASH_AES_GCM_DIGEST_SIZE 16
/**< @ingroup LacHashDefs
* untruncated size of authentication field */
/* constants for KASUMI F9 */
#define LAC_HASH_KASUMI_F9_BLOCK_SIZE 8
/**< @ingroup LacHashDefs
* KASUMI_F9 block size in bytes */
#define LAC_HASH_KASUMI_F9_DIGEST_SIZE 4
/**< @ingroup LacHashDefs
* KASUMI_F9 digest size in bytes */
/* constants for SNOW3G UIA2 */
#define LAC_HASH_SNOW3G_UIA2_BLOCK_SIZE 8
/**< @ingroup LacHashDefs
* SNOW3G UIA2 block size in bytes */
#define LAC_HASH_SNOW3G_UIA2_DIGEST_SIZE 4
/**< @ingroup LacHashDefs
* SNOW3G UIA2 digest size in bytes */
/* constants for AES CBC MAC */
#define LAC_HASH_AES_CBC_MAC_BLOCK_SIZE 16
/**< @ingroup LacHashDefs
* AES CBC MAC block size in bytes */
#define LAC_HASH_AES_CBC_MAC_DIGEST_SIZE 16
/**< @ingroup LacHashDefs
* AES CBC MAC digest size in bytes */
#define LAC_HASH_ZUC_EIA3_BLOCK_SIZE 4
/**< @ingroup LacHashDefs
* ZUC EIA3 block size in bytes */
#define LAC_HASH_ZUC_EIA3_DIGEST_SIZE 4
/**< @ingroup LacHashDefs
* ZUC EIA3 digest size in bytes */
/* constants for AES GCM ICV allowed sizes */
#define LAC_HASH_AES_GCM_ICV_SIZE_8 8
#define LAC_HASH_AES_GCM_ICV_SIZE_12 12
#define LAC_HASH_AES_GCM_ICV_SIZE_16 16
/* constants for AES CCM ICV allowed sizes */
#define LAC_HASH_AES_CCM_ICV_SIZE_MIN 4
#define LAC_HASH_AES_CCM_ICV_SIZE_MAX 16
/* constants for authentication algorithms */
#define LAC_HASH_IPAD_BYTE 0x36
/**< @ingroup LacHashDefs
* Ipad Byte */
#define LAC_HASH_OPAD_BYTE 0x5c
/**< @ingroup LacHashDefs
* Opad Byte */
#define LAC_HASH_IPAD_4_BYTES 0x36363636
/**< @ingroup LacHashDefs
* Ipad for 4 Bytes */
#define LAC_HASH_OPAD_4_BYTES 0x5c5c5c5c
/**< @ingroup LacHashDefs
* Opad for 4 Bytes */
/* Key Modifier (KM) value used in Kasumi algorithm in F9 mode to XOR
* Integrity Key (IK) */
#define LAC_HASH_KASUMI_F9_KEY_MODIFIER_4_BYTES 0xAAAAAAAA
/**< @ingroup LacHashDefs
* Kasumi F9 Key Modifier for 4 bytes */
#define LAC_SYM_QAT_HASH_IV_REQ_MAX_SIZE_QW 2
/**< @ingroup LacSymQatHash
* Maximum size of IV embedded in the request.
* This is set to 2, namely 4 LONGWORDS. */
#define LAC_SYM_QAT_HASH_STATE1_MAX_SIZE_BYTES LAC_HASH_SHA512_BLOCK_SIZE
/**< @ingroup LacSymQatHash
* Maximum size of state1 in the hash setup block of the content descriptor.
* This is set to the block size of SHA512. */
#define LAC_SYM_QAT_HASH_STATE2_MAX_SIZE_BYTES LAC_HASH_SHA512_BLOCK_SIZE
/**< @ingroup LacSymQatHash
* Maximum size of state2 in the hash setup block of the content descriptor.
* This is set to the block size of SHA512. */
#define LAC_MAX_INNER_OUTER_PREFIX_SIZE_BYTES 255
/**< Maximum size of the inner and outer prefix for nested hashing operations.
* This is got from the maximum size supported by the accelerator which stores
* the size in an 8bit field */
#define LAC_MAX_HASH_STATE_STORAGE_SIZE \
(sizeof(icp_qat_hw_auth_counter_t) + LAC_HASH_SHA512_STATE_SIZE)
/**< Maximum size of the hash state storage section of the hash state prefix
* buffer */
#define LAC_MAX_HASH_STATE_BUFFER_SIZE_BYTES \
LAC_MAX_HASH_STATE_STORAGE_SIZE + \
(LAC_ALIGN_POW2_ROUNDUP(LAC_MAX_INNER_OUTER_PREFIX_SIZE_BYTES, \
LAC_QUAD_WORD_IN_BYTES) * \
2)
/**< Maximum size of the hash state prefix buffer will be for nested hash when
* there is the maximum sized inner prefix and outer prefix */
#define LAC_MAX_AAD_SIZE_BYTES 256
/**< Maximum size of AAD in bytes */
#define IS_HMAC_ALG(algorithm) \
((algorithm == CPA_CY_SYM_HASH_MD5) || \
(algorithm == CPA_CY_SYM_HASH_SHA1) || \
(algorithm == CPA_CY_SYM_HASH_SHA224) || \
(algorithm == CPA_CY_SYM_HASH_SHA256) || \
(algorithm == CPA_CY_SYM_HASH_SHA384) || \
(algorithm == CPA_CY_SYM_HASH_SHA512) || \
(algorithm == CPA_CY_SYM_HASH_SHA3_224) || \
(algorithm == CPA_CY_SYM_HASH_SHA3_256) || \
(algorithm == CPA_CY_SYM_HASH_SHA3_384) || \
(algorithm == CPA_CY_SYM_HASH_SHA3_512) || \
(algorithm == CPA_CY_SYM_HASH_SM3))
/**< @ingroup LacSymQatHash
* Macro to detect if the hash algorithm is a HMAC algorithm */
#define IS_HASH_MODE_1(qatHashMode) (ICP_QAT_HW_AUTH_MODE1 == qatHashMode)
/**< @ingroup LacSymQatHash
* Macro to detect is qat hash mode is set to 1 (precompute mode)
* only used with algorithms in hash mode CPA_CY_SYM_HASH_MODE_AUTH */
#define IS_HASH_MODE_2(qatHashMode) (ICP_QAT_HW_AUTH_MODE2 == qatHashMode)
/**< @ingroup LacSymQatHash
* Macro to detect is qat hash mode is set to 2. This is used for TLS and
* mode 2 HMAC (no preompute mode) */
#define IS_HASH_MODE_2_AUTH(qatHashMode, hashMode) \
((IS_HASH_MODE_2(qatHashMode)) && \
(CPA_CY_SYM_HASH_MODE_AUTH == hashMode))
/**< @ingroup LacSymQatHash
* Macro to check for qat hash mode is set to 2 and the hash mode is
* Auth. This applies to HMAC algorithms (no pre compute). This is used
* to differntiate between TLS and HMAC */
#define IS_HASH_MODE_2_NESTED(qatHashMode, hashMode) \
((IS_HASH_MODE_2(qatHashMode)) && \
(CPA_CY_SYM_HASH_MODE_NESTED == hashMode))
/**< @ingroup LacSymQatHash
* Macro to check for qat hash mode is set to 2 and the LAC hash mode is
* Nested. This applies to TLS. This is used to differentiate between
* TLS and HMAC */
#endif /* LAC_SYM_HASH_DEFS_H */

176
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_hash_precomputes.h Normal file
View File

@ -0,0 +1,176 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_hash_precomputes.h
*
* @defgroup LacHashDefs Hash Definitions
*
* @ingroup LacHash
*
* Constants for hash algorithms
*
***************************************************************************/
#ifndef LAC_SYM_HASH_PRECOMPUTES_H
#define LAC_SYM_HASH_PRECOMPUTES_H
#include "lac_sym_hash.h"
#define LAC_SYM_AES_CMAC_RB_128 0x87 /* constant used for */
/* CMAC calculation */
#define LAC_SYM_HASH_MSBIT_MASK 0x80 /* Mask to check MSB top bit */
/* zero or one */
#define LAC_SINGLE_BUFFER_HW_META_SIZE \
(sizeof(icp_buffer_list_desc_t) + sizeof(icp_flat_buffer_desc_t))
/**< size of memory to allocate for the HW buffer list that is sent to the
* QAT */
#define LAC_SYM_HASH_PRECOMP_MAX_WORKING_BUFFER \
((sizeof(lac_sym_hash_precomp_op_data_t) * 2) + \
sizeof(lac_sym_hash_precomp_op_t))
/**< maximum size of the working data for the HMAC precompute operations
*
* Maximum size of lac_sym_hash_precomp_op_data_t is 264 bytes. For hash
* precomputes there are 2 of these structrues and a further
* lac_sym_hash_precomp_op_t structure required. This comes to a total of 536
* bytes.
* For the asynchronous version of the precomputes, the memory for the hash
* state prefix buffer is used as the working memory. There are 584 bytes
* which are alloacted for the hash state prefix buffer which is enough to
* carve up for the precomputes.
*/
#define LAC_SYM_HASH_PRECOMP_MAX_AES_ECB_DATA \
((ICP_QAT_HW_AES_128_KEY_SZ) * (3))
/**< Maximum size for the data that an AES ECB precompute is generated on */
/**
*****************************************************************************
* @ingroup LacHashDefs
* Precompute type enum
* @description
* Enum used to distinguish between precompute types
*
*****************************************************************************/
typedef enum {
LAC_SYM_HASH_PRECOMP_HMAC = 1,
/**< Hmac precompute operation. Copy state from hash state buffer */
LAC_SYM_HASH_PRECOMP_AES_ECB,
/**< XCBC/CGM precompute, Copy state from data buffer */
} lac_sym_hash_precomp_type_t;
/**
*****************************************************************************
* @ingroup LacHashDefs
* overall precompute management structure
* @description
* structure used to manage the precompute operations for a session
*
*****************************************************************************/
typedef struct lac_sym_hash_precomp_op_s {
lac_hash_precompute_done_cb_t callbackFn;
/**< Callback function to be invoked when the final precompute completes
*/
void *pCallbackTag;
/**< Opaque data to be passed back as a parameter in the callback */
QatUtilsAtomic opsPending;
/**< counter used to determine if the current precompute is the
* final one. */
} lac_sym_hash_precomp_op_t;
/**
*****************************************************************************
* @ingroup LacHashDefs
* hmac precompute structure as used by the QAT
* @description
* data used by the QAT for HMAC precomputes
*
* Must be allocated on an 8-byte aligned memory address.
*
*****************************************************************************/
typedef struct lac_sym_hash_hmac_precomp_qat_s {
Cpa8U data[LAC_HASH_SHA512_BLOCK_SIZE];
/**< data to be hashed - block size of data for the algorithm */
/* NOTE: to save space we could have got the QAT to overwrite
* this with the hash state storage */
icp_qat_fw_la_auth_req_params_t hashReqParams;
/**< Request parameters as read in by the QAT */
Cpa8U bufferDesc[LAC_SINGLE_BUFFER_HW_META_SIZE];
/**< Buffer descriptor structure */
Cpa8U hashStateStorage[LAC_MAX_HASH_STATE_STORAGE_SIZE];
/**< Internal buffer where QAT writes the intermediate partial
* state that is used in the precompute */
} lac_sym_hash_hmac_precomp_qat_t;
/**
*****************************************************************************
* @ingroup LacHashDefs
* AES ECB precompute structure as used by the QAT
* @description
* data used by the QAT for AES ECB precomptes
*
* Must be allocated on an 8-byte aligned memory address.
*
*****************************************************************************/
typedef struct lac_sym_hash_aes_precomp_qat_s {
Cpa8U contentDesc[LAC_SYM_QAT_MAX_CIPHER_SETUP_BLK_SZ];
/**< Content descriptor for a cipher operation */
Cpa8U data[LAC_SYM_HASH_PRECOMP_MAX_AES_ECB_DATA];
/**< The data to be ciphered is conatined here and the result is
* written in place back into this buffer */
icp_qat_fw_la_cipher_req_params_t cipherReqParams;
/**< Request parameters as read in by the QAT */
Cpa8U bufferDesc[LAC_SINGLE_BUFFER_HW_META_SIZE];
/**< Buffer descriptor structure */
} lac_sym_hash_aes_precomp_qat_t;
/**
*****************************************************************************
* @ingroup LacHashDefs
* overall structure for managing a single precompute operation
* @description
* overall structure for managing a single precompute operation
*
* Must be allocated on an 8-byte aligned memory address.
*
*****************************************************************************/
typedef struct lac_sym_hash_precomp_op_data_s {
sal_crypto_service_t *pInstance;
/**< Instance handle for the operation */
Cpa8U reserved[4];
/**< padding to align later structures on minimum 8-Byte address */
lac_sym_hash_precomp_type_t opType;
/**< operation type to determine the precompute type in the callback */
lac_sym_hash_precomp_op_t *pOpStatus;
/**< structure containing the counter and the condition for the overall
* precompute operation. This is a pointer because the memory structure
* may be shared between precomputes when there are more than 1 as in
* the
* case of HMAC */
union {
lac_sym_hash_hmac_precomp_qat_t hmacQatData;
/**< Data sent to the QAT for hmac precomputes */
lac_sym_hash_aes_precomp_qat_t aesQatData;
/**< Data sent to the QAT for AES ECB precomputes */
} u;
/**< ASSUMPTION: The above structures are 8 byte aligned if the overall
* struct is 8 byte aligned, as there are two 4 byte fields before this
* union */
Cpa32U stateSize;
/**< Size of the state to be copied into the state pointer in the
* content
* descriptor */
Cpa8U *pState;
/**< pointer to the state in the content descriptor where the result of
* the precompute should be copied to */
} lac_sym_hash_precomp_op_data_t;
#endif /* LAC_SYM_HASH_PRECOMPUTES_H */

184
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_key.h Normal file
View File

@ -0,0 +1,184 @@
/***************************************************************************
*
* <COPYRIGHT_TAG>
*
***************************************************************************/
/**
*****************************************************************************
* @file lac_sym_key.h
*
* @defgroup LacSymKey Key Generation
*
* @ingroup LacSym
*
* @lld_start
*
* @lld_overview
*
* Key generation component is reponsible for SSL, TLS & MGF operations. All
* memory required for the keygen operations is got from the keygen cookie
* structure which is carved up as required.
*
* For SSL the QAT accelerates the nested hash function with MD5 as the
* outer hash and SHA1 as the inner hash.
*
* Refer to sections in draft-freier-ssl-version3-02.txt:
* 6.1 Asymmetric cryptographic computations - This refers to coverting
* the pre master secret to the master secret.
* 6.2.2 Converting the master secret into keys and MAC secrets - Using
* the master secret to generate the key material.
*
* For TLS the QAT accelerates the PRF function as described in
* rfc4346 - TLS version 1.1 (this obsoletes rfc2246 - TLS version 1.0)
* 5. HMAC and the pseudorandom function - For the TLS PRF and getting
* S1 and S2 from the secret.
* 6.3. Key calculation - For how the key material is generated
* 7.4.9. Finished - How the finished message uses the TLS PRF
* 8.1. Computing the master secret
*
*
* @lld_dependencies
* \ref LacSymQatHash: for building up hash content descriptor
* \ref LacMem: for virt to phys coversions
*
* @lld_initialisation
* The reponse handler is registered with Symmetric. The Maximum SSL is
* allocated. A structure is allocated containing all the TLS lables that
* are supported. On shutdown the memory for these structures are freed.
*
* @lld_module_algorithms
* @lld_process_context
*
* @lld_end
*
*
*****************************************************************************/
#ifndef LAC_SYM_KEY_H_
#define LAC_SYM_KEY_H_
#include "icp_qat_fw_la.h"
#include "cpa_cy_key.h"
/**< @ingroup LacSymKey
* Label for SSL. Size is 136 bytes for 16 iterations, which can theroretically
* generate up to 256 bytes of output data. QAT will generate a maximum of
* 255 bytes */
#define LAC_SYM_KEY_TLS_MASTER_SECRET_LABEL ("master secret")
/**< @ingroup LacSymKey
* Label for TLS Master Secret Key Derivation, as defined in RFC4346 */
#define LAC_SYM_KEY_TLS_KEY_MATERIAL_LABEL ("key expansion")
/**< @ingroup LacSymKey
* Label for TLS Key Material Generation, as defined in RFC4346. */
#define LAC_SYM_KEY_TLS_CLIENT_FIN_LABEL ("client finished")
/**< @ingroup LacSymKey
* Label for TLS Client finished Message, as defined in RFC4346. */
#define LAC_SYM_KEY_TLS_SERVER_FIN_LABEL ("server finished")
/**< @ingroup LacSymKey
* Label for TLS Server finished Message, as defined in RFC4346. */
/*
*******************************************************************************
* Define Constants and Macros for SSL, TLS and MGF
*******************************************************************************
*/
#define LAC_SYM_KEY_NO_HASH_BLK_OFFSET_QW 0
/**< Used to indicate there is no hash block offset in the content descriptor
*/
/*
*******************************************************************************
* Define Constant lengths for HKDF TLS v1.3 sublabels.
*******************************************************************************
*/
#define HKDF_SUB_LABEL_KEY_LENGTH ((Cpa8U)13)
#define HKDF_SUB_LABEL_IV_LENGTH ((Cpa8U)12)
#define HKDF_SUB_LABEL_RESUMPTION_LENGTH ((Cpa8U)20)
#define HKDF_SUB_LABEL_FINISHED_LENGTH ((Cpa8U)18)
#define HKDF_SUB_LABELS_ALL \
(CPA_CY_HKDF_SUBLABEL_KEY | CPA_CY_HKDF_SUBLABEL_IV | \
CPA_CY_HKDF_SUBLABEL_RESUMPTION | CPA_CY_HKDF_SUBLABEL_FINISHED)
#define LAC_KEY_HKDF_SUBLABELS_NUM 4
#define LAC_KEY_HKDF_DIGESTS 0
#define LAC_KEY_HKDF_CIPHERS_MAX (CPA_CY_HKDF_TLS_AES_128_CCM_8_SHA256 + 1)
#define LAC_KEY_HKDF_SUBLABELS_MAX (LAC_KEY_HKDF_SUBLABELS_NUM + 1)
/**
******************************************************************************
* @ingroup LacSymKey
* TLS label struct
*
* @description
* This structure is used to hold the various TLS labels. Each field is
* on an 8 byte boundary provided the structure itslef is 8 bytes aligned.
*****************************************************************************/
typedef struct lac_sym_key_tls_labels_s {
Cpa8U masterSecret[ICP_QAT_FW_LA_TLS_LABEL_LEN_MAX];
/**< Master secret label */
Cpa8U keyMaterial[ICP_QAT_FW_LA_TLS_LABEL_LEN_MAX];
/**< Key material label */
Cpa8U clientFinished[ICP_QAT_FW_LA_TLS_LABEL_LEN_MAX];
/**< client finished label */
Cpa8U serverFinished[ICP_QAT_FW_LA_TLS_LABEL_LEN_MAX];
/**< server finished label */
} lac_sym_key_tls_labels_t;
/**
******************************************************************************
* @ingroup LacSymKey
* TLS HKDF sub label struct
*
* @description
* This structure is used to hold the various TLS HKDF sub labels.
* Each field is on an 8 byte boundary.
*****************************************************************************/
typedef struct lac_sym_key_tls_hkdf_sub_labels_s {
CpaCyKeyGenHKDFExpandLabel keySublabel256;
/**< CPA_CY_HKDF_SUBLABEL_KEY */
CpaCyKeyGenHKDFExpandLabel ivSublabel256;
/**< CPA_CY_HKDF_SUBLABEL_IV */
CpaCyKeyGenHKDFExpandLabel resumptionSublabel256;
/**< CPA_CY_HKDF_SUBLABEL_RESUMPTION */
CpaCyKeyGenHKDFExpandLabel finishedSublabel256;
/**< CPA_CY_HKDF_SUBLABEL_FINISHED */
CpaCyKeyGenHKDFExpandLabel keySublabel384;
/**< CPA_CY_HKDF_SUBLABEL_KEY */
CpaCyKeyGenHKDFExpandLabel ivSublabel384;
/**< CPA_CY_HKDF_SUBLABEL_IV */
CpaCyKeyGenHKDFExpandLabel resumptionSublabel384;
/**< CPA_CY_HKDF_SUBLABEL_RESUMPTION */
CpaCyKeyGenHKDFExpandLabel finishedSublabel384;
/**< CPA_CY_HKDF_SUBLABEL_FINISHED */
CpaCyKeyGenHKDFExpandLabel keySublabelChaChaPoly;
/**< CPA_CY_HKDF_SUBLABEL_KEY */
CpaCyKeyGenHKDFExpandLabel ivSublabelChaChaPoly;
/**< CPA_CY_HKDF_SUBLABEL_IV */
CpaCyKeyGenHKDFExpandLabel resumptionSublabelChaChaPoly;
/**< CPA_CY_HKDF_SUBLABEL_RESUMPTION */
CpaCyKeyGenHKDFExpandLabel finishedSublabelChaChaPoly;
/**< CPA_CY_HKDF_SUBLABEL_FINISHED */
Cpa64U sublabelPhysAddr256;
/**< Physical address of the SHA-256 subLabels */
Cpa64U sublabelPhysAddr384;
/**< Physical address of the SHA-384 subLabels */
Cpa64U sublabelPhysAddrChaChaPoly;
/**< Physical address of the ChaChaPoly subLabels */
} lac_sym_key_tls_hkdf_sub_labels_t;
/**
******************************************************************************
* @ingroup LacSymKey
* This function prints the stats to standard out.
*
* @retval CPA_STATUS_SUCCESS Status Success
* @retval CPA_STATUS_FAIL General failure
*
*****************************************************************************/
void LacKeygen_StatsShow(CpaInstanceHandle instanceHandle);
#endif

121
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_partial.h Normal file
View File

@ -0,0 +1,121 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_partial.h
*
* @defgroup LacSymPartial Partial Packets
*
* @ingroup LacSymCommon
*
* Partial packet handling code
*
* @lld_start
*
* <b>Partials In Flight</b>\n
* The API states that for partial packets the client should not submit
* the next partial request until the callback for the current partial has
* been called. We have chosen to enforce this rather than letting the user
* proceed where they would get an incorrect digest, cipher result.
*
* Maintain a SpinLock for partials in flight per session. Try and acquire this
* SpinLock. If it cant be acquired return an error straight away to the client
* as there is already a partial in flight. There is no blocking in the data
* path for this.
*
* By preventing any other partials from coming in while a partial is in flight
* we can check and change the state of the session without having to lock
* round it (dont want to have to lock and block in the data path). The state
* of the session indicates the previous packet type that a request was
* successfully completed for. The last packet type is only updated for partial
* packets. This state determines the packet types that can be accepted.
* e.g a last partial will not be accepted unless the previous packet was a
* partial. By only allowing one partial packet to be in flight, there is no
* need to lock around the update of the previous packet type for the session.
*
* The ECB Cipher mode, ciphers each block separately. No state is maintained
* between blocks. There is no need to wait for the callback for previous
* partial in ECB mode as the result of the previous partial has no impact on
* it. The API and our implementation only allows 1 partial packet to be in
* flight per session, therefore a partial packet request for ECB mode must
* be fully completed (ie. callback called) before the next partial request
* can be issued.
*
* <b>Partial Ordering</b>\n
* The ordering that the user submits partial packets will be checked.
* (we could have let the user proceed where they will get an incorrect
* digest/cipher result but chose against this).
*
* -# Maintain the last packet type of a partial operation for the session. If
* there have been no previous partials, we will accept only first partials
* -# The state must be set to partial before we will accept a final partial.
* i.e. a partial request must have already completed.
*
* The last packet type is updated in the callback for partial packets as this
* is the only place we can guarantee that a partial packet operation has been
* completed. When a partial completes the state can be updated from FULL to
* PARTIAL. The SpinLock for partial packets in flight for the session can be
* unlocked at this point. On a final Partial request the last packet type is
* reset back to FULL. NOTE: This is not done at the same time as the check in
* the perform as if an error occurs we would have to roll back the state
*
* For Hash mode it is possible to interleave full and a single partial
* packet stream in a session as the hash state buffer is updated for partial
* packets. It is not touched by full packets. For cipher mode, as the client
* manages the state, they can interleave full and a single partial packets.
* For ARC4, the state is managed internally and the packet type will always
* be set to partial internally.
*
* @lld_end
*
***************************************************************************/
/***************************************************************************/
#ifndef LAC_SYM_PARTIAL_H
#define LAC_SYM_PARTIAL_H
#include "cpa.h"
#include "cpa_cy_sym.h"
/***************************************************************************/
/**
*******************************************************************************
* @ingroup LacSymPartial
* check if partial packet request is valid for a session
*
* @description
* This function checks to see if there is a partial packet request in
* flight and then if the partial state is correct
*
* @param[in] packetType Partial packet request
* @param[in] partialState Partial state of session
*
* @retval CPA_STATUS_SUCCESS Normal Operation
* @retval CPA_STATUS_INVALID_PARAM Invalid Parameter
*
*****************************************************************************/
CpaStatus LacSym_PartialPacketStateCheck(CpaCySymPacketType packetType,
CpaCySymPacketType partialState);
/**
*******************************************************************************
* @ingroup LacSymPartial
* update the state of the partial packet in a session
*
* @description
* This function is called in callback operation. It updates the state
* of a partial packet in a session and indicates that there is no
* longer a partial packet in flight for the session
*
* @param[in] packetType Partial packet request
* @param[out] pPartialState Pointer to partial state of session
*
*****************************************************************************/
void LacSym_PartialPacketStateUpdate(CpaCySymPacketType packetType,
CpaCySymPacketType *pPartialState);
#endif /* LAC_SYM_PARTIAL_H */

209
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_qat.h Normal file
View File

@ -0,0 +1,209 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file lac_sym_qat.h
*
* @defgroup LacSymQat Symmetric QAT
*
* @ingroup LacSym
*
* Interfaces for populating the qat structures for a symmetric operation
*
* @lld_start
*
* @lld_overview
* This file documents the interfaces for populating the qat structures
* that are common for all symmetric operations.
*
* @lld_dependencies
* - \ref LacSymQatHash "Hash QAT Comms" Sym Qat commons for Hash
* - \ref LacSymQat_Cipher "Cipher QAT Comms" Sym Qat commons for Cipher
* - OSAL: logging
* - \ref LacMem "Memory" - Inline memory functions
*
* @lld_initialisation
* This component is initialied during the LAC initialisation sequence. It
* is called by the Symmetric Initialisation function.
*
* @lld_module_algorithms
*
* @lld_process_context
* Refer to \ref LacHash "Hash" and \ref LacCipher "Cipher" for sequence
* diagrams to see their interactions with this code.
*
*
* @lld_end
*
*****************************************************************************/
/*****************************************************************************/
#ifndef LAC_SYM_QAT_H
#define LAC_SYM_QAT_H
/*
******************************************************************************
* Include public/global header files
******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
#include "icp_accel_devices.h"
#include "icp_qat_fw_la.h"
#include "icp_qat_hw.h"
#include "lac_session.h"
#include "sal_qat_cmn_msg.h"
#include "lac_common.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#define LAC_SYM_DEFAULT_QAT_PTR_TYPE QAT_COMN_PTR_TYPE_SGL
#define LAC_SYM_DP_QAT_PTR_TYPE QAT_COMN_PTR_TYPE_FLAT
#define LAC_SYM_KEY_QAT_PTR_TYPE QAT_COMN_PTR_TYPE_FLAT
/**< @ingroup LacSymQat
* LAC SYM Source & Destination buffer type (FLAT/SGL) */
#define LAC_QAT_SYM_REQ_SZ_LW 32
#define SYM_TX_MSG_SIZE (LAC_QAT_SYM_REQ_SZ_LW * LAC_LONG_WORD_IN_BYTES)
#define NRBG_TX_MSG_SIZE (LAC_QAT_SYM_REQ_SZ_LW * LAC_LONG_WORD_IN_BYTES)
#define LAC_QAT_SYM_RESP_SZ_LW 8
#define SYM_RX_MSG_SIZE (LAC_QAT_SYM_RESP_SZ_LW * LAC_LONG_WORD_IN_BYTES)
#define NRBG_RX_MSG_SIZE (LAC_QAT_SYM_RESP_SZ_LW * LAC_LONG_WORD_IN_BYTES)
/**
*******************************************************************************
* @ingroup LacSymQat
* Symmetric crypto response handler
*
* @description
* This function handles the symmetric crypto response
*
* @param[in] trans_handle transport handle (if ICP_QAT_DBG set)
* @param[in] instanceHandle void* pRespMsg
*
*
*****************************************************************************/
void LacSymQat_SymRespHandler(void *pRespMsg);
/**
*******************************************************************************
* @ingroup LacSymQat
* Initialise the Symmetric QAT code
*
* @description
* This function initialises the symmetric QAT code
*
* @param[in] device Pointer to the acceleration device
* structure
* @param[in] instanceHandle Instance handle
* @param[in] numSymRequests Number of concurrent requests a pair
* (tx and rx) need to support
*
* @return CPA_STATUS_SUCCESS Operation successful
* @return CPA_STATUS_FAIL Initialisation Failed
*
*****************************************************************************/
CpaStatus LacSymQat_Init(CpaInstanceHandle instanceHandle);
/**
*******************************************************************************
* @ingroup LacSymQat
* Register a response handler function for a symmetric command ID
*
* @description
* This function registers a response handler function for a symmetric
* operation.
*
* Note: This operation should only be performed once by the init function
* of a component. There is no corresponding deregister function, but
* registering a NULL function pointer will have the same effect. There
* MUST not be any requests in flight when calling this function.
*
* @param[in] lacCmdId Command Id of operation
* @param[in] pCbHandler callback handler function
*
* @return None
*
*****************************************************************************/
void LacSymQat_RespHandlerRegister(icp_qat_fw_la_cmd_id_t lacCmdId,
sal_qat_resp_handler_func_t pCbHandler);
/**
******************************************************************************
* @ingroup LacSymQat
* get the QAT packet type
*
* @description
* This function returns the QAT packet type for a LAC packet type. The
* LAC packet type does not indicate a first partial. therefore for a
* partial request, the previous packet type needs to be looked at to
* figure out if the current partial request is a first partial.
*
*
* @param[in] packetType LAC Packet type
* @param[in] packetState LAC Previous Packet state
* @param[out] pQatPacketType Packet type using the QAT macros
*
* @return none
*
*****************************************************************************/
void LacSymQat_packetTypeGet(CpaCySymPacketType packetType,
CpaCySymPacketType packetState,
Cpa32U *pQatPacketType);
/**
******************************************************************************
* @ingroup LacSymQat
* Populate the command flags based on the packet type
*
* @description
* This function populates the following flags in the Symmetric Crypto
* service_specif_flags field of the common header of the request:
* - LA_PARTIAL
* - UPDATE_STATE
* - RET_AUTH_RES
* - CMP_AUTH_RES
* based on looking at the input params listed below.
*
* @param[in] qatPacketType Packet type
* @param[in] cmdId Command Id
* @param[in] cipherAlgorithm Cipher Algorithm
* @param[out] pLaCommandFlags Command Flags
*
* @return none
*
*****************************************************************************/
void LacSymQat_LaPacketCommandFlagSet(Cpa32U qatPacketType,
icp_qat_fw_la_cmd_id_t laCmdId,
CpaCySymCipherAlgorithm cipherAlgorithm,
Cpa16U *pLaCommandFlags,
Cpa32U ivLenInBytes);
/**
******************************************************************************
* @ingroup LacSymQat
*
*
* @description
* defaults the common request service specific flags
*
* @param[in] laCmdFlags Common request service specific flags
* @param[in] symOp Type of operation performed e.g hash or cipher
*
* @return none
*
*****************************************************************************/
void LacSymQat_LaSetDefaultFlags(icp_qat_fw_serv_specif_flags *laCmdFlags,
CpaCySymOp symOp);
#endif /* LAC_SYM_QAT_H */

291
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_qat_cipher.h Normal file
View File

@ -0,0 +1,291 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file lac_sym_qat_cipher.h
*
* @defgroup LacSymQat_Cipher Cipher QAT
*
* @ingroup LacSymQat
*
* external interfaces for populating QAT structures for cipher operations.
*
*****************************************************************************/
/*****************************************************************************/
#ifndef LAC_SYM_QAT_CIPHER_H
#define LAC_SYM_QAT_CIPHER_H
/*
******************************************************************************
* Include public/global header files
******************************************************************************
*/
#include "cpa_cy_sym.h"
#include "icp_qat_fw_la.h"
#include "lac_session.h"
#include "lac_sal_types_crypto.h"
/*
**************************************************************************
* @ingroup LacSymQat_Cipher
*
* @description
* Defines for building the cipher request params cache
*
************************************************************************** */
#define LAC_SYM_QAT_CIPHER_NEXT_ID_BIT_OFFSET 24
#define LAC_SYM_QAT_CIPHER_CURR_ID_BIT_OFFSET 16
#define LAC_SYM_QAT_CIPHER_STATE_SIZE_BIT_OFFSET 8
#define LAC_SYM_QAT_CIPHER_OFFSET_IN_DRAM_GCM_SPC 9
#define LAC_SYM_QAT_CIPHER_OFFSET_IN_DRAM_CHACHA_SPC 2
#define LAC_SYM_QAT_CIPHER_STATE_SIZE_SPC 48
/**
******************************************************************************
* @ingroup LacSymQat_Cipher
* Retrieve the cipher block size in bytes for a given algorithm
*
* @description
* This function returns a hard-coded block size for the specific cipher
* algorithm
*
* @param[in] cipherAlgorithm Cipher algorithm for the current session
*
* @retval The block size, in bytes, for the given cipher algorithm
*
*****************************************************************************/
Cpa8U
LacSymQat_CipherBlockSizeBytesGet(CpaCySymCipherAlgorithm cipherAlgorithm);
/**
******************************************************************************
* @ingroup LacSymQat_Cipher
* Retrieve the cipher IV/state size in bytes for a given algorithm
*
* @description
* This function returns a hard-coded IV/state size for the specific cipher
* algorithm
*
* @param[in] cipherAlgorithm Cipher algorithm for the current session
*
* @retval The IV/state size, in bytes, for the given cipher algorithm
*
*****************************************************************************/
Cpa32U LacSymQat_CipherIvSizeBytesGet(CpaCySymCipherAlgorithm cipherAlgorithm);
/**
******************************************************************************
* @ingroup LacSymQat_Cipher
* Populate the cipher request params structure
*
* @description
* This function is passed a pointer to the 128B request block.
* (This memory must be allocated prior to calling this function). It
* populates:
* - the cipher fields of the req_params block in the request. No
* need to zero this first, all fields will be populated.
* - the corresponding CIPH_IV_FLD flag in the serv_specif_flags field
* of the common header.
* To do this it uses the parameters described below and the following
*fields from the request block which must be populated prior to calling this
*function:
* - cd_ctrl.cipher_state_sz
* - UPDATE_STATE flag in comn_hdr.serv_specif_flags
*
*
* @param[in] pReq Pointer to request block.
* *
* @param[in] cipherOffsetInBytes Offset to cipher data in user data buffer
*
* @param[in] cipherLenInBytes Length of cipher data in buffer
*
* @param[in] ivBufferPhysAddr Physical address of aligned IV/state
* buffer
* @param[in] pIvBufferVirt Virtual address of aligned IV/state
* buffer
* @retval void
*
*****************************************************************************/
CpaStatus LacSymQat_CipherRequestParamsPopulate(icp_qat_fw_la_bulk_req_t *pReq,
Cpa32U cipherOffsetInBytes,
Cpa32U cipherLenInBytes,
Cpa64U ivBufferPhysAddr,
Cpa8U *pIvBufferVirt);
/**
******************************************************************************
* @ingroup LacSymQat_Cipher
* Derive initial ARC4 cipher state from a base key
*
* @description
* An initial state for an ARC4 cipher session is derived from the base
* key provided by the user, using the ARC4 Key Scheduling Algorithm (KSA)
*
* @param[in] pKey The base key provided by the user
*
* @param[in] keyLenInBytes The length of the base key provided.
* The range of valid values is 1-256 bytes
*
* @param[out] pArc4CipherState The initial state is written to this buffer,
* including i and j values, and 6 bytes of padding
* so 264 bytes must be allocated for this buffer
* by the caller
*
* @retval void
*
*****************************************************************************/
void LacSymQat_CipherArc4StateInit(const Cpa8U *pKey,
Cpa32U keyLenInBytes,
Cpa8U *pArc4CipherState);
/**
******************************************************************************
* @ingroup LacSymQat_CipherXTSModeUpdateKeyLen
* Update the initial XTS key after the first partial has been received.
*
* @description
* For XTS mode using partial packets, after the first partial response
* has been received, the the key length needs to be halved for subsequent
* partials.
*
* @param[in] pSessionDesc The session descriptor.
*
* @param[in] newKeySizeInBytes The new key size..
*
* @retval void
*
*****************************************************************************/
void LacSymQat_CipherXTSModeUpdateKeyLen(lac_session_desc_t *pSessionDesc,
Cpa32U newKeySizeInBytes);
/**
******************************************************************************
* @ingroup LacSymQat_Cipher
* LacSymQat_CipherCtrlBlockInitialize()
*
* @description
* intialize the cipher control block with all zeros
*
* @param[in] pMsg Pointer to the common request message
*
* @retval void
*
*****************************************************************************/
void LacSymQat_CipherCtrlBlockInitialize(icp_qat_fw_la_bulk_req_t *pMsg);
/**
******************************************************************************
* @ingroup LacSymQat_Cipher
* LacSymQat_CipherCtrlBlockWrite()
*
* @description
* This function populates the cipher control block of the common request
* message
*
* @param[in] pMsg Pointer to the common request message
*
* @param[in] cipherAlgorithm Cipher Algorithm to be used
*
* @param[in] targetKeyLenInBytes cipher key length in bytes of selected
* algorithm
*
* @param[out] nextSlice SliceID for next control block
* entry. This value is known only by
* the calling component
*
* @param[out] cipherCfgOffsetInQuadWord Offset into the config table in QW
*
* @retval void
*
*****************************************************************************/
void LacSymQat_CipherCtrlBlockWrite(icp_qat_la_bulk_req_ftr_t *pMsg,
Cpa32U cipherAlgorithm,
Cpa32U targetKeyLenInBytes,
icp_qat_fw_slice_t nextSlice,
Cpa8U cipherCfgOffsetInQuadWord);
/**
******************************************************************************
* @ingroup LacSymQat_Cipher
* LacSymQat_CipherHwBlockPopulateCfgData()
*
* @description
* Populate the physical HW block with config data
*
* @param[in] pSession Pointer to the session data
*
* @param[in] pCipherHwBlock pointer to the hardware control block
* in the common message
*
* @param[in] pSizeInBytes
*
* @retval void
*
*****************************************************************************/
void LacSymQat_CipherHwBlockPopulateCfgData(lac_session_desc_t *pSession,
const void *pCipherHwBlock,
Cpa32U *pSizeInBytes);
/**
******************************************************************************
* @ingroup LacSymQat_Cipher
* LacSymQat_CipherGetCfgData()
*
* @description
* setup the config data for cipher
*
* @param[in] pSession Pointer to the session data
*
* @param[in] pAlgorithm *
* @param[in] pMode
* @param[in] pDir
* @param[in] pKey_convert
*
* @retval void
*
*****************************************************************************/
void LacSymQat_CipherGetCfgData(lac_session_desc_t *pSession,
icp_qat_hw_cipher_algo_t *pAlgorithm,
icp_qat_hw_cipher_mode_t *pMode,
icp_qat_hw_cipher_dir_t *pDir,
icp_qat_hw_cipher_convert_t *pKey_convert);
/**
******************************************************************************
* @ingroup LacSymQat_Cipher
* LacSymQat_CipherHwBlockPopulateKeySetup()
*
* @description
* populate the key setup data in the cipher hardware control block
* in the common request message
*
* param[in] pCipherSetupData Pointer to cipher setup data
*
* @param[in] targetKeyLenInBytes Target key length. If key length given
* in cipher setup data is less that this,
* the key will be "rounded up" to this
* target length by padding it with 0's.
* In normal no-padding case, the target
* key length MUST match the key length
* in the cipher setup data.
*
* @param[in] pCipherHwBlock Pointer to the cipher hardware block
*
* @param[out] pCipherHwBlockSizeBytes Size in bytes of cipher setup block
*
*
* @retval void
*
*****************************************************************************/
void LacSymQat_CipherHwBlockPopulateKeySetup(
const CpaCySymCipherSetupData *pCipherSetupData,
Cpa32U targetKeyLenInBytes,
const void *pCipherHwBlock,
Cpa32U *pCipherHwBlockSizeBytes);
#endif /* LAC_SYM_QAT_CIPHER_H */

309
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_qat_hash.h Normal file
View File

@ -0,0 +1,309 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file lac_sym_qat_hash.h
*
* @defgroup LacSymQatHash Hash QAT
*
* @ingroup LacSymQat
*
* interfaces for populating qat structures for a hash operation
*
*****************************************************************************/
/*****************************************************************************/
#ifndef LAC_SYM_QAT_HASH_H
#define LAC_SYM_QAT_HASH_H
/*
******************************************************************************
* Include public/global header files
******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
#include "icp_qat_fw_la.h"
#include "icp_qat_hw.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "lac_common.h"
/**
******************************************************************************
* @ingroup LacSymQatHash
* hash precomputes
*
* @description
* This structure contains infomation on the hash precomputes
*
*****************************************************************************/
typedef struct lac_sym_qat_hash_precompute_info_s {
Cpa8U *pState1;
/**< state1 pointer */
Cpa32U state1Size;
/**< state1 size */
Cpa8U *pState2;
/**< state2 pointer */
Cpa32U state2Size;
/**< state2 size */
} lac_sym_qat_hash_precompute_info_t;
/**
******************************************************************************
* @ingroup LacSymQatHash
* hash state prefix buffer info
*
* @description
* This structure contains infomation on the hash state prefix aad buffer
*
*****************************************************************************/
typedef struct lac_sym_qat_hash_state_buffer_info_s {
Cpa64U pDataPhys;
/**< Physical pointer to the hash state prefix buffer */
Cpa8U *pData;
/**< Virtual pointer to the hash state prefix buffer */
Cpa8U stateStorageSzQuadWords;
/**< hash state storage size in quad words */
Cpa8U prefixAadSzQuadWords;
/**< inner prefix/aad and outer prefix size in quad words */
} lac_sym_qat_hash_state_buffer_info_t;
/**
******************************************************************************
* @ingroup LacSymQatHash
* Init the hash specific part of the content descriptor.
*
* @description
* This function populates the hash specific fields of the control block
* and the hardware setup block for a digest session. This function sets
* the size param to hold the size of the hash setup block.
*
* In the case of hash only, the content descriptor will contain just a
* hash control block and hash setup block. In the case of chaining it
* will contain the hash control block and setup block along with the
* control block and setup blocks of additional services.
*
* Note: The memory for the content descriptor MUST be allocated prior to
* calling this function. The memory for the hash control block and hash
* setup block MUST be set to 0 prior to calling this function.
*
* @image html contentDescriptor.png "Content Descriptor"
*
* @param[in] pMsg Pointer to req Parameter Footer
*
* @param[in] pHashSetupData Pointer to the hash setup data as
* defined in the LAC API.
*
* @param[in] pHwBlockBase Pointer to the base of the hardware
* setup block
*
* @param[in] hashBlkOffsetInHwBlock Offset in quad-words from the base of
* the hardware setup block where the
* hash block will start. This offset
* is stored in the control block. It
* is used to figure out where to write
* that hash setup block.
*
* @param[in] nextSlice SliceID for next control block
* entry This value is known only by
* the calling component
*
* @param[in] qatHashMode QAT hash mode
*
* @param[in] useSymConstantsTable Indicate if Shared-SRAM constants table
* is used for this session. If TRUE, the
* h/w setup block is NOT populated
*
* @param[in] useOptimisedContentDesc Indicate if optimised content desc
* is used for this session.
*
* @param[in] pPrecompute For auth mode, this is the pointer
* to the precompute data. Otherwise this
* should be set to NULL
*
* @param[out] pHashBlkSizeInBytes size in bytes of hash setup block
*
* @return void
*
*****************************************************************************/
void
LacSymQat_HashContentDescInit(icp_qat_la_bulk_req_ftr_t *pMsg,
CpaInstanceHandle instanceHandle,
const CpaCySymHashSetupData *pHashSetupData,
void *pHwBlockBase,
Cpa32U hashBlkOffsetInHwBlock,
icp_qat_fw_slice_t nextSlice,
icp_qat_hw_auth_mode_t qatHashMode,
CpaBoolean useSymConstantsTable,
CpaBoolean useOptimisedContentDesc,
lac_sym_qat_hash_precompute_info_t *pPrecompute,
Cpa32U *pHashBlkSizeInBytes);
/**
******************************************************************************
* @ingroup LacSymQatHash
* Calculate the size of the hash state prefix aad buffer
*
* @description
* This function inspects the hash control block and based on the values
* in the fields, it calculates the size of the hash state prefix aad
* buffer.
*
* A partial packet processing request is possible at any stage during a
* hash session. In this case, there will always be space for the hash
* state storage field of the hash state prefix buffer. When there is
* AAD data just the inner prefix AAD data field is used.
*
* @param[in] pMsg Pointer to the Request Message
*
* @param[out] pHashStateBuf Pointer to hash state prefix buffer info
* structure.
*
* @return None
*
*****************************************************************************/
void LacSymQat_HashStatePrefixAadBufferSizeGet(
icp_qat_la_bulk_req_ftr_t *pMsg,
lac_sym_qat_hash_state_buffer_info_t *pHashStateBuf);
/**
******************************************************************************
* @ingroup LacSymQatHash
* Populate the fields of the hash state prefix buffer
*
* @description
* This function populates the inner prefix/aad fields and/or the outer
* prefix field of the hash state prefix buffer.
*
* @param[in] pHashStateBuf Pointer to hash state prefix buffer info
* structure.
*
* @param[in] pMsg Pointer to the Request Message
*
* @param[in] pInnerPrefixAad Pointer to the Inner Prefix or Aad data
* This is NULL where if the data size is 0
*
* @param[in] innerPrefixSize Size of inner prefix/aad data in bytes
*
* @param[in] pOuterPrefix Pointer to the Outer Prefix data. This is
* NULL where the data size is 0.
*
* @param[in] outerPrefixSize Size of the outer prefix data in bytes
*
* @return void
*
*****************************************************************************/
void LacSymQat_HashStatePrefixAadBufferPopulate(
lac_sym_qat_hash_state_buffer_info_t *pHashStateBuf,
icp_qat_la_bulk_req_ftr_t *pMsg,
Cpa8U *pInnerPrefixAad,
Cpa8U innerPrefixSize,
Cpa8U *pOuterPrefix,
Cpa8U outerPrefixSize);
/**
******************************************************************************
* @ingroup LacSymQatHash
* Populate the hash request params structure
*
* @description
* This function is passed a pointer to the 128B Request block.
* (This memory must be allocated prior to calling this function). It
* populates the fields of this block using the parameters as described
* below. It is also expected that this structure has been set to 0
* prior to calling this function.
*
*
* @param[in] pReq Pointer to 128B request block.
*
* @param[in] authOffsetInBytes start offset of data that the digest is to
* be computed on.
*
* @param[in] authLenInBytes Length of data digest calculated on
*
* @param[in] pService Pointer to service data
*
* @param[in] pHashStateBuf Pointer to hash state buffer info. This
* structure contains the pointers and sizes.
* If there is no hash state prefix buffer
* required, this parameter can be set to NULL
*
* @param[in] qatPacketType Packet type using QAT macros. The hash
* state buffer pointer and state size will be
* different depending on the packet type
*
* @param[in] hashResultSize Size of the final hash result in bytes.
*
* @param[in] digestVerify Indicates if verify is enabled or not
*
* @param[in] pAuthResult Virtual pointer to digest
*
* @return CPA_STATUS_SUCCESS or CPA_STATUS_FAIL
*
*****************************************************************************/
CpaStatus LacSymQat_HashRequestParamsPopulate(
icp_qat_fw_la_bulk_req_t *pReq,
Cpa32U authOffsetInBytes,
Cpa32U authLenInBytes,
sal_service_t *pService,
lac_sym_qat_hash_state_buffer_info_t *pHashStateBuf,
Cpa32U qatPacketType,
Cpa32U hashResultSize,
CpaBoolean digestVerify,
Cpa8U *pAuthResult,
CpaCySymHashAlgorithm alg,
void *data);
/**
******************************************************************************
* @ingroup LacSymQatHash
*
*
* @description
* This fn returns the QAT values for hash algorithm and nested fields
*
*
* @param[in] pInstance Pointer to service instance.
*
* @param[in] qatHashMode value for hash mode on the fw qat
*interface.
*
* @param[in] apiHashMode value for hash mode on the QA API.
*
* @param[in] apiHashAlgorithm value for hash algorithm on the QA API.
*
* @param[out] pQatAlgorithm Pointer to return fw qat value for
*algorithm.
*
* @param[out] pQatNested Pointer to return fw qat value for nested.
*
*
* @return
* none
*
*****************************************************************************/
void LacSymQat_HashGetCfgData(CpaInstanceHandle pInstance,
icp_qat_hw_auth_mode_t qatHashMode,
CpaCySymHashMode apiHashMode,
CpaCySymHashAlgorithm apiHashAlgorithm,
icp_qat_hw_auth_algo_t *pQatAlgorithm,
CpaBoolean *pQatNested);
void LacSymQat_HashSetupReqParamsMetaData(
icp_qat_la_bulk_req_ftr_t *pMsg,
CpaInstanceHandle instanceHandle,
const CpaCySymHashSetupData *pHashSetupData,
CpaBoolean hashStateBuffer,
icp_qat_hw_auth_mode_t qatHashMode,
CpaBoolean digestVerify);
#endif /* LAC_SYM_QAT_HASH_H */

139
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_qat_hash_defs_lookup.h Normal file
View File

@ -0,0 +1,139 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file lac_sym_qat_hash_defs_lookup.h
*
* @defgroup LacSymQatHashDefsLookup Hash Defs Lookup
*
* @ingroup LacSymQatHash
*
* API to be used for the hash defs lookup table.
*
*****************************************************************************/
#ifndef LAC_SYM_QAT_HASH_DEFS_LOOKUP_P_H
#define LAC_SYM_QAT_HASH_DEFS_LOOKUP_P_H
#include "cpa.h"
#include "cpa_cy_sym.h"
/**
******************************************************************************
* @ingroup LacSymQatHashDefsLookup
* Finishing Hash algorithm
* @description
* This define points to the last available hash algorithm
* @NOTE: If a new algorithm is added to the api, this #define
* MUST be updated to being the last hash algorithm in the struct
* CpaCySymHashAlgorithm in the file cpa_cy_sym.h
*****************************************************************************/
#define CPA_CY_HASH_ALG_END CPA_CY_SYM_HASH_SM3
/***************************************************************************/
/**
******************************************************************************
* @ingroup LacSymQatHashDefsLookup
* hash algorithm specific structure
* @description
* This structure contain constants specific to an algorithm.
*****************************************************************************/
typedef struct lac_sym_qat_hash_alg_info_s {
Cpa32U digestLength; /**< Digest length in bytes */
Cpa32U blockLength; /**< Block length in bytes */
Cpa8U *initState; /**< Initialiser state for hash algorithm */
Cpa32U stateSize; /**< size of above state in bytes */
} lac_sym_qat_hash_alg_info_t;
/**
******************************************************************************
* @ingroup LacSymQatHashDefsLookup
* hash qat specific structure
* @description
* This structure contain constants as defined by the QAT for an
* algorithm.
*****************************************************************************/
typedef struct lac_sym_qat_hash_qat_info_s {
Cpa32U algoEnc; /**< QAT Algorithm encoding */
Cpa32U authCounter; /**< Counter value for Auth */
Cpa32U state1Length; /**< QAT state1 length in bytes */
Cpa32U state2Length; /**< QAT state2 length in bytes */
} lac_sym_qat_hash_qat_info_t;
/**
******************************************************************************
* @ingroup LacSymQatHashDefsLookup
* hash defs structure
* @description
* This type contains pointers to the hash algorithm structure and
* to the hash qat specific structure
*****************************************************************************/
typedef struct lac_sym_qat_hash_defs_s {
lac_sym_qat_hash_alg_info_t *algInfo;
/**< pointer to hash info structure */
lac_sym_qat_hash_qat_info_t *qatInfo;
/**< pointer to hash QAT info structure */
} lac_sym_qat_hash_defs_t;
/**
*******************************************************************************
* @ingroup LacSymQatHashDefsLookup
* initialise the hash lookup table
*
* @description
* This function initialises the digest lookup table.
*
* @note
* This function does not have a corresponding shutdown function.
*
* @return CPA_STATUS_SUCCESS Operation successful
* @return CPA_STATUS_RESOURCE Allocating of hash lookup table failed
*
*****************************************************************************/
CpaStatus LacSymQat_HashLookupInit(CpaInstanceHandle instanceHandle);
/**
*******************************************************************************
* @ingroup LacSymQatHashDefsLookup
* get hash algorithm specific structure from lookup table
*
* @description
* This function looks up the hash lookup array for a structure
* containing data specific to a hash algorithm. The hashAlgorithm enum
* value MUST be in the correct range prior to calling this function.
*
* @param[in] hashAlgorithm Hash Algorithm
* @param[out] ppHashAlgInfo Hash Alg Info structure
*
* @return None
*
*****************************************************************************/
void LacSymQat_HashAlgLookupGet(CpaInstanceHandle instanceHandle,
CpaCySymHashAlgorithm hashAlgorithm,
lac_sym_qat_hash_alg_info_t **ppHashAlgInfo);
/**
*******************************************************************************
* @ingroup LacSymQatHashDefsLookup
* get hash defintions from lookup table.
*
* @description
* This function looks up the hash lookup array for a structure
* containing data specific to a hash algorithm. This includes both
* algorithm specific info and qat specific infro. The hashAlgorithm enum
* value MUST be in the correct range prior to calling this function.
*
* @param[in] hashAlgorithm Hash Algorithm
* @param[out] ppHashDefsInfo Hash Defs structure
*
* @return void
*
*****************************************************************************/
void LacSymQat_HashDefsLookupGet(CpaInstanceHandle instanceHandle,
CpaCySymHashAlgorithm hashAlgorithm,
lac_sym_qat_hash_defs_t **ppHashDefsInfo);
#endif /* LAC_SYM_QAT_HASH_DEFS_LOOKUP_P_H */

189
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_qat_key.h Normal file
View File

@ -0,0 +1,189 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file lac_sym_qat_key.h
*
* @defgroup LacSymQatKey Key QAT
*
* @ingroup LacSymQat
*
* interfaces for populating qat structures for a key operation
*
*****************************************************************************/
#ifndef LAC_SYM_QAT_KEY_H
#define LAC_SYM_QAT_KEY_H
#include "cpa.h"
#include "lac_sym.h"
#include "icp_qat_fw_la.h"
/**
******************************************************************************
* @ingroup LacSymQatKey
* Number of bytes generated per iteration
* @description
* This define is the number of bytes generated per iteration
*****************************************************************************/
#define LAC_SYM_QAT_KEY_SSL_BYTES_PER_ITERATION (16)
/**
******************************************************************************
* @ingroup LacSymQatKey
* Shift to calculate the number of iterations
* @description
* This define is the shift to calculate the number of iterations
*****************************************************************************/
#define LAC_SYM_QAT_KEY_SSL_ITERATIONS_SHIFT LAC_16BYTE_ALIGNMENT_SHIFT
/**
*******************************************************************************
* @ingroup LacSymKey
* Populate the SSL request
*
* @description
* Populate the SSL request
*
* @param[out] pKeyGenReqHdr Pointer to Key Generation request Header
* @param[out] pKeyGenReqMid Pointer to LW's 14/15 of Key Gen request
* @param[in] generatedKeyLenInBytes Length of Key generated
* @param[in] labelLenInBytes Length of Label
* @param[in] secretLenInBytes Length of Secret
* @param[in] iterations Number of iterations. This is related
* to the label length.
*
* @return None
*
*****************************************************************************/
void
LacSymQat_KeySslRequestPopulate(icp_qat_la_bulk_req_hdr_t *pKeyGenReqHdr,
icp_qat_fw_la_key_gen_common_t *pKeyGenReqMid,
Cpa32U generatedKeyLenInBytes,
Cpa32U labelLenInBytes,
Cpa32U secretLenInBytes,
Cpa32U iterations);
/**
*******************************************************************************
* @ingroup LacSymKey
* Populate the TLS request
*
* @description
* Populate the TLS request
*
* @param[out] pKeyGenReq Pointer to Key Generation request
* @param[in] generatedKeyLenInBytes Length of Key generated
* @param[in] labelLenInBytes Length of Label
* @param[in] secretLenInBytes Length of Secret
* @param[in] seedLenInBytes Length of Seed
* @param[in] cmdId Command Id to differentiate TLS versions
*
* @return None
*
*****************************************************************************/
void LacSymQat_KeyTlsRequestPopulate(
icp_qat_fw_la_key_gen_common_t *pKeyGenReqParams,
Cpa32U generatedKeyLenInBytes,
Cpa32U labelLenInBytes,
Cpa32U secretLenInBytes,
Cpa8U seedLenInBytes,
icp_qat_fw_la_cmd_id_t cmdId);
/**
*******************************************************************************
* @ingroup LacSymKey
* Populate MGF request
*
* @description
* Populate MGF request
*
* @param[out] pKeyGenReqHdr Pointer to Key Generation request Header
* @param[out] pKeyGenReqMid Pointer to LW's 14/15 of Key Gen request
* @param[in] seedLenInBytes Length of Seed
* @param[in] maskLenInBytes Length of Mask
* @param[in] hashLenInBytes Length of hash
*
* @return None
*
*****************************************************************************/
void
LacSymQat_KeyMgfRequestPopulate(icp_qat_la_bulk_req_hdr_t *pKeyGenReqHdr,
icp_qat_fw_la_key_gen_common_t *pKeyGenReqMid,
Cpa8U seedLenInBytes,
Cpa16U maskLenInBytes,
Cpa8U hashLenInBytes);
/**
*******************************************************************************
* @ingroup LacSymKey
* Populate the SSL key material input
*
* @description
* Populate the SSL key material input
*
* @param[in] pService Pointer to service
* @param[out] pSslKeyMaterialInput Pointer to SSL key material input
* @param[in] pSeed Pointer to Seed
* @param[in] labelPhysAddr Physical address of the label
* @param[in] pSecret Pointer to Secret
*
* @return None
*
*****************************************************************************/
void LacSymQat_KeySslKeyMaterialInputPopulate(
sal_service_t *pService,
icp_qat_fw_la_ssl_key_material_input_t *pSslKeyMaterialInput,
void *pSeed,
Cpa64U labelPhysAddr,
void *pSecret);
/**
*******************************************************************************
* @ingroup LacSymKey
* Populate the TLS key material input
*
* @description
* Populate the TLS key material input
*
* @param[in] pService Pointer to service
* @param[out] pTlsKeyMaterialInput Pointer to TLS key material input
* @param[in] pSeed Pointer to Seed
* @param[in] labelPhysAddr Physical address of the label
*
* @return None
*
*****************************************************************************/
void LacSymQat_KeyTlsKeyMaterialInputPopulate(
sal_service_t *pService,
icp_qat_fw_la_tls_key_material_input_t *pTlsKeyMaterialInput,
void *pSeed,
Cpa64U labelPhysAddr);
/**
*******************************************************************************
* @ingroup LacSymKey
* Populate the TLS HKDF key material input
*
* @description
* Populate the TLS HKDF key material input
*
* @param[in] pService Pointer to service
* @param[out] pTlsKeyMaterialInput Pointer to TLS key material input
* @param[in] pSeed Pointer to Seed
* @param[in] labelPhysAddr Physical address of the label
* @param[in] cmdId Command ID
*
* @return None
*
*****************************************************************************/
void LacSymQat_KeyTlsHKDFKeyMaterialInputPopulate(
sal_service_t *pService,
icp_qat_fw_la_hkdf_key_material_input_t *pTlsKeyMaterialInput,
CpaCyKeyGenHKDFOpData *pKeyGenTlsOpData,
Cpa64U subLabelsPhysAddr,
icp_qat_fw_la_cmd_id_t cmdId);
#endif /* LAC_SYM_QAT_KEY_H */

51
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_queue.h Normal file
View File

@ -0,0 +1,51 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
******************************************************************************
* @file lac_sym_queue.h
*
* @defgroup LacSymQueue Symmetric request queueing functions
*
* @ingroup LacSym
*
* Function prototypes for sending/queuing symmetric requests
*****************************************************************************/
#ifndef LAC_SYM_QUEUE_H
#define LAC_SYM_QUEUE_H
#include "cpa.h"
#include "lac_session.h"
#include "lac_sym.h"
/**
*******************************************************************************
* @ingroup LacSymQueue
* Send a request message to the QAT, or queue it if necessary
*
* @description
* This function will send a request message to the QAT. However, if a
* blocking condition exists on the session (e.g. partial packet in flight,
* precompute in progress), then the message will instead be pushed on to
* the request queue for the session and will be sent later to the QAT
* once the blocking condition is cleared.
*
* @param[in] instanceHandle Handle for instance of QAT
* @param[in] pRequest Pointer to request cookie
* @param[out] pSessionDesc Pointer to session descriptor
*
*
* @retval CPA_STATUS_SUCCESS Success
* @retval CPA_STATUS_FAIL Function failed.
* @retval CPA_STATUS_RESOURCE Problem Acquiring system resource
* @retval CPA_STATUS_RETRY Failed to send message to QAT due to queue
* full condition
*
*****************************************************************************/
CpaStatus LacSymQueue_RequestSend(const CpaInstanceHandle instanceHandle,
lac_sym_bulk_cookie_t *pRequest,
lac_session_desc_t *pSessionDesc);
#endif /* LAC_SYM_QUEUE_H */

191
sys/dev/qat/qat_api/common/crypto/sym/include/lac_sym_stats.h Normal file
View File

@ -0,0 +1,191 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_stats.h
*
* @defgroup LacSymCommon Symmetric Common
*
* @ingroup LacSym
*
* Symetric Common consists of common statistics, buffer and partial packet
* functionality.
*
***************************************************************************/
/**
***************************************************************************
* @defgroup LacSymStats Statistics
*
* @ingroup LacSymCommon
*
* definitions and prototypes for LAC symmetric statistics.
*
* @lld_start
* In the LAC API the stats fields are defined as Cpa32U but
* QatUtilsAtomic is the type that the atomic API supports. Therefore we
* need to define a structure internally with the same fields as the API
* stats structure, but each field must be of type QatUtilsAtomic.
*
* - <b>Incrementing Statistics:</b>\n
* Atomically increment the statistic on the internal stats structure.
*
* - <b>Providing a copy of the stats back to the user:</b>\n
* Use atomicGet to read the atomic variable for each stat field in the
* local internal stat structure. These values are saved in structure
* (as defined by the LAC API) that the client will provide a pointer
* to as a parameter.
*
* - <b>Stats Show:</b>\n
* Use atomicGet to read the atomic variables for each field in the local
* internal stat structure and print to the screen
*
* - <b>Stats Array:</b>\n
* A macro is used to get the offset off the stat in the structure. This
* offset is passed to a function which uses it to increment the stat
* at that offset.
*
* @lld_end
*
***************************************************************************/
/***************************************************************************/
#ifndef LAC_SYM_STATS_H
#define LAC_SYM_STATS_H
/*
******************************************************************************
* Include public/global header files
******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
#include "cpa_cy_common.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
/**
*******************************************************************************
* @ingroup LacSymStats
* increment a symmetric statistic
*
* @description
* Increment the statistics
*
* @param statistic IN The field in the symmetric statistics structure to be
* incremented
* @param instanceHandle IN engine Id Number
*
* @retval None
*
*****************************************************************************/
#define LAC_SYM_STAT_INC(statistic, instanceHandle) \
LacSym_StatsInc(offsetof(CpaCySymStats64, statistic), instanceHandle)
/**
*******************************************************************************
* @ingroup LacSymStats
* initialises the symmetric stats
*
* @description
* This function allocates and initialises the stats array to 0
*
* @param instanceHandle Instance Handle
*
* @retval CPA_STATUS_SUCCESS initialisation successful
* @retval CPA_STATUS_RESOURCE array allocation failed
*
*****************************************************************************/
CpaStatus LacSym_StatsInit(CpaInstanceHandle instanceHandle);
/**
*******************************************************************************
* @ingroup LacSymStats
* Frees the symmetric stats
*
* @description
* This function frees the stats array
*
* @param instanceHandle Instance Handle
*
* @retval None
*
*****************************************************************************/
void LacSym_StatsFree(CpaInstanceHandle instanceHandle);
/**
*******************************************************************************
* @ingroup LacSymStats
* Inrement a stat
*
* @description
* This function incrementes a stat for a specific engine.
*
* @param offset IN offset of stat field in structure
* @param instanceHandle IN qat Handle
*
* @retval None
*
*****************************************************************************/
void LacSym_StatsInc(Cpa32U offset, CpaInstanceHandle instanceHandle);
/**
*******************************************************************************
* @ingroup LacSymStats
* Copy the contents of the statistics structure for an engine
*
* @description
* This function copies the 32bit symmetric statistics structure for
* a specific engine into an address supplied as a parameter.
*
* @param instanceHandle IN engine Id Number
* @param pSymStats OUT stats structure to copy the stats for the into
*
* @retval None
*
*****************************************************************************/
void LacSym_Stats32CopyGet(CpaInstanceHandle instanceHandle,
struct _CpaCySymStats *const pSymStats);
/**
*******************************************************************************
* @ingroup LacSymStats
* Copy the contents of the statistics structure for an engine
*
* @description
* This function copies the 64bit symmetric statistics structure for
* a specific engine into an address supplied as a parameter.
*
* @param instanceHandle IN engine Id Number
* @param pSymStats OUT stats structure to copy the stats for the into
*
* @retval None
*
*****************************************************************************/
void LacSym_Stats64CopyGet(CpaInstanceHandle instanceHandle,
CpaCySymStats64 *const pSymStats);
/**
*******************************************************************************
* @ingroup LacSymStats
* print the symmetric stats to standard output
*
* @description
* The statistics for symmetric are printed to standard output.
*
* @retval None
*
* @see LacSym_StatsCopyGet()
*
*****************************************************************************/
void LacSym_StatsShow(CpaInstanceHandle instanceHandle);
#endif /*LAC_SYM_STATS_H_*/

3021
sys/dev/qat/qat_api/common/crypto/sym/key/lac_sym_key.c Normal file
View File

File diff suppressed because it is too large Load Diff

1860
sys/dev/qat/qat_api/common/crypto/sym/lac_sym_alg_chain.c Normal file
View File

File diff suppressed because it is too large Load Diff

1130
sys/dev/qat/qat_api/common/crypto/sym/lac_sym_api.c Normal file
View File

File diff suppressed because it is too large Load Diff

197
sys/dev/qat/qat_api/common/crypto/sym/lac_sym_auth_enc.c Normal file
View File

@ -0,0 +1,197 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_auth_enc.c
*
* @ingroup LacAuthEnc
*
* @description
* Authenticated encryption specific functionality.
* For CCM related code NIST SP 800-38C is followed.
* For GCM related code NIST SP 800-38D is followed.
***************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
#include "icp_accel_devices.h"
#include "icp_adf_init.h"
#include "icp_adf_transport.h"
#include "icp_adf_debug.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "lac_log.h"
#include "lac_common.h"
#include "lac_session.h"
#include "lac_sym_auth_enc.h"
/* These defines describe position of the flag fields
* in B0 block for CCM algorithm*/
#define LAC_ALG_CHAIN_CCM_B0_FLAGS_ADATA_SHIFT 6
#define LAC_ALG_CHAIN_CCM_B0_FLAGS_T_SHIFT 3
/* This macro builds flags field to be put in B0 block for CCM algorithm */
#define LAC_ALG_CHAIN_CCM_BUILD_B0_FLAGS(Adata, t, q) \
((((Adata) > 0 ? 1 : 0) << LAC_ALG_CHAIN_CCM_B0_FLAGS_ADATA_SHIFT) | \
((((t)-2) >> 1) << LAC_ALG_CHAIN_CCM_B0_FLAGS_T_SHIFT) | ((q)-1))
/**
* @ingroup LacAuthEnc
*/
CpaStatus
LacSymAlgChain_CheckCCMData(Cpa8U *pAdditionalAuthData,
Cpa8U *pIv,
Cpa32U messageLenToCipherInBytes,
Cpa32U ivLenInBytes)
{
Cpa8U q = 0;
LAC_CHECK_NULL_PARAM(pIv);
LAC_CHECK_NULL_PARAM(pAdditionalAuthData);
/* check if n is within permitted range */
if (ivLenInBytes < LAC_ALG_CHAIN_CCM_N_LEN_IN_BYTES_MIN ||
ivLenInBytes > LAC_ALG_CHAIN_CCM_N_LEN_IN_BYTES_MAX) {
LAC_INVALID_PARAM_LOG2("ivLenInBytes for CCM algorithm "
"must be between %d and %d inclusive",
LAC_ALG_CHAIN_CCM_N_LEN_IN_BYTES_MIN,
LAC_ALG_CHAIN_CCM_N_LEN_IN_BYTES_MAX);
return CPA_STATUS_INVALID_PARAM;
}
q = LAC_ALG_CHAIN_CCM_NQ_CONST - ivLenInBytes;
/* Check if q is big enough to hold actual length of message to cipher
* if q = 8 -> maxlen = 2^64 always good as
* messageLenToCipherInBytes is 32 bits
* if q = 7 -> maxlen = 2^56 always good
* if q = 6 -> maxlen = 2^48 always good
* if q = 5 -> maxlen = 2^40 always good
* if q = 4 -> maxlen = 2^32 always good.
*/
if ((messageLenToCipherInBytes >= (1 << (q * LAC_NUM_BITS_IN_BYTE))) &&
(q < sizeof(Cpa32U))) {
LAC_INVALID_PARAM_LOG(
"messageLenToCipherInBytes too long for the given"
" ivLenInBytes for CCM algorithm\n");
return CPA_STATUS_INVALID_PARAM;
}
return CPA_STATUS_SUCCESS;
}
/**
* @ingroup LacAuthEnc
*/
void
LacSymAlgChain_PrepareCCMData(lac_session_desc_t *pSessionDesc,
Cpa8U *pAdditionalAuthData,
Cpa8U *pIv,
Cpa32U messageLenToCipherInBytes,
Cpa32U ivLenInBytes)
{
Cpa8U n =
ivLenInBytes; /* assumes ivLenInBytes has been param checked */
Cpa8U q = LAC_ALG_CHAIN_CCM_NQ_CONST - n;
Cpa8U lenOfEncodedLen = 0;
Cpa16U lenAEncoded = 0;
Cpa32U bitStrQ = 0;
/* populate Ctr0 block - stored in pIv */
pIv[0] = (q - 1);
/* bytes 1 to n are already set with nonce by the user */
/* set last q bytes with 0 */
memset(pIv + n + 1, 0, q);
/* Encode the length of associated data 'a'. As the API limits the
* length
* of an array pointed by pAdditionalAuthData to be 240 bytes max, the
* maximum length of 'a' might be 240 - 16 - 2 = 222. Hence the encoding
* below is simplified. */
if (pSessionDesc->aadLenInBytes > 0) {
lenOfEncodedLen = sizeof(Cpa16U);
lenAEncoded = QAT_UTILS_HOST_TO_NW_16(
(Cpa16U)pSessionDesc->aadLenInBytes);
}
/* populate B0 block */
/* first, set the flags field */
pAdditionalAuthData[0] =
LAC_ALG_CHAIN_CCM_BUILD_B0_FLAGS(lenOfEncodedLen,
pSessionDesc->hashResultSize,
q);
/* bytes 1 to n are already set with nonce by the user*/
/* put Q in bytes 16-q...15 */
bitStrQ = QAT_UTILS_HOST_TO_NW_32(messageLenToCipherInBytes);
if (q > sizeof(bitStrQ)) {
memset(pAdditionalAuthData + n + 1, 0, q);
memcpy(pAdditionalAuthData + n + 1 + (q - sizeof(bitStrQ)),
(Cpa8U *)&bitStrQ,
sizeof(bitStrQ));
} else {
memcpy(pAdditionalAuthData + n + 1,
((Cpa8U *)&bitStrQ) + (sizeof(bitStrQ) - q),
q);
}
/* populate B1-Bn blocks */
if (lenAEncoded > 0) {
*(Cpa16U
*)(&pAdditionalAuthData[1 + LAC_ALG_CHAIN_CCM_NQ_CONST]) =
lenAEncoded;
/* Next bytes are already set by the user with
* the associated data 'a' */
/* Check if padding is required */
if (((pSessionDesc->aadLenInBytes + lenOfEncodedLen) %
LAC_HASH_AES_CCM_BLOCK_SIZE) != 0) {
Cpa8U paddingLen = 0;
Cpa8U paddingIndex = 0;
paddingLen = LAC_HASH_AES_CCM_BLOCK_SIZE -
((pSessionDesc->aadLenInBytes + lenOfEncodedLen) %
LAC_HASH_AES_CCM_BLOCK_SIZE);
paddingIndex = 1 + LAC_ALG_CHAIN_CCM_NQ_CONST;
paddingIndex +=
lenOfEncodedLen + pSessionDesc->aadLenInBytes;
memset(&pAdditionalAuthData[paddingIndex],
0,
paddingLen);
}
}
}
/**
* @ingroup LacAuthEnc
*/
void
LacSymAlgChain_PrepareGCMData(lac_session_desc_t *pSessionDesc,
Cpa8U *pAdditionalAuthData)
{
Cpa8U paddingLen = 0;
if ((pSessionDesc->aadLenInBytes % LAC_HASH_AES_GCM_BLOCK_SIZE) != 0) {
paddingLen = LAC_HASH_AES_GCM_BLOCK_SIZE -
(pSessionDesc->aadLenInBytes % LAC_HASH_AES_GCM_BLOCK_SIZE);
memset(&pAdditionalAuthData[pSessionDesc->aadLenInBytes],
0,
paddingLen);
}
}

545
sys/dev/qat/qat_api/common/crypto/sym/lac_sym_cb.c Normal file
View File

@ -0,0 +1,545 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_cb.c Callback handler functions for symmetric components
*
* @ingroup LacSym
*
***************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
#include "icp_accel_devices.h"
#include "icp_adf_init.h"
#include "icp_qat_fw_la.h"
#include "icp_adf_transport.h"
#include "icp_adf_debug.h"
#include "lac_sym.h"
#include "lac_sym_cipher.h"
#include "lac_common.h"
#include "lac_list.h"
#include "lac_sal_types_crypto.h"
#include "lac_sal.h"
#include "lac_sal_ctrl.h"
#include "lac_session.h"
#include "lac_sym_stats.h"
#include "lac_log.h"
#include "lac_sym_cb.h"
#include "lac_sym_hash.h"
#include "lac_sym_qat_cipher.h"
#include "lac_sym_qat.h"
#define DEQUEUE_MSGPUT_MAX_RETRIES 10000
/*
*******************************************************************************
* Define static function definitions
*******************************************************************************
*/
/**
*****************************************************************************
* @ingroup LacSymCb
* Function to clean computed data.
*
* @description
* This function cleans GCM or CCM data in the case of a failure.
*
* @param[in] pSessionDesc pointer to the session descriptor
* @param[out] pBufferList pointer to the bufferlist to clean
* @param[in] pOpData pointer to operation data
* @param[in] isCCM is it a CCM operation boolean
*
* @return None
*****************************************************************************/
static void
LacSymCb_CleanUserData(const lac_session_desc_t *pSessionDesc,
CpaBufferList *pBufferList,
const CpaCySymOpData *pOpData,
CpaBoolean isCCM)
{
Cpa8U authTagLen = 0;
/* Retrieve authTagLen */
authTagLen = pSessionDesc->hashResultSize;
/* Cleaning */
if (isCCM) {
/* for CCM the digest is inside the buffer list */
LacBuffDesc_BufferListZeroFromOffset(
pBufferList,
pOpData->cryptoStartSrcOffsetInBytes,
pOpData->messageLenToCipherInBytes + authTagLen);
} else {
/* clean buffer list */
LacBuffDesc_BufferListZeroFromOffset(
pBufferList,
pOpData->cryptoStartSrcOffsetInBytes,
pOpData->messageLenToCipherInBytes);
}
if ((CPA_TRUE != pSessionDesc->digestIsAppended) &&
(NULL != pOpData->pDigestResult)) {
/* clean digest */
memset(pOpData->pDigestResult, 0, authTagLen);
}
}
/**
*****************************************************************************
* @ingroup LacSymCb
* Definition of callback function for processing symmetric responses
*
* @description
* This callback is invoked to process symmetric response messages from
* the QAT. It will extract some details from the message and invoke
* the user's callback to complete a symmetric operation.
*
* @param[in] pCookie Pointer to cookie associated with this request
* @param[in] qatRespStatusOkFlag Boolean indicating ok/fail status from QAT
* @param[in] status Status variable indicating an error occurred
* in sending the message (e.g. when dequeueing)
* @param[in] pSessionDesc Session descriptor
*
* @return None
*****************************************************************************/
static void
LacSymCb_ProcessCallbackInternal(lac_sym_bulk_cookie_t *pCookie,
CpaBoolean qatRespStatusOkFlag,
CpaStatus status,
lac_session_desc_t *pSessionDesc)
{
CpaCySymCbFunc pSymCb = NULL;
void *pCallbackTag = NULL;
CpaCySymOpData *pOpData = NULL;
CpaBufferList *pDstBuffer = NULL;
CpaCySymOp operationType = CPA_CY_SYM_OP_NONE;
CpaStatus dequeueStatus = CPA_STATUS_SUCCESS;
CpaInstanceHandle instanceHandle = CPA_INSTANCE_HANDLE_SINGLE;
/* NOTE: cookie pointer validated in previous function */
instanceHandle = pCookie->instanceHandle;
pOpData = (CpaCySymOpData *)LAC_CONST_PTR_CAST(pCookie->pOpData);
operationType = pSessionDesc->symOperation;
/* Set the destination pointer to the one supplied in the cookie. */
pDstBuffer = pCookie->pDstBuffer;
/* For a digest verify operation - for full packet and final partial
* only, perform a comparison with the digest generated and with the one
* supplied in the packet. */
if (((pSessionDesc->isSinglePass &&
(CPA_CY_SYM_CIPHER_AES_GCM == pSessionDesc->cipherAlgorithm)) ||
(CPA_CY_SYM_OP_CIPHER != operationType)) &&
(CPA_TRUE == pSessionDesc->digestVerify) &&
((CPA_CY_SYM_PACKET_TYPE_FULL == pOpData->packetType) ||
(CPA_CY_SYM_PACKET_TYPE_LAST_PARTIAL == pOpData->packetType))) {
if (CPA_FALSE == qatRespStatusOkFlag) {
LAC_SYM_STAT_INC(numSymOpVerifyFailures,
instanceHandle);
/* The comparison has failed at this point (status is
* fail),
* need to clean any sensitive calculated data up to
* this point.
* The data calculated is no longer useful to the end
* result and
* does not need to be returned to the user so setting
* buffers to
* zero.
*/
if (pSessionDesc->cipherAlgorithm ==
CPA_CY_SYM_CIPHER_AES_CCM) {
LacSymCb_CleanUserData(pSessionDesc,
pDstBuffer,
pOpData,
CPA_TRUE);
} else if (pSessionDesc->cipherAlgorithm ==
CPA_CY_SYM_CIPHER_AES_GCM) {
LacSymCb_CleanUserData(pSessionDesc,
pDstBuffer,
pOpData,
CPA_FALSE);
}
}
} else {
/* Most commands have no point of failure and always return
* success. This is the default response from the QAT.
* If status is already set to an error value, don't overwrite
* it
*/
if ((CPA_STATUS_SUCCESS == status) &&
(CPA_TRUE != qatRespStatusOkFlag)) {
LAC_LOG_ERROR("Response status value not as expected");
status = CPA_STATUS_FAIL;
}
}
pSymCb = pSessionDesc->pSymCb;
pCallbackTag = pCookie->pCallbackTag;
/* State returned to the client for intermediate partials packets
* for hash only and cipher only partial packets. Cipher update
* allow next partial through */
if (CPA_CY_SYM_PACKET_TYPE_PARTIAL == pOpData->packetType) {
if ((CPA_CY_SYM_OP_CIPHER == operationType) ||
(CPA_CY_SYM_OP_ALGORITHM_CHAINING == operationType)) {
if (CPA_TRUE == pCookie->updateUserIvOnRecieve) {
/* Update the user's IV buffer
* Very important to do this BEFORE dequeuing
* subsequent partial requests, as the state
* buffer
* may get overwritten
*/
memcpy(pCookie->pOpData->pIv,
pSessionDesc->cipherPartialOpState,
pCookie->pOpData->ivLenInBytes);
}
if (CPA_TRUE == pCookie->updateKeySizeOnRecieve &&
LAC_CIPHER_IS_XTS_MODE(
pSessionDesc->cipherAlgorithm)) {
LacSymQat_CipherXTSModeUpdateKeyLen(
pSessionDesc,
pSessionDesc->cipherKeyLenInBytes / 2);
}
}
} else if (CPA_CY_SYM_PACKET_TYPE_LAST_PARTIAL == pOpData->packetType) {
if ((CPA_CY_SYM_OP_CIPHER == operationType) ||
(CPA_CY_SYM_OP_ALGORITHM_CHAINING == operationType)) {
if (CPA_TRUE == LAC_CIPHER_IS_XTS_MODE(
pSessionDesc->cipherAlgorithm)) {
/*
* For XTS mode, we replace the updated key with
* the original key - for subsequent partial
* requests
*
*/
LacSymQat_CipherXTSModeUpdateKeyLen(
pSessionDesc,
pSessionDesc->cipherKeyLenInBytes);
}
}
}
if ((CPA_CY_SYM_PACKET_TYPE_FULL != pOpData->packetType) &&
(qatRespStatusOkFlag != CPA_FALSE)) {
/* There may be requests blocked pending the completion of this
* operation
*/
dequeueStatus = LacSymCb_PendingReqsDequeue(pSessionDesc);
if (CPA_STATUS_SUCCESS != dequeueStatus) {
LAC_SYM_STAT_INC(numSymOpCompletedErrors,
instanceHandle);
qatRespStatusOkFlag = CPA_FALSE;
if (CPA_STATUS_SUCCESS == status) {
status = dequeueStatus;
}
}
}
if (CPA_STATUS_SUCCESS == status) {
/* update stats */
if (pSessionDesc->internalSession == CPA_FALSE) {
LAC_SYM_STAT_INC(numSymOpCompleted, instanceHandle);
if (CPA_STATUS_SUCCESS != status) {
LAC_SYM_STAT_INC(numSymOpCompletedErrors,
instanceHandle);
}
}
}
qatUtilsAtomicDec(&(pSessionDesc->u.pendingCbCount));
/* deallocate the memory for the internal callback cookie */
Lac_MemPoolEntryFree(pCookie);
/* user callback function is the last thing to be called */
pSymCb(pCallbackTag,
status,
operationType,
pOpData,
pDstBuffer,
qatRespStatusOkFlag);
}
/**
******************************************************************************
* @ingroup LacSymCb
* Definition of callback function for processing symmetric Data Plane
* responses
*
* @description
* This callback checks the status, decrements the number of operations
* pending and calls the user callback
*
* @param[in/out] pResponse pointer to the response structure
* @param[in] qatRespStatusOkFlag status
* @param[in] pSessionDesc pointer to the session descriptor
*
* @return None
******************************************************************************/
static void
LacSymCb_ProcessDpCallback(CpaCySymDpOpData *pResponse,
CpaBoolean qatRespStatusOkFlag,
lac_session_desc_t *pSessionDesc)
{
CpaStatus status = CPA_STATUS_SUCCESS;
/* For CCM and GCM, if qatRespStatusOkFlag is false, the data has to be
* cleaned as stated in RFC 3610; in DP mode, it is the user
* responsability
* to do so */
if (CPA_FALSE == pSessionDesc->isSinglePass) {
if ((CPA_CY_SYM_OP_CIPHER == pSessionDesc->symOperation) ||
(CPA_FALSE == pSessionDesc->digestVerify)) {
/* If not doing digest compare and qatRespStatusOkFlag
!=
CPA_TRUE
then there is something very wrong */
if (CPA_FALSE == qatRespStatusOkFlag) {
LAC_LOG_ERROR(
"Response status value not as expected");
status = CPA_STATUS_FAIL;
}
}
}
((sal_crypto_service_t *)pResponse->instanceHandle)
->pSymDpCb(pResponse, status, qatRespStatusOkFlag);
/*
* Decrement the number of pending CB.
*
* If the @pendingDpCbCount becomes zero, we may remove the session,
* please
* read more information in the cpaCySymRemoveSession().
*
* But there is a field in the @pResponse to store the session,
* the "sessionCtx". In another word, in the above @->pSymDpCb()
* callback,
* it may use the session again. If we decrease the @pendingDpCbCount
* before
* the @->pSymDpCb(), there is a _risk_ the @->pSymDpCb() may reference
* to
* a deleted session.
*
* So in order to avoid the risk, we decrease the @pendingDpCbCount
* after
* the @->pSymDpCb() callback.
*/
qatUtilsAtomicDec(&pSessionDesc->u.pendingDpCbCount);
}
/**
******************************************************************************
* @ingroup LacSymCb
* Definition of callback function for processing symmetric responses
*
* @description
* This callback, which is registered with the common symmetric response
* message handler, is invoked to process symmetric response messages from
* the QAT. It will extract the response status from the cmnRespFlags set
* by the QAT, and then will pass it to @ref
* LacSymCb_ProcessCallbackInternal to complete the response processing.
*
* @param[in] lacCmdId ID of the symmetric QAT command of the request
* message
* @param[in] pOpaqueData pointer to opaque data in the request message
* @param[in] cmnRespFlags Flags set by QAT to indicate response status
*
* @return None
******************************************************************************/
static void
LacSymCb_ProcessCallback(icp_qat_fw_la_cmd_id_t lacCmdId,
void *pOpaqueData,
icp_qat_fw_comn_flags cmnRespFlags)
{
CpaCySymDpOpData *pDpOpData = (CpaCySymDpOpData *)pOpaqueData;
lac_session_desc_t *pSessionDesc =
LAC_SYM_SESSION_DESC_FROM_CTX_GET(pDpOpData->sessionCtx);
CpaBoolean qatRespStatusOkFlag =
(CpaBoolean)(ICP_QAT_FW_COMN_STATUS_FLAG_OK ==
ICP_QAT_FW_COMN_RESP_CRYPTO_STAT_GET(cmnRespFlags));
if (CPA_TRUE == pSessionDesc->isDPSession) {
/* DP session */
LacSymCb_ProcessDpCallback(pDpOpData,
qatRespStatusOkFlag,
pSessionDesc);
} else {
/* Trad session */
LacSymCb_ProcessCallbackInternal((lac_sym_bulk_cookie_t *)
pOpaqueData,
qatRespStatusOkFlag,
CPA_STATUS_SUCCESS,
pSessionDesc);
}
}
/*
*******************************************************************************
* Define public/global function definitions
*******************************************************************************
*/
/**
* @ingroup LacSymCb
*
* @return CpaStatus
* value returned will be the result of icp_adf_transPutMsg
*/
CpaStatus
LacSymCb_PendingReqsDequeue(lac_session_desc_t *pSessionDesc)
{
CpaStatus status = CPA_STATUS_SUCCESS;
sal_crypto_service_t *pService = NULL;
Cpa32U retries = 0;
pService = (sal_crypto_service_t *)pSessionDesc->pInstance;
/* Need to protect access to queue head and tail pointers, which may
* be accessed by multiple contexts simultaneously for enqueue and
* dequeue operations
*/
if (CPA_STATUS_SUCCESS !=
LAC_SPINLOCK(&pSessionDesc->requestQueueLock)) {
LAC_LOG_ERROR("Failed to lock request queue");
return CPA_STATUS_RESOURCE;
}
/* Clear the blocking flag in the session descriptor */
pSessionDesc->nonBlockingOpsInProgress = CPA_TRUE;
while ((NULL != pSessionDesc->pRequestQueueHead) &&
(CPA_TRUE == pSessionDesc->nonBlockingOpsInProgress)) {
/* If we send a partial packet request, set the
* blockingOpsInProgress
* flag for the session to indicate that subsequent requests
* must be
* queued up until this request completes
*/
if (CPA_CY_SYM_PACKET_TYPE_FULL !=
pSessionDesc->pRequestQueueHead->pOpData->packetType) {
pSessionDesc->nonBlockingOpsInProgress = CPA_FALSE;
}
/* At this point, we're clear to send the request. For cipher
* requests,
* we need to check if the session IV needs to be updated. This
* can
* only be done when no other partials are in flight for this
* session,
* to ensure the cipherPartialOpState buffer in the session
* descriptor
* is not currently in use
*/
if (CPA_TRUE ==
pSessionDesc->pRequestQueueHead->updateSessionIvOnSend) {
if (LAC_CIPHER_IS_ARC4(pSessionDesc->cipherAlgorithm)) {
memcpy(pSessionDesc->cipherPartialOpState,
pSessionDesc->cipherARC4InitialState,
LAC_CIPHER_ARC4_STATE_LEN_BYTES);
} else {
memcpy(pSessionDesc->cipherPartialOpState,
pSessionDesc->pRequestQueueHead->pOpData
->pIv,
pSessionDesc->pRequestQueueHead->pOpData
->ivLenInBytes);
}
}
/*
* Now we'll attempt to send the message directly to QAT. We'll
* keep
* looing until it succeeds (or at least a very high number of
* retries),
* as the failure only happens when the ring is full, and this
* is only
* a temporary situation. After a few retries, space will become
* availble, allowing the putMsg to succeed.
*/
retries = 0;
do {
/* Send to QAT */
status = icp_adf_transPutMsg(
pService->trans_handle_sym_tx,
(void *)&(pSessionDesc->pRequestQueueHead->qatMsg),
LAC_QAT_SYM_REQ_SZ_LW);
retries++;
/*
* Yield to allow other threads that may be on this
* session to poll
* and make some space on the ring
*/
if (CPA_STATUS_SUCCESS != status) {
qatUtilsYield();
}
} while ((CPA_STATUS_SUCCESS != status) &&
(retries < DEQUEUE_MSGPUT_MAX_RETRIES));
if ((CPA_STATUS_SUCCESS != status) ||
(retries >= DEQUEUE_MSGPUT_MAX_RETRIES)) {
LAC_LOG_ERROR(
"Failed to SalQatMsg_transPutMsg, maximum retries exceeded.");
goto cleanup;
}
pSessionDesc->pRequestQueueHead =
pSessionDesc->pRequestQueueHead->pNext;
}
/* If we've drained the queue, ensure the tail pointer is set to NULL */
if (NULL == pSessionDesc->pRequestQueueHead) {
pSessionDesc->pRequestQueueTail = NULL;
}
cleanup:
if (CPA_STATUS_SUCCESS !=
LAC_SPINUNLOCK(&pSessionDesc->requestQueueLock)) {
LAC_LOG_ERROR("Failed to unlock request queue");
}
return status;
}
/**
* @ingroup LacSymCb
*/
void
LacSymCb_CallbacksRegister()
{
/*** HASH ***/
LacSymQat_RespHandlerRegister(ICP_QAT_FW_LA_CMD_AUTH,
LacSymCb_ProcessCallback);
/*** ALGORITHM-CHAINING CIPHER_HASH***/
LacSymQat_RespHandlerRegister(ICP_QAT_FW_LA_CMD_CIPHER_HASH,
LacSymCb_ProcessCallback);
/*** ALGORITHM-CHAINING HASH_CIPHER***/
LacSymQat_RespHandlerRegister(ICP_QAT_FW_LA_CMD_HASH_CIPHER,
LacSymCb_ProcessCallback);
/*** CIPHER ***/
LacSymQat_RespHandlerRegister(ICP_QAT_FW_LA_CMD_CIPHER,
LacSymCb_ProcessCallback);
/* Call compile time param check function to ensure it is included
in the build by the compiler - this compile time check
ensures callbacks run as expected */
LacSym_CompileTimeAssertions();
}

416
sys/dev/qat/qat_api/common/crypto/sym/lac_sym_cipher.c Normal file
View File

@ -0,0 +1,416 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_cipher.c Cipher
*
* @ingroup LacCipher
*
* @description Functions specific to cipher
***************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
#include "icp_adf_init.h"
#include "icp_adf_transport.h"
#include "icp_accel_devices.h"
#include "icp_adf_debug.h"
#include "icp_qat_fw_la.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "lac_sym_cipher.h"
#include "lac_session.h"
#include "lac_mem.h"
#include "lac_common.h"
#include "lac_list.h"
#include "lac_sym.h"
#include "lac_sym_key.h"
#include "lac_sym_qat_hash_defs_lookup.h"
#include "lac_sal_types_crypto.h"
#include "lac_sal.h"
#include "lac_sal_ctrl.h"
#include "lac_sym_cipher_defs.h"
#include "lac_sym_cipher.h"
#include "lac_sym_stats.h"
#include "lac_sym.h"
#include "lac_sym_qat_cipher.h"
#include "lac_log.h"
#include "lac_buffer_desc.h"
/*
*******************************************************************************
* Static Variables
*******************************************************************************
*/
CpaStatus
LacCipher_PerformIvCheck(sal_service_t *pService,
lac_sym_bulk_cookie_t *pCbCookie,
Cpa32U qatPacketType,
Cpa8U **ppIvBuffer)
{
const CpaCySymOpData *pOpData = pCbCookie->pOpData;
lac_session_desc_t *pSessionDesc =
LAC_SYM_SESSION_DESC_FROM_CTX_GET(pOpData->sessionCtx);
CpaCySymCipherAlgorithm algorithm = pSessionDesc->cipherAlgorithm;
/* Perform IV check. */
if (LAC_CIPHER_IS_CTR_MODE(algorithm) ||
LAC_CIPHER_IS_CBC_MODE(algorithm) ||
LAC_CIPHER_IS_AES_F8(algorithm) ||
LAC_CIPHER_IS_XTS_MODE(algorithm)) {
unsigned ivLenInBytes =
LacSymQat_CipherIvSizeBytesGet(algorithm);
LAC_CHECK_NULL_PARAM(pOpData->pIv);
if (pOpData->ivLenInBytes != ivLenInBytes) {
if (!(/* GCM with 12 byte IV is OK */
(LAC_CIPHER_IS_GCM(algorithm) &&
pOpData->ivLenInBytes ==
LAC_CIPHER_IV_SIZE_GCM_12) ||
/* IV len for CCM has been checked before */
LAC_CIPHER_IS_CCM(algorithm))) {
LAC_INVALID_PARAM_LOG("invalid cipher IV size");
return CPA_STATUS_INVALID_PARAM;
}
}
/* Always copy the user's IV into another cipher state buffer if
* the request is part of a partial packet sequence
* (ensures that pipelined partial requests use same
* buffer)
*/
if (ICP_QAT_FW_LA_PARTIAL_NONE == qatPacketType) {
/* Set the value of the ppIvBuffer to that supplied
* by the user.
* NOTE: There is no guarantee that this address is
* aligned on
* an 8 or 64 Byte address. */
*ppIvBuffer = pOpData->pIv;
} else {
/* For partial packets, we use a per-session buffer to
* maintain
* the IV. This allows us to easily pass the updated IV
* forward
* to the next partial in the sequence. This makes
* internal
* buffering of partials easier to implement.
*/
*ppIvBuffer = pSessionDesc->cipherPartialOpState;
/* Ensure that the user's IV buffer gets updated between
* partial
* requests so that they may also see the residue from
* the
* previous partial. Not needed for final partials
* though.
*/
if ((ICP_QAT_FW_LA_PARTIAL_START == qatPacketType) ||
(ICP_QAT_FW_LA_PARTIAL_MID == qatPacketType)) {
pCbCookie->updateUserIvOnRecieve = CPA_TRUE;
if (ICP_QAT_FW_LA_PARTIAL_START ==
qatPacketType) {
/* if the previous partial state was
* full, then this is
* the first partial in the sequence so
* we need to copy
* in the user's IV. But, we have to be
* very careful
* here not to overwrite the
* cipherPartialOpState just
* yet in case there's a previous
* partial sequence in
* flight, so we defer the copy for now.
* This will be
* completed in the
* LacSymQueue_RequestSend() function.
*/
pCbCookie->updateSessionIvOnSend =
CPA_TRUE;
}
/* For subsequent partials in a sequence, we'll
* re-use the
* IV that was written back by the QAT, using
* internal
* request queueing if necessary to ensure that
* the next
* partial request isn't issued to the QAT until
* the
* previous one completes
*/
}
}
} else if (LAC_CIPHER_IS_KASUMI(algorithm)) {
LAC_CHECK_NULL_PARAM(pOpData->pIv);
if (LAC_CIPHER_IS_KASUMI(algorithm) &&
(pOpData->ivLenInBytes != LAC_CIPHER_KASUMI_F8_IV_LENGTH)) {
LAC_INVALID_PARAM_LOG("invalid cipher IV size");
return CPA_STATUS_INVALID_PARAM;
}
*ppIvBuffer = pOpData->pIv;
} else if (LAC_CIPHER_IS_SNOW3G_UEA2(algorithm)) {
LAC_CHECK_NULL_PARAM(pOpData->pIv);
if (LAC_CIPHER_IS_SNOW3G_UEA2(algorithm) &&
(pOpData->ivLenInBytes != ICP_QAT_HW_SNOW_3G_UEA2_IV_SZ)) {
LAC_INVALID_PARAM_LOG("invalid cipher IV size");
return CPA_STATUS_INVALID_PARAM;
}
*ppIvBuffer = pOpData->pIv;
} else if (LAC_CIPHER_IS_ARC4(algorithm)) {
if (ICP_QAT_FW_LA_PARTIAL_NONE == qatPacketType) {
/* For full packets, the initial ARC4 state is stored in
* the
* session descriptor. Use it directly.
*/
*ppIvBuffer = pSessionDesc->cipherARC4InitialState;
} else {
/* For partial packets, we maintain the running ARC4
* state in
* dedicated buffer in the session descriptor
*/
*ppIvBuffer = pSessionDesc->cipherPartialOpState;
if (ICP_QAT_FW_LA_PARTIAL_START == qatPacketType) {
/* if the previous partial state was full, then
* this is the
* first partial in the sequence so we need to
* (re-)initialise
* the contents of the state buffer using the
* initial state
* that is stored in the session descriptor.
* But, we have to be
* very careful here not to overwrite the
* cipherPartialOpState
* just yet in case there's a previous partial
* sequence in
* flight, so we defer the copy for now. This
* will be completed
* in the LacSymQueue_RequestSend() function
* when clear to send.
*/
pCbCookie->updateSessionIvOnSend = CPA_TRUE;
}
}
} else if (LAC_CIPHER_IS_ZUC_EEA3(algorithm)) {
LAC_CHECK_NULL_PARAM(pOpData->pIv);
if (pOpData->ivLenInBytes != ICP_QAT_HW_ZUC_3G_EEA3_IV_SZ) {
LAC_INVALID_PARAM_LOG("invalid cipher IV size");
return CPA_STATUS_INVALID_PARAM;
}
*ppIvBuffer = pOpData->pIv;
} else {
*ppIvBuffer = NULL;
}
return CPA_STATUS_SUCCESS;
}
CpaStatus
LacCipher_SessionSetupDataCheck(const CpaCySymCipherSetupData *pCipherSetupData)
{
/* No key required for NULL algorithm */
if (!LAC_CIPHER_IS_NULL(pCipherSetupData->cipherAlgorithm)) {
LAC_CHECK_NULL_PARAM(pCipherSetupData->pCipherKey);
/* Check that algorithm and keys passed in are correct size */
if (LAC_CIPHER_IS_ARC4(pCipherSetupData->cipherAlgorithm)) {
if (pCipherSetupData->cipherKeyLenInBytes >
ICP_QAT_HW_ARC4_KEY_SZ) {
LAC_INVALID_PARAM_LOG(
"Invalid ARC4 cipher key length");
return CPA_STATUS_INVALID_PARAM;
}
} else if (LAC_CIPHER_IS_CCM(
pCipherSetupData->cipherAlgorithm)) {
if (pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_AES_128_KEY_SZ) {
LAC_INVALID_PARAM_LOG(
"Invalid AES CCM cipher key length");
return CPA_STATUS_INVALID_PARAM;
}
} else if (LAC_CIPHER_IS_XTS_MODE(
pCipherSetupData->cipherAlgorithm)) {
if ((pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_AES_128_XTS_KEY_SZ) &&
(pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_AES_256_XTS_KEY_SZ)) {
LAC_INVALID_PARAM_LOG(
"Invalid AES XTS cipher key length");
return CPA_STATUS_INVALID_PARAM;
}
} else if (LAC_CIPHER_IS_AES(
pCipherSetupData->cipherAlgorithm)) {
if ((pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_AES_128_KEY_SZ) &&
(pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_AES_192_KEY_SZ) &&
(pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_AES_256_KEY_SZ)) {
LAC_INVALID_PARAM_LOG(
"Invalid AES cipher key length");
return CPA_STATUS_INVALID_PARAM;
}
} else if (LAC_CIPHER_IS_AES_F8(
pCipherSetupData->cipherAlgorithm)) {
if ((pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_AES_128_F8_KEY_SZ) &&
(pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_AES_192_F8_KEY_SZ) &&
(pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_AES_256_F8_KEY_SZ)) {
LAC_INVALID_PARAM_LOG(
"Invalid AES cipher key length");
return CPA_STATUS_INVALID_PARAM;
}
} else if (LAC_CIPHER_IS_DES(
pCipherSetupData->cipherAlgorithm)) {
if (pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_DES_KEY_SZ) {
LAC_INVALID_PARAM_LOG(
"Invalid DES cipher key length");
return CPA_STATUS_INVALID_PARAM;
}
} else if (LAC_CIPHER_IS_TRIPLE_DES(
pCipherSetupData->cipherAlgorithm)) {
if (pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_3DES_KEY_SZ) {
LAC_INVALID_PARAM_LOG(
"Invalid Triple-DES cipher key length");
return CPA_STATUS_INVALID_PARAM;
}
} else if (LAC_CIPHER_IS_KASUMI(
pCipherSetupData->cipherAlgorithm)) {
/* QAT-FW only supports 128 bits Cipher Key size for
* Kasumi F8
* Ref: 3GPP TS 55.216 V6.2.0 */
if (pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_KASUMI_KEY_SZ) {
LAC_INVALID_PARAM_LOG(
"Invalid Kasumi cipher key length");
return CPA_STATUS_INVALID_PARAM;
}
} else if (LAC_CIPHER_IS_SNOW3G_UEA2(
pCipherSetupData->cipherAlgorithm)) {
/* QAT-FW only supports 256 bits Cipher Key size for
* Snow_3G */
if (pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_SNOW_3G_UEA2_KEY_SZ) {
LAC_INVALID_PARAM_LOG(
"Invalid Snow_3G cipher key length");
return CPA_STATUS_INVALID_PARAM;
}
} else if (LAC_CIPHER_IS_ZUC_EEA3(
pCipherSetupData->cipherAlgorithm)) {
/* ZUC EEA3 */
if (pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_ZUC_3G_EEA3_KEY_SZ) {
LAC_INVALID_PARAM_LOG(
"Invalid ZUC cipher key length");
return CPA_STATUS_INVALID_PARAM;
}
} else if (LAC_CIPHER_IS_CHACHA(
pCipherSetupData->cipherAlgorithm)) {
if (pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_CHACHAPOLY_KEY_SZ) {
LAC_INVALID_PARAM_LOG(
"Invalid CHACHAPOLY cipher key length");
return CPA_STATUS_INVALID_PARAM;
}
} else if (LAC_CIPHER_IS_SM4(
pCipherSetupData->cipherAlgorithm)) {
if (pCipherSetupData->cipherKeyLenInBytes !=
ICP_QAT_HW_SM4_KEY_SZ) {
LAC_INVALID_PARAM_LOG(
"Invalid SM4 cipher key length");
return CPA_STATUS_INVALID_PARAM;
}
} else {
LAC_INVALID_PARAM_LOG("Invalid cipher algorithm");
return CPA_STATUS_INVALID_PARAM;
}
}
return CPA_STATUS_SUCCESS;
}
CpaStatus
LacCipher_PerformParamCheck(CpaCySymCipherAlgorithm algorithm,
const CpaCySymOpData *pOpData,
const Cpa64U packetLen)
{
CpaStatus status = CPA_STATUS_SUCCESS;
/* The following check will cover the dstBuffer as well, since
* the dstBuffer cannot be smaller than the srcBuffer (checked in
* LacSymPerform_BufferParamCheck() called from LacSym_Perform())
*/
if ((pOpData->messageLenToCipherInBytes +
pOpData->cryptoStartSrcOffsetInBytes) > packetLen) {
LAC_INVALID_PARAM_LOG("cipher len + offset greater than "
"srcBuffer packet len");
status = CPA_STATUS_INVALID_PARAM;
}
if (CPA_STATUS_SUCCESS == status) {
/*
* XTS Mode allow for ciphers which are not multiples of
* the block size.
*/
/* Perform algorithm-specific checks */
if (LAC_CIPHER_IS_XTS_MODE(algorithm) &&
((pOpData->packetType == CPA_CY_SYM_PACKET_TYPE_FULL) ||
(pOpData->packetType ==
CPA_CY_SYM_PACKET_TYPE_LAST_PARTIAL))) {
/*
* If this is the last of a partial request
*/
if (pOpData->messageLenToCipherInBytes <
ICP_QAT_HW_AES_BLK_SZ) {
LAC_INVALID_PARAM_LOG(
"data size must be greater than block "
"size for last XTS partial or XTS "
"full packet");
status = CPA_STATUS_INVALID_PARAM;
}
} else if (!(LAC_CIPHER_IS_ARC4(algorithm) ||
LAC_CIPHER_IS_CTR_MODE(algorithm) ||
LAC_CIPHER_IS_F8_MODE(algorithm) ||
LAC_CIPHER_IS_SNOW3G_UEA2(algorithm) ||
LAC_CIPHER_IS_XTS_MODE(algorithm) ||
LAC_CIPHER_IS_CHACHA(algorithm) ||
LAC_CIPHER_IS_ZUC_EEA3(algorithm))) {
/* Mask & check below is based on assumption that block
* size is
* a power of 2. If data size is not a multiple of the
* block size,
* the "remainder" bits selected by the mask be non-zero
*/
if (pOpData->messageLenToCipherInBytes &
(LacSymQat_CipherBlockSizeBytesGet(algorithm) -
1)) {
LAC_INVALID_PARAM_LOG(
"data size must be block size multiple");
status = CPA_STATUS_INVALID_PARAM;
}
}
}
return status;
}

45
sys/dev/qat/qat_api/common/crypto/sym/lac_sym_compile_check.c Normal file
View File

@ -0,0 +1,45 @@
/***************************************************************************
*
* <COPYRIGHT_TAG>
*
***************************************************************************/
/**
***************************************************************************
* @file lac_sym_compile_check.c
*
* @ingroup LacSym
*
* This file checks at compile time that some assumptions about the layout
* of key structures are as expected.
*
*
***************************************************************************/
#include "cpa.h"
#include "lac_common.h"
#include "icp_accel_devices.h"
#include "icp_adf_debug.h"
#include "lac_sym.h"
#include "cpa_cy_sym_dp.h"
#define COMPILE_TIME_ASSERT(pred) \
switch (0) { \
case 0: \
case pred:; \
}
void
LacSym_CompileTimeAssertions(void)
{
/* *************************************************************
* Check sessionCtx is at the same location in bulk cookie and
* CpaCySymDpOpData.
* This is required for the callbacks to work as expected -
* see LacSymCb_ProcessCallback
* ************************************************************* */
COMPILE_TIME_ASSERT(offsetof(lac_sym_bulk_cookie_t, sessionCtx) ==
offsetof(CpaCySymDpOpData, sessionCtx));
}

1080
sys/dev/qat/qat_api/common/crypto/sym/lac_sym_dp.c Normal file
View File

File diff suppressed because it is too large Load Diff

783
sys/dev/qat/qat_api/common/crypto/sym/lac_sym_hash.c Normal file
View File

@ -0,0 +1,783 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_hash.c
*
* @ingroup LacHash
*
* Hash specific functionality
***************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
#include "icp_accel_devices.h"
#include "icp_adf_debug.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "lac_common.h"
#include "lac_mem.h"
#include "lac_sym.h"
#include "lac_session.h"
#include "lac_sym_hash.h"
#include "lac_log.h"
#include "lac_sym_qat_hash.h"
#include "lac_sym_qat_hash_defs_lookup.h"
#include "lac_sym_cb.h"
#include "lac_sync.h"
#define LAC_HASH_ALG_MODE_NOT_SUPPORTED(alg, mode) \
((((CPA_CY_SYM_HASH_KASUMI_F9 == (alg)) || \
(CPA_CY_SYM_HASH_SNOW3G_UIA2 == (alg)) || \
(CPA_CY_SYM_HASH_AES_XCBC == (alg)) || \
(CPA_CY_SYM_HASH_AES_CCM == (alg)) || \
(CPA_CY_SYM_HASH_AES_GCM == (alg)) || \
(CPA_CY_SYM_HASH_AES_GMAC == (alg)) || \
(CPA_CY_SYM_HASH_AES_CMAC == (alg)) || \
(CPA_CY_SYM_HASH_ZUC_EIA3 == (alg))) && \
(CPA_CY_SYM_HASH_MODE_AUTH != (mode))) || \
(((CPA_CY_SYM_HASH_SHA3_224 == (alg)) || \
(CPA_CY_SYM_HASH_SHA3_256 == (alg)) || \
(CPA_CY_SYM_HASH_SHA3_384 == (alg)) || \
(CPA_CY_SYM_HASH_SHA3_512 == (alg))) && \
(CPA_CY_SYM_HASH_MODE_NESTED == (mode))) || \
(((CPA_CY_SYM_HASH_SHAKE_128 == (alg)) || \
(CPA_CY_SYM_HASH_SHAKE_256 == (alg))) && \
(CPA_CY_SYM_HASH_MODE_AUTH == (mode))))
/**< Macro to check for valid algorithm-mode combination */
/**
* @ingroup LacHash
* This callback function will be invoked whenever a synchronous
* hash precompute operation completes. It will set the wait
* queue flag for the synchronous operation.
*
* @param[in] pCallbackTag Opaque value provided by user. This will
* be a pointer to a wait queue flag.
*
* @retval
* None
*
*/
static void
LacHash_SyncPrecomputeDoneCb(void *pCallbackTag)
{
LacSync_GenWakeupSyncCaller(pCallbackTag, CPA_STATUS_SUCCESS);
}
/** @ingroup LacHash */
CpaStatus
LacHash_StatePrefixAadBufferInit(
sal_service_t *pService,
const CpaCySymHashSetupData *pHashSetupData,
icp_qat_la_bulk_req_ftr_t *pReq,
icp_qat_hw_auth_mode_t qatHashMode,
Cpa8U *pHashStateBuffer,
lac_sym_qat_hash_state_buffer_info_t *pHashStateBufferInfo)
{
/* set up the hash state prefix buffer info structure */
pHashStateBufferInfo->pData = pHashStateBuffer;
pHashStateBufferInfo->pDataPhys = LAC_MEM_CAST_PTR_TO_UINT64(
LAC_OS_VIRT_TO_PHYS_EXTERNAL((*pService), pHashStateBuffer));
if (pHashStateBufferInfo->pDataPhys == 0) {
LAC_LOG_ERROR("Unable to get the physical address of "
"the hash state buffer");
return CPA_STATUS_FAIL;
}
LacSymQat_HashStatePrefixAadBufferSizeGet(pReq, pHashStateBufferInfo);
/* Prefix data gets copied to the hash state buffer for nested mode */
if (CPA_CY_SYM_HASH_MODE_NESTED == pHashSetupData->hashMode) {
LacSymQat_HashStatePrefixAadBufferPopulate(
pHashStateBufferInfo,
pReq,
pHashSetupData->nestedModeSetupData.pInnerPrefixData,
pHashSetupData->nestedModeSetupData.innerPrefixLenInBytes,
pHashSetupData->nestedModeSetupData.pOuterPrefixData,
pHashSetupData->nestedModeSetupData.outerPrefixLenInBytes);
}
/* For mode2 HMAC the key gets copied into both the inner and
* outer prefix fields */
else if (IS_HASH_MODE_2_AUTH(qatHashMode, pHashSetupData->hashMode)) {
LacSymQat_HashStatePrefixAadBufferPopulate(
pHashStateBufferInfo,
pReq,
pHashSetupData->authModeSetupData.authKey,
pHashSetupData->authModeSetupData.authKeyLenInBytes,
pHashSetupData->authModeSetupData.authKey,
pHashSetupData->authModeSetupData.authKeyLenInBytes);
}
/* else do nothing for the other cases */
return CPA_STATUS_SUCCESS;
}
/** @ingroup LacHash */
CpaStatus
LacHash_PrecomputeDataCreate(const CpaInstanceHandle instanceHandle,
CpaCySymSessionSetupData *pSessionSetup,
lac_hash_precompute_done_cb_t callbackFn,
void *pCallbackTag,
Cpa8U *pWorkingBuffer,
Cpa8U *pState1,
Cpa8U *pState2)
{
CpaStatus status = CPA_STATUS_SUCCESS;
Cpa8U *pAuthKey = NULL;
Cpa32U authKeyLenInBytes = 0;
CpaCySymHashAlgorithm hashAlgorithm =
pSessionSetup->hashSetupData.hashAlgorithm;
CpaCySymHashAuthModeSetupData *pAuthModeSetupData =
&pSessionSetup->hashSetupData.authModeSetupData;
/* synchronous operation */
if (NULL == callbackFn) {
lac_sync_op_data_t *pSyncCallbackData = NULL;
status = LacSync_CreateSyncCookie(&pSyncCallbackData);
if (CPA_STATUS_SUCCESS == status) {
status = LacHash_PrecomputeDataCreate(
instanceHandle,
pSessionSetup,
LacHash_SyncPrecomputeDoneCb,
/* wait queue condition from sync cookie */
pSyncCallbackData,
pWorkingBuffer,
pState1,
pState2);
} else {
return status;
}
if (CPA_STATUS_SUCCESS == status) {
CpaStatus syncStatus = CPA_STATUS_SUCCESS;
syncStatus = LacSync_WaitForCallback(
pSyncCallbackData,
LAC_SYM_SYNC_CALLBACK_TIMEOUT,
&status,
NULL);
/* If callback doesn't come back */
if (CPA_STATUS_SUCCESS != syncStatus) {
QAT_UTILS_LOG(
"callback functions for precomputes did not return\n");
status = syncStatus;
}
} else {
/* As the Request was not sent the Callback will never
* be called, so need to indicate that we're finished
* with cookie so it can be destroyed. */
LacSync_SetSyncCookieComplete(pSyncCallbackData);
}
LacSync_DestroySyncCookie(&pSyncCallbackData);
return status;
}
/* set up convenience pointers */
pAuthKey = pAuthModeSetupData->authKey;
authKeyLenInBytes = pAuthModeSetupData->authKeyLenInBytes;
/* Pre-compute data state pointers must already be set up
* by LacSymQat_HashSetupBlockInit()
*/
/* state1 is not allocated for AES XCBC/CCM/GCM/Kasumi/UIA2
* so for these algorithms set state2 only */
if (CPA_CY_SYM_HASH_AES_XCBC == hashAlgorithm) {
status = LacSymHash_AesECBPreCompute(instanceHandle,
hashAlgorithm,
authKeyLenInBytes,
pAuthKey,
pWorkingBuffer,
pState2,
callbackFn,
pCallbackTag);
} else if (CPA_CY_SYM_HASH_AES_CMAC == hashAlgorithm) {
/* First, copy the original key to pState2 */
memcpy(pState2, pAuthKey, authKeyLenInBytes);
/* Then precompute */
status = LacSymHash_AesECBPreCompute(instanceHandle,
hashAlgorithm,
authKeyLenInBytes,
pAuthKey,
pWorkingBuffer,
pState2,
callbackFn,
pCallbackTag);
} else if (CPA_CY_SYM_HASH_AES_CCM == hashAlgorithm) {
/*
* The Inner Hash Initial State2 block must contain K
* (the cipher key) and 16 zeroes which will be replaced with
* EK(Ctr0) by the QAT-ME.
*/
/* write the auth key which for CCM is equivalent to cipher key
*/
memcpy(pState2,
pSessionSetup->cipherSetupData.pCipherKey,
pSessionSetup->cipherSetupData.cipherKeyLenInBytes);
/* initialize remaining buffer space to all zeroes */
LAC_OS_BZERO(
pState2 +
pSessionSetup->cipherSetupData.cipherKeyLenInBytes,
ICP_QAT_HW_AES_CCM_CBC_E_CTR0_SZ);
/* There is no request sent to the QAT for this operation,
* so just invoke the user's callback directly to signal
* completion of the precompute
*/
callbackFn(pCallbackTag);
} else if (CPA_CY_SYM_HASH_AES_GCM == hashAlgorithm ||
CPA_CY_SYM_HASH_AES_GMAC == hashAlgorithm) {
/*
* The Inner Hash Initial State2 block contains the following
* H (the Galois Hash Multiplier)
* len(A) (the length of A), (length before padding)
* 16 zeroes which will be replaced with EK(Ctr0) by the
* QAT.
*/
/* Memset state2 to 0 */
LAC_OS_BZERO(pState2,
ICP_QAT_HW_GALOIS_H_SZ +
ICP_QAT_HW_GALOIS_LEN_A_SZ +
ICP_QAT_HW_GALOIS_E_CTR0_SZ);
/* write H (the Galois Hash Multiplier) where H = E(K, 0...0)
* This will only write bytes 0-15 of pState2
*/
status = LacSymHash_AesECBPreCompute(
instanceHandle,
hashAlgorithm,
pSessionSetup->cipherSetupData.cipherKeyLenInBytes,
pSessionSetup->cipherSetupData.pCipherKey,
pWorkingBuffer,
pState2,
callbackFn,
pCallbackTag);
if (CPA_STATUS_SUCCESS == status) {
/* write len(A) (the length of A) into bytes 16-19 of
* pState2
* in big-endian format. This field is 8 bytes */
*(Cpa32U *)&pState2[ICP_QAT_HW_GALOIS_H_SZ] =
LAC_MEM_WR_32(pAuthModeSetupData->aadLenInBytes);
}
} else if (CPA_CY_SYM_HASH_KASUMI_F9 == hashAlgorithm) {
Cpa32U wordIndex = 0;
Cpa32U *pTempKey = (Cpa32U *)(pState2 + authKeyLenInBytes);
/*
* The Inner Hash Initial State2 block must contain IK
* (Initialisation Key), followed by IK XOR-ed with KM
* (Key Modifier): IK||(IK^KM).
*/
/* write the auth key */
memcpy(pState2, pAuthKey, authKeyLenInBytes);
/* initialise temp key with auth key */
memcpy(pTempKey, pAuthKey, authKeyLenInBytes);
/* XOR Key with KASUMI F9 key modifier at 4 bytes level */
for (wordIndex = 0;
wordIndex < LAC_BYTES_TO_LONGWORDS(authKeyLenInBytes);
wordIndex++) {
pTempKey[wordIndex] ^=
LAC_HASH_KASUMI_F9_KEY_MODIFIER_4_BYTES;
}
/* There is no request sent to the QAT for this operation,
* so just invoke the user's callback directly to signal
* completion of the precompute
*/
callbackFn(pCallbackTag);
} else if (CPA_CY_SYM_HASH_SNOW3G_UIA2 == hashAlgorithm) {
/*
* The Inner Hash Initial State2 should be all zeros
*/
LAC_OS_BZERO(pState2, ICP_QAT_HW_SNOW_3G_UIA2_STATE2_SZ);
/* There is no request sent to the QAT for this operation,
* so just invoke the user's callback directly to signal
* completion of the precompute
*/
callbackFn(pCallbackTag);
} else if (CPA_CY_SYM_HASH_ZUC_EIA3 == hashAlgorithm) {
/*
* The Inner Hash Initial State2 should contain the key
* and zero the rest of the state.
*/
LAC_OS_BZERO(pState2, ICP_QAT_HW_ZUC_3G_EIA3_STATE2_SZ);
memcpy(pState2, pAuthKey, authKeyLenInBytes);
/* There is no request sent to the QAT for this operation,
* so just invoke the user's callback directly to signal
* completion of the precompute
*/
callbackFn(pCallbackTag);
} else if (CPA_CY_SYM_HASH_POLY == hashAlgorithm) {
/* There is no request sent to the QAT for this operation,
* so just invoke the user's callback directly to signal
* completion of the precompute
*/
callbackFn(pCallbackTag);
} else /* For Hmac Precomputes */
{
status = LacSymHash_HmacPreComputes(instanceHandle,
hashAlgorithm,
authKeyLenInBytes,
pAuthKey,
pWorkingBuffer,
pState1,
pState2,
callbackFn,
pCallbackTag);
}
return status;
}
/** @ingroup LacHash */
CpaStatus
LacHash_HashContextCheck(CpaInstanceHandle instanceHandle,
const CpaCySymHashSetupData *pHashSetupData)
{
lac_sym_qat_hash_alg_info_t *pHashAlgInfo = NULL;
lac_sym_qat_hash_alg_info_t *pOuterHashAlgInfo = NULL;
CpaCySymCapabilitiesInfo capInfo;
/*Protect against value of hash outside the bitmap*/
if ((pHashSetupData->hashAlgorithm) >=
CPA_CY_SYM_HASH_CAP_BITMAP_SIZE) {
LAC_INVALID_PARAM_LOG("hashAlgorithm");
return CPA_STATUS_INVALID_PARAM;
}
cpaCySymQueryCapabilities(instanceHandle, &capInfo);
if (!CPA_BITMAP_BIT_TEST(capInfo.hashes,
pHashSetupData->hashAlgorithm) &&
pHashSetupData->hashAlgorithm != CPA_CY_SYM_HASH_AES_CBC_MAC) {
/* Ensure SHAKE algorithms are not supported */
if ((CPA_CY_SYM_HASH_SHAKE_128 ==
pHashSetupData->hashAlgorithm) ||
(CPA_CY_SYM_HASH_SHAKE_256 ==
pHashSetupData->hashAlgorithm)) {
LAC_INVALID_PARAM_LOG(
"Hash algorithms SHAKE-128 and SHAKE-256 "
"are not supported.");
return CPA_STATUS_UNSUPPORTED;
}
LAC_INVALID_PARAM_LOG("hashAlgorithm");
return CPA_STATUS_INVALID_PARAM;
}
switch (pHashSetupData->hashMode) {
case CPA_CY_SYM_HASH_MODE_PLAIN:
case CPA_CY_SYM_HASH_MODE_AUTH:
case CPA_CY_SYM_HASH_MODE_NESTED:
break;
default: {
LAC_INVALID_PARAM_LOG("hashMode");
return CPA_STATUS_INVALID_PARAM;
}
}
if (LAC_HASH_ALG_MODE_NOT_SUPPORTED(pHashSetupData->hashAlgorithm,
pHashSetupData->hashMode)) {
LAC_INVALID_PARAM_LOG("hashAlgorithm and hashMode combination");
return CPA_STATUS_INVALID_PARAM;
}
LacSymQat_HashAlgLookupGet(instanceHandle,
pHashSetupData->hashAlgorithm,
&pHashAlgInfo);
/* note: nested hash mode checks digest length against outer algorithm
*/
if ((CPA_CY_SYM_HASH_MODE_PLAIN == pHashSetupData->hashMode) ||
(CPA_CY_SYM_HASH_MODE_AUTH == pHashSetupData->hashMode)) {
/* Check Digest Length is permitted by the algorithm */
if ((0 == pHashSetupData->digestResultLenInBytes) ||
(pHashSetupData->digestResultLenInBytes >
pHashAlgInfo->digestLength)) {
LAC_INVALID_PARAM_LOG("digestResultLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
}
if (CPA_CY_SYM_HASH_MODE_AUTH == pHashSetupData->hashMode) {
if (CPA_CY_SYM_HASH_AES_GCM == pHashSetupData->hashAlgorithm ||
CPA_CY_SYM_HASH_AES_GMAC == pHashSetupData->hashAlgorithm) {
Cpa32U aadDataSize = 0;
/* RFC 4106: Implementations MUST support a full-length
* 16-octet
* ICV, and MAY support 8 or 12 octet ICVs, and MUST NOT
* support
* other ICV lengths. */
if ((pHashSetupData->digestResultLenInBytes !=
LAC_HASH_AES_GCM_ICV_SIZE_8) &&
(pHashSetupData->digestResultLenInBytes !=
LAC_HASH_AES_GCM_ICV_SIZE_12) &&
(pHashSetupData->digestResultLenInBytes !=
LAC_HASH_AES_GCM_ICV_SIZE_16)) {
LAC_INVALID_PARAM_LOG("digestResultLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
/* ensure aadLen is within maximum limit imposed by QAT
*/
aadDataSize =
pHashSetupData->authModeSetupData.aadLenInBytes;
/* round the aad size to the multiple of GCM hash block
* size. */
aadDataSize =
LAC_ALIGN_POW2_ROUNDUP(aadDataSize,
LAC_HASH_AES_GCM_BLOCK_SIZE);
if (aadDataSize > ICP_QAT_FW_CCM_GCM_AAD_SZ_MAX &&
CPA_CY_SYM_HASH_AES_GMAC !=
pHashSetupData->hashAlgorithm) {
LAC_INVALID_PARAM_LOG("aadLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
} else if (CPA_CY_SYM_HASH_AES_CCM ==
pHashSetupData->hashAlgorithm) {
Cpa32U aadDataSize = 0;
/* RFC 3610: Valid values are 4, 6, 8, 10, 12, 14, and
* 16 octets */
if ((pHashSetupData->digestResultLenInBytes >=
LAC_HASH_AES_CCM_ICV_SIZE_MIN) &&
(pHashSetupData->digestResultLenInBytes <=
LAC_HASH_AES_CCM_ICV_SIZE_MAX)) {
if ((pHashSetupData->digestResultLenInBytes &
0x01) != 0) {
LAC_INVALID_PARAM_LOG(
"digestResultLenInBytes must be a multiple of 2");
return CPA_STATUS_INVALID_PARAM;
}
} else {
LAC_INVALID_PARAM_LOG("digestResultLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
/* ensure aadLen is within maximum limit imposed by QAT
*/
/* at the beginning of the buffer there is B0 block */
aadDataSize = LAC_HASH_AES_CCM_BLOCK_SIZE;
/* then, if there is some 'a' data, the buffer will
* store encoded
* length of 'a' and 'a' itself */
if (pHashSetupData->authModeSetupData.aadLenInBytes >
0) {
/* as the QAT API puts the requirement on the
* pAdditionalAuthData not to be bigger than 240
* bytes then we
* just need 2 bytes to store encoded length of
* 'a' */
aadDataSize += sizeof(Cpa16U);
aadDataSize += pHashSetupData->authModeSetupData
.aadLenInBytes;
}
/* round the aad size to the multiple of CCM block
* size.*/
aadDataSize =
LAC_ALIGN_POW2_ROUNDUP(aadDataSize,
LAC_HASH_AES_CCM_BLOCK_SIZE);
if (aadDataSize > ICP_QAT_FW_CCM_GCM_AAD_SZ_MAX) {
LAC_INVALID_PARAM_LOG("aadLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
} else if (CPA_CY_SYM_HASH_KASUMI_F9 ==
pHashSetupData->hashAlgorithm) {
/* QAT-FW only supports 128 bit Integrity Key size for
* Kasumi f9
* Ref: 3GPP TS 35.201 version 7.0.0 Release 7 */
if (pHashSetupData->authModeSetupData
.authKeyLenInBytes !=
ICP_QAT_HW_KASUMI_KEY_SZ) {
LAC_INVALID_PARAM_LOG("authKeyLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
} else if (CPA_CY_SYM_HASH_SNOW3G_UIA2 ==
pHashSetupData->hashAlgorithm) {
/* QAT-FW only supports 128 bits Integrity Key size for
* Snow3g */
if (pHashSetupData->authModeSetupData
.authKeyLenInBytes !=
ICP_QAT_HW_SNOW_3G_UEA2_KEY_SZ) {
LAC_INVALID_PARAM_LOG("authKeyLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
/* For Snow3g hash aad field contains IV - it needs to
* be 16
* bytes long
*/
if (pHashSetupData->authModeSetupData.aadLenInBytes !=
ICP_QAT_HW_SNOW_3G_UEA2_IV_SZ) {
LAC_INVALID_PARAM_LOG("aadLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
} else if (CPA_CY_SYM_HASH_AES_XCBC ==
pHashSetupData->hashAlgorithm ||
CPA_CY_SYM_HASH_AES_CMAC ==
pHashSetupData->hashAlgorithm ||
CPA_CY_SYM_HASH_AES_CBC_MAC ==
pHashSetupData->hashAlgorithm) {
/* ensure auth key len is valid (128-bit keys supported)
*/
if ((pHashSetupData->authModeSetupData
.authKeyLenInBytes !=
ICP_QAT_HW_AES_128_KEY_SZ)) {
LAC_INVALID_PARAM_LOG("authKeyLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
} else if (CPA_CY_SYM_HASH_ZUC_EIA3 ==
pHashSetupData->hashAlgorithm) {
/* QAT-FW only supports 128 bits Integrity Key size for
* ZUC */
if (pHashSetupData->authModeSetupData
.authKeyLenInBytes !=
ICP_QAT_HW_ZUC_3G_EEA3_KEY_SZ) {
LAC_INVALID_PARAM_LOG("authKeyLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
/* For ZUC EIA3 hash aad field contains IV - it needs to
* be 16
* bytes long
*/
if (pHashSetupData->authModeSetupData.aadLenInBytes !=
ICP_QAT_HW_ZUC_3G_EEA3_IV_SZ) {
LAC_INVALID_PARAM_LOG("aadLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
} else if (CPA_CY_SYM_HASH_POLY ==
pHashSetupData->hashAlgorithm) {
if (pHashSetupData->digestResultLenInBytes !=
ICP_QAT_HW_SPC_CTR_SZ) {
LAC_INVALID_PARAM_LOG("Digest Length for CCP");
return CPA_STATUS_INVALID_PARAM;
}
if (pHashSetupData->authModeSetupData.aadLenInBytes >
ICP_QAT_FW_CCM_GCM_AAD_SZ_MAX) {
LAC_INVALID_PARAM_LOG("AAD Length for CCP");
return CPA_STATUS_INVALID_PARAM;
}
} else {
/* The key size must be less than or equal the block
* length */
if (pHashSetupData->authModeSetupData
.authKeyLenInBytes >
pHashAlgInfo->blockLength) {
LAC_INVALID_PARAM_LOG("authKeyLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
}
/* when the key size is greater than 0 check pointer is not null
*/
if (CPA_CY_SYM_HASH_AES_CCM != pHashSetupData->hashAlgorithm &&
CPA_CY_SYM_HASH_AES_GCM != pHashSetupData->hashAlgorithm &&
pHashSetupData->authModeSetupData.authKeyLenInBytes > 0) {
LAC_CHECK_NULL_PARAM(
pHashSetupData->authModeSetupData.authKey);
}
} else if (CPA_CY_SYM_HASH_MODE_NESTED == pHashSetupData->hashMode) {
if (!CPA_BITMAP_BIT_TEST(capInfo.hashes,
pHashSetupData->nestedModeSetupData
.outerHashAlgorithm)) {
/* Ensure SHAKE algorithms are not supported */
if ((CPA_CY_SYM_HASH_SHAKE_128 ==
pHashSetupData->nestedModeSetupData
.outerHashAlgorithm) ||
(CPA_CY_SYM_HASH_SHAKE_256 ==
pHashSetupData->nestedModeSetupData
.outerHashAlgorithm)) {
LAC_INVALID_PARAM_LOG(
"Hash algorithms SHAKE-128 and SHAKE-256 "
"are not supported.");
return CPA_STATUS_UNSUPPORTED;
}
LAC_INVALID_PARAM_LOG("outerHashAlgorithm");
return CPA_STATUS_INVALID_PARAM;
}
if (LAC_HASH_ALG_MODE_NOT_SUPPORTED(
pHashSetupData->nestedModeSetupData.outerHashAlgorithm,
pHashSetupData->hashMode)) {
LAC_INVALID_PARAM_LOG(
"outerHashAlgorithm and hashMode combination");
return CPA_STATUS_INVALID_PARAM;
}
LacSymQat_HashAlgLookupGet(
instanceHandle,
pHashSetupData->nestedModeSetupData.outerHashAlgorithm,
&pOuterHashAlgInfo);
/* Check Digest Length is permitted by the algorithm */
if ((0 == pHashSetupData->digestResultLenInBytes) ||
(pHashSetupData->digestResultLenInBytes >
pOuterHashAlgInfo->digestLength)) {
LAC_INVALID_PARAM_LOG("digestResultLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
if (pHashSetupData->nestedModeSetupData.innerPrefixLenInBytes >
LAC_MAX_INNER_OUTER_PREFIX_SIZE_BYTES) {
LAC_INVALID_PARAM_LOG("innerPrefixLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
if (pHashSetupData->nestedModeSetupData.innerPrefixLenInBytes >
0) {
LAC_CHECK_NULL_PARAM(pHashSetupData->nestedModeSetupData
.pInnerPrefixData);
}
if (pHashSetupData->nestedModeSetupData.outerPrefixLenInBytes >
LAC_MAX_INNER_OUTER_PREFIX_SIZE_BYTES) {
LAC_INVALID_PARAM_LOG("outerPrefixLenInBytes");
return CPA_STATUS_INVALID_PARAM;
}
if (pHashSetupData->nestedModeSetupData.outerPrefixLenInBytes >
0) {
LAC_CHECK_NULL_PARAM(pHashSetupData->nestedModeSetupData
.pOuterPrefixData);
}
}
return CPA_STATUS_SUCCESS;
}
/** @ingroup LacHash */
CpaStatus
LacHash_PerformParamCheck(CpaInstanceHandle instanceHandle,
lac_session_desc_t *pSessionDesc,
const CpaCySymOpData *pOpData,
Cpa64U srcPktSize,
const CpaBoolean *pVerifyResult)
{
CpaStatus status = CPA_STATUS_SUCCESS;
lac_sym_qat_hash_alg_info_t *pHashAlgInfo = NULL;
/* digestVerify and digestIsAppended on Hash-Only operation not
* supported */
if (pSessionDesc->digestIsAppended && pSessionDesc->digestVerify &&
(CPA_CY_SYM_OP_HASH == pSessionDesc->symOperation)) {
LAC_INVALID_PARAM_LOG(
"digestVerify and digestIsAppended set "
"on Hash-Only operation is not supported");
return CPA_STATUS_INVALID_PARAM;
}
/* check the digest result pointer */
if ((CPA_CY_SYM_PACKET_TYPE_PARTIAL != pOpData->packetType) &&
!pSessionDesc->digestIsAppended &&
(NULL == pOpData->pDigestResult)) {
LAC_INVALID_PARAM_LOG("pDigestResult is NULL");
return CPA_STATUS_INVALID_PARAM;
}
/*
* Check if the pVerifyResult pointer is not null for hash operation
* when
* the packet is the last one and user has set verifyDigest flag
* Also, this is only needed for symchronous operation, so check if the
* callback pointer is the internal synchronous one rather than a user-
* supplied one.
*/
if ((CPA_TRUE == pSessionDesc->digestVerify) &&
(CPA_CY_SYM_PACKET_TYPE_PARTIAL != pOpData->packetType) &&
(LacSync_GenBufListVerifyCb == pSessionDesc->pSymCb)) {
if (NULL == pVerifyResult) {
LAC_INVALID_PARAM_LOG(
"Null pointer pVerifyResult for hash op");
return CPA_STATUS_INVALID_PARAM;
}
}
/* verify start offset + messageLenToDigest is inside the source packet.
* this also verifies that the start offset is inside the packet
* Note: digest is specified as a pointer therefore it can be
* written anywhere so we cannot check for this been inside a buffer
* CCM/GCM specify the auth region using just the cipher params as this
* region is the same for auth and cipher. It is not checked here */
if ((CPA_CY_SYM_HASH_AES_CCM == pSessionDesc->hashAlgorithm) ||
(CPA_CY_SYM_HASH_AES_GCM == pSessionDesc->hashAlgorithm)) {
/* ensure AAD data pointer is non-NULL if AAD len > 0 */
if ((pSessionDesc->aadLenInBytes > 0) &&
(NULL == pOpData->pAdditionalAuthData)) {
LAC_INVALID_PARAM_LOG("pAdditionalAuthData is NULL");
return CPA_STATUS_INVALID_PARAM;
}
} else {
if ((pOpData->hashStartSrcOffsetInBytes +
pOpData->messageLenToHashInBytes) > srcPktSize) {
LAC_INVALID_PARAM_LOG(
"hashStartSrcOffsetInBytes + "
"messageLenToHashInBytes > Src Buffer Packet Length");
return CPA_STATUS_INVALID_PARAM;
}
}
/* For Snow3g & ZUC hash pAdditionalAuthData field
* of OpData should contain IV */
if ((CPA_CY_SYM_HASH_SNOW3G_UIA2 == pSessionDesc->hashAlgorithm) ||
(CPA_CY_SYM_HASH_ZUC_EIA3 == pSessionDesc->hashAlgorithm)) {
if (NULL == pOpData->pAdditionalAuthData) {
LAC_INVALID_PARAM_LOG("pAdditionalAuthData is NULL");
return CPA_STATUS_INVALID_PARAM;
}
}
/* partial packets need to be multiples of the algorithm block size in
* hash
* only mode (except for final partial packet) */
if ((CPA_CY_SYM_PACKET_TYPE_PARTIAL == pOpData->packetType) &&
(CPA_CY_SYM_OP_HASH == pSessionDesc->symOperation)) {
LacSymQat_HashAlgLookupGet(instanceHandle,
pSessionDesc->hashAlgorithm,
&pHashAlgInfo);
/* check if the message is a multiple of the block size. */
if ((pOpData->messageLenToHashInBytes %
pHashAlgInfo->blockLength) != 0) {
LAC_INVALID_PARAM_LOG(
"messageLenToHashInBytes not block size");
return CPA_STATUS_INVALID_PARAM;
}
}
return status;
}

353
sys/dev/qat/qat_api/common/crypto/sym/lac_sym_hash_sw_precomputes.c Normal file
View File

@ -0,0 +1,353 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_hash_sw_precomputes.c
*
* @ingroup LacHashDefs
*
* Hash Software
***************************************************************************/
/*
******************************************************************************
* Include public/global header files
******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
#include "icp_accel_devices.h"
#include "icp_adf_init.h"
#include "icp_adf_transport.h"
#include "icp_adf_debug.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "qat_utils.h"
#include "lac_mem.h"
#include "lac_sym.h"
#include "lac_log.h"
#include "lac_mem_pools.h"
#include "lac_list.h"
#include "lac_sym_hash_defs.h"
#include "lac_sym_qat_hash_defs_lookup.h"
#include "lac_sal_types_crypto.h"
#include "lac_sal.h"
#include "lac_session.h"
#include "lac_sym_hash_precomputes.h"
static CpaStatus
LacSymHash_Compute(CpaCySymHashAlgorithm hashAlgorithm,
lac_sym_qat_hash_alg_info_t *pHashAlgInfo,
Cpa8U *in,
Cpa8U *out)
{
/*
* Note: from SHA hashes appropriate endian swapping is required.
* For sha1, sha224 and sha256 double words based swapping.
* For sha384 and sha512 quad words swapping.
* No endianes swapping for md5 is required.
*/
CpaStatus status = CPA_STATUS_FAIL;
Cpa32U i = 0;
switch (hashAlgorithm) {
case CPA_CY_SYM_HASH_MD5:
if (CPA_STATUS_SUCCESS != qatUtilsHashMD5(in, out)) {
LAC_LOG_ERROR("qatUtilsHashMD5 Failed\n");
return status;
}
status = CPA_STATUS_SUCCESS;
break;
case CPA_CY_SYM_HASH_SHA1:
if (CPA_STATUS_SUCCESS != qatUtilsHashSHA1(in, out)) {
LAC_LOG_ERROR("qatUtilsHashSHA1 Failed\n");
return status;
}
for (i = 0; i < LAC_BYTES_TO_LONGWORDS(pHashAlgInfo->stateSize);
i++) {
((Cpa32U *)(out))[i] =
LAC_MEM_WR_32(((Cpa32U *)(out))[i]);
}
status = CPA_STATUS_SUCCESS;
break;
case CPA_CY_SYM_HASH_SHA224:
if (CPA_STATUS_SUCCESS != qatUtilsHashSHA224(in, out)) {
LAC_LOG_ERROR("qatUtilsHashSHA224 Failed\n");
return status;
}
for (i = 0; i < LAC_BYTES_TO_LONGWORDS(pHashAlgInfo->stateSize);
i++) {
((Cpa32U *)(out))[i] =
LAC_MEM_WR_32(((Cpa32U *)(out))[i]);
}
status = CPA_STATUS_SUCCESS;
break;
case CPA_CY_SYM_HASH_SHA256:
if (CPA_STATUS_SUCCESS != qatUtilsHashSHA256(in, out)) {
LAC_LOG_ERROR("qatUtilsHashSHA256 Failed\n");
return status;
}
for (i = 0; i < LAC_BYTES_TO_LONGWORDS(pHashAlgInfo->stateSize);
i++) {
((Cpa32U *)(out))[i] =
LAC_MEM_WR_32(((Cpa32U *)(out))[i]);
}
status = CPA_STATUS_SUCCESS;
break;
case CPA_CY_SYM_HASH_SHA384:
if (CPA_STATUS_SUCCESS != qatUtilsHashSHA384(in, out)) {
LAC_LOG_ERROR("qatUtilsHashSHA384 Failed\n");
return status;
}
for (i = 0; i < LAC_BYTES_TO_QUADWORDS(pHashAlgInfo->stateSize);
i++) {
((Cpa64U *)(out))[i] =
LAC_MEM_WR_64(((Cpa64U *)(out))[i]);
}
status = CPA_STATUS_SUCCESS;
break;
case CPA_CY_SYM_HASH_SHA512:
if (CPA_STATUS_SUCCESS != qatUtilsHashSHA512(in, out)) {
LAC_LOG_ERROR("qatUtilsHashSHA512 Failed\n");
return status;
}
for (i = 0; i < LAC_BYTES_TO_QUADWORDS(pHashAlgInfo->stateSize);
i++) {
((Cpa64U *)(out))[i] =
LAC_MEM_WR_64(((Cpa64U *)(out))[i]);
}
status = CPA_STATUS_SUCCESS;
break;
default:
return CPA_STATUS_INVALID_PARAM;
}
return status;
}
CpaStatus
LacSymHash_HmacPreComputes(CpaInstanceHandle instanceHandle,
CpaCySymHashAlgorithm hashAlgorithm,
Cpa32U authKeyLenInBytes,
Cpa8U *pAuthKey,
Cpa8U *pWorkingMemory,
Cpa8U *pState1,
Cpa8U *pState2,
lac_hash_precompute_done_cb_t callbackFn,
void *pCallbackTag)
{
Cpa8U *pIpadData = NULL;
Cpa8U *pOpadData = NULL;
CpaStatus status = CPA_STATUS_FAIL;
lac_sym_hash_precomp_op_data_t *pHmacIpadOpData =
(lac_sym_hash_precomp_op_data_t *)pWorkingMemory;
lac_sym_hash_precomp_op_data_t *pHmacOpadOpData = pHmacIpadOpData + 1;
/* Convenience pointers */
lac_sym_hash_hmac_precomp_qat_t *pHmacIpadQatData =
&pHmacIpadOpData->u.hmacQatData;
lac_sym_hash_hmac_precomp_qat_t *pHmacOpadQatData =
&pHmacOpadOpData->u.hmacQatData;
lac_sym_qat_hash_alg_info_t *pHashAlgInfo = NULL;
Cpa32U i = 0;
Cpa32U padLenBytes = 0;
LacSymQat_HashAlgLookupGet(instanceHandle,
hashAlgorithm,
&pHashAlgInfo);
pHmacIpadOpData->stateSize = pHashAlgInfo->stateSize;
pHmacOpadOpData->stateSize = pHashAlgInfo->stateSize;
/* Copy HMAC key into buffers */
if (authKeyLenInBytes > 0) {
memcpy(pHmacIpadQatData->data, pAuthKey, authKeyLenInBytes);
memcpy(pHmacOpadQatData->data, pAuthKey, authKeyLenInBytes);
}
padLenBytes = pHashAlgInfo->blockLength - authKeyLenInBytes;
/* Clear the remaining buffer space */
if (padLenBytes > 0) {
LAC_OS_BZERO(pHmacIpadQatData->data + authKeyLenInBytes,
padLenBytes);
LAC_OS_BZERO(pHmacOpadQatData->data + authKeyLenInBytes,
padLenBytes);
}
/* XOR Key with IPAD at 4-byte level */
for (i = 0; i < pHashAlgInfo->blockLength; i++) {
Cpa8U *ipad = pHmacIpadQatData->data + i;
Cpa8U *opad = pHmacOpadQatData->data + i;
*ipad ^= LAC_HASH_IPAD_BYTE;
*opad ^= LAC_HASH_OPAD_BYTE;
}
pIpadData = (Cpa8U *)pHmacIpadQatData->data;
pOpadData = (Cpa8U *)pHmacOpadQatData->data;
status = LacSymHash_Compute(hashAlgorithm,
pHashAlgInfo,
(Cpa8U *)pIpadData,
pState1);
if (CPA_STATUS_SUCCESS == status) {
status = LacSymHash_Compute(hashAlgorithm,
pHashAlgInfo,
(Cpa8U *)pOpadData,
pState2);
}
if (CPA_STATUS_SUCCESS == status) {
callbackFn(pCallbackTag);
}
return status;
}
CpaStatus
LacSymHash_AesECBPreCompute(CpaInstanceHandle instanceHandle,
CpaCySymHashAlgorithm hashAlgorithm,
Cpa32U authKeyLenInBytes,
Cpa8U *pAuthKey,
Cpa8U *pWorkingMemory,
Cpa8U *pState,
lac_hash_precompute_done_cb_t callbackFn,
void *pCallbackTag)
{
CpaStatus status = CPA_STATUS_FAIL;
Cpa32U stateSize = 0, x = 0;
lac_sym_qat_hash_alg_info_t *pHashAlgInfo = NULL;
if (CPA_CY_SYM_HASH_AES_XCBC == hashAlgorithm) {
Cpa8U *in = pWorkingMemory;
Cpa8U *out = pState;
LacSymQat_HashAlgLookupGet(instanceHandle,
hashAlgorithm,
&pHashAlgInfo);
stateSize = pHashAlgInfo->stateSize;
memcpy(pWorkingMemory, pHashAlgInfo->initState, stateSize);
for (x = 0; x < LAC_HASH_XCBC_PRECOMP_KEY_NUM; x++) {
if (CPA_STATUS_SUCCESS !=
qatUtilsAESEncrypt(
pAuthKey, authKeyLenInBytes, in, out)) {
return status;
}
in += LAC_HASH_XCBC_MAC_BLOCK_SIZE;
out += LAC_HASH_XCBC_MAC_BLOCK_SIZE;
}
status = CPA_STATUS_SUCCESS;
} else if (CPA_CY_SYM_HASH_AES_CMAC == hashAlgorithm) {
Cpa8U *out = pState;
Cpa8U k1[LAC_HASH_CMAC_BLOCK_SIZE],
k2[LAC_HASH_CMAC_BLOCK_SIZE];
Cpa8U *ptr = NULL, i = 0;
stateSize = LAC_HASH_CMAC_BLOCK_SIZE;
LacSymQat_HashAlgLookupGet(instanceHandle,
hashAlgorithm,
&pHashAlgInfo);
/* Original state size includes K, K1 and K2 which are of equal
* length.
* For precompute state size is only of the length of K which is
* equal
* to the block size for CPA_CY_SYM_HASH_AES_CMAC.
* The algorithm is described in rfc4493
* K is just copeid, K1 and K2 need to be single inplace encrypt
* with AES.
* */
memcpy(out, pHashAlgInfo->initState, stateSize);
memcpy(out, pAuthKey, authKeyLenInBytes);
out += LAC_HASH_CMAC_BLOCK_SIZE;
for (x = 0; x < LAC_HASH_XCBC_PRECOMP_KEY_NUM - 1; x++) {
if (CPA_STATUS_SUCCESS !=
qatUtilsAESEncrypt(
pAuthKey, authKeyLenInBytes, out, out)) {
return status;
}
out += LAC_HASH_CMAC_BLOCK_SIZE;
}
ptr = pState + LAC_HASH_CMAC_BLOCK_SIZE;
/* Derived keys (k1 and k2), copy them to
* pPrecompOpData->pState,
* but remember that at the beginning is original key (K0)
*/
/* Calculating K1 */
for (i = 0; i < LAC_HASH_CMAC_BLOCK_SIZE; i++, ptr++) {
k1[i] = (*ptr) << 1;
if (i != 0) {
k1[i - 1] |=
(*ptr) >> (LAC_NUM_BITS_IN_BYTE - 1);
}
if (i + 1 == LAC_HASH_CMAC_BLOCK_SIZE) {
/* If msb of pState + LAC_HASH_CMAC_BLOCK_SIZE
is set xor
with RB. Because only the final byte of RB is
non-zero
this is all we need to xor */
if ((*(pState + LAC_HASH_CMAC_BLOCK_SIZE)) &
LAC_SYM_HASH_MSBIT_MASK) {
k1[i] ^= LAC_SYM_AES_CMAC_RB_128;
}
}
}
/* Calculating K2 */
for (i = 0; i < LAC_HASH_CMAC_BLOCK_SIZE; i++) {
k2[i] = (k1[i]) << 1;
if (i != 0) {
k2[i - 1] |=
(k1[i]) >> (LAC_NUM_BITS_IN_BYTE - 1);
}
if (i + 1 == LAC_HASH_CMAC_BLOCK_SIZE) {
/* If msb of k1 is set xor last byte with RB */
if (k1[0] & LAC_SYM_HASH_MSBIT_MASK) {
k2[i] ^= LAC_SYM_AES_CMAC_RB_128;
}
}
}
/* Now, when we have K1 & K2 lets copy them to the state2 */
ptr = pState + LAC_HASH_CMAC_BLOCK_SIZE;
memcpy(ptr, k1, LAC_HASH_CMAC_BLOCK_SIZE);
ptr += LAC_HASH_CMAC_BLOCK_SIZE;
memcpy(ptr, k2, LAC_HASH_CMAC_BLOCK_SIZE);
status = CPA_STATUS_SUCCESS;
} else if (CPA_CY_SYM_HASH_AES_GCM == hashAlgorithm ||
CPA_CY_SYM_HASH_AES_GMAC == hashAlgorithm) {
Cpa8U *in = pWorkingMemory;
Cpa8U *out = pState;
LAC_OS_BZERO(pWorkingMemory, ICP_QAT_HW_GALOIS_H_SZ);
if (CPA_STATUS_SUCCESS !=
qatUtilsAESEncrypt(pAuthKey, authKeyLenInBytes, in, out)) {
return status;
}
status = CPA_STATUS_SUCCESS;
} else {
return CPA_STATUS_INVALID_PARAM;
}
callbackFn(pCallbackTag);
return status;
}
CpaStatus
LacSymHash_HmacPrecompInit(CpaInstanceHandle instanceHandle)
{
CpaStatus status = CPA_STATUS_SUCCESS;
return status;
}
void
LacSymHash_HmacPrecompShutdown(CpaInstanceHandle instanceHandle)
{
return;
}

75
sys/dev/qat/qat_api/common/crypto/sym/lac_sym_partial.c Normal file
View File

@ -0,0 +1,75 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_partial.c common partial packet functions
*
* @ingroup LacSym
*
***************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "icp_accel_devices.h"
#include "icp_adf_debug.h"
#include "lac_log.h"
#include "lac_sym.h"
#include "cpa_cy_sym.h"
#include "lac_common.h"
#include "lac_sym_partial.h"
CpaStatus
LacSym_PartialPacketStateCheck(CpaCySymPacketType packetType,
CpaCySymPacketType partialState)
{
CpaStatus status = CPA_STATUS_SUCCESS;
/* ASSUMPTION - partial requests on a given session must be issued
* sequentially to guarantee ordering
* (i.e. issuing partials on concurrent threads for a particular session
* just wouldn't work)
*/
/* state is no partial - only a partial is allowed */
if (((CPA_CY_SYM_PACKET_TYPE_FULL == partialState) &&
(CPA_CY_SYM_PACKET_TYPE_PARTIAL == packetType)) ||
/* state is partial - only a partial or final partial is allowed */
((CPA_CY_SYM_PACKET_TYPE_PARTIAL == partialState) &&
((CPA_CY_SYM_PACKET_TYPE_PARTIAL == packetType) ||
(CPA_CY_SYM_PACKET_TYPE_LAST_PARTIAL == packetType)))) {
status = CPA_STATUS_SUCCESS;
} else /* invalid sequence */
{
LAC_INVALID_PARAM_LOG("invalid partial packet sequence");
status = CPA_STATUS_INVALID_PARAM;
}
return status;
}
void
LacSym_PartialPacketStateUpdate(CpaCySymPacketType packetType,
CpaCySymPacketType *pPartialState)
{
/* if previous packet was either a full or ended a partial stream,
* update
* state to partial to indicate a new partial stream was created */
if (CPA_CY_SYM_PACKET_TYPE_FULL == *pPartialState) {
*pPartialState = CPA_CY_SYM_PACKET_TYPE_PARTIAL;
} else {
/* if packet type is final - reset the partial state */
if (CPA_CY_SYM_PACKET_TYPE_LAST_PARTIAL == packetType) {
*pPartialState = CPA_CY_SYM_PACKET_TYPE_FULL;
}
}
}

165
sys/dev/qat/qat_api/common/crypto/sym/lac_sym_queue.c Normal file
View File

@ -0,0 +1,165 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_queue.c Functions for sending/queuing symmetric requests
*
* @ingroup LacSym
*
***************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "icp_accel_devices.h"
#include "icp_adf_init.h"
#include "icp_adf_debug.h"
#include "icp_adf_transport.h"
#include "lac_sym_queue.h"
#include "lac_sym_qat.h"
#include "lac_session.h"
#include "lac_sym.h"
#include "lac_log.h"
#include "icp_qat_fw_la.h"
#include "lac_sal_types_crypto.h"
#define GetSingleBitFromByte(byte, bit) ((byte) & (1 << (bit)))
/*
*******************************************************************************
* Define public/global function definitions
*******************************************************************************
*/
CpaStatus
LacSymQueue_RequestSend(const CpaInstanceHandle instanceHandle,
lac_sym_bulk_cookie_t *pRequest,
lac_session_desc_t *pSessionDesc)
{
CpaStatus status = CPA_STATUS_SUCCESS;
CpaBoolean enqueued = CPA_FALSE;
sal_crypto_service_t *pService = (sal_crypto_service_t *)instanceHandle;
/* Enqueue the message instead of sending directly if:
* (i) a blocking operation is in progress
* (ii) there are previous requests already in the queue
*/
if ((CPA_FALSE == pSessionDesc->nonBlockingOpsInProgress) ||
(NULL != pSessionDesc->pRequestQueueTail)) {
if (CPA_STATUS_SUCCESS !=
LAC_SPINLOCK(&pSessionDesc->requestQueueLock)) {
LAC_LOG_ERROR("Failed to lock request queue");
return CPA_STATUS_RESOURCE;
}
/* Re-check blockingOpsInProgress and pRequestQueueTail in case
* either
* changed before the lock was acquired. The lock is shared
* with
* the callback context which drains this queue
*/
if ((CPA_FALSE == pSessionDesc->nonBlockingOpsInProgress) ||
(NULL != pSessionDesc->pRequestQueueTail)) {
/* Enqueue the message and exit */
/* The FIFO queue is made up of a head and tail pointer.
* The head pointer points to the first/oldest, entry
* in the queue, and the tail pointer points to the
* last/newest
* entry in the queue
*/
if (NULL != pSessionDesc->pRequestQueueTail) {
/* Queue is non-empty. Add this request to the
* list */
pSessionDesc->pRequestQueueTail->pNext =
pRequest;
} else {
/* Queue is empty. Initialise the head pointer
* as well */
pSessionDesc->pRequestQueueHead = pRequest;
}
pSessionDesc->pRequestQueueTail = pRequest;
/* request is queued, don't send to QAT here */
enqueued = CPA_TRUE;
}
if (CPA_STATUS_SUCCESS !=
LAC_SPINUNLOCK(&pSessionDesc->requestQueueLock)) {
LAC_LOG_ERROR("Failed to unlock request queue");
}
}
if (CPA_FALSE == enqueued) {
/* If we send a partial packet request, set the
* blockingOpsInProgress
* flag for the session to indicate that subsequent requests
* must be
* queued up until this request completes
*
* @assumption
* If we have got here it means that there were no previous
* blocking
* operations in progress and, since multiple partial packet
* requests
* on a given session cannot be issued concurrently, there
* should be
* no need for a critical section around the following code
*/
if (CPA_CY_SYM_PACKET_TYPE_FULL !=
pRequest->pOpData->packetType) {
/* Select blocking operations which this reqest will
* complete */
pSessionDesc->nonBlockingOpsInProgress = CPA_FALSE;
}
/* At this point, we're clear to send the request. For cipher
* requests,
* we need to check if the session IV needs to be updated. This
* can
* only be done when no other partials are in flight for this
* session,
* to ensure the cipherPartialOpState buffer in the session
* descriptor
* is not currently in use
*/
if (CPA_TRUE == pRequest->updateSessionIvOnSend) {
if (LAC_CIPHER_IS_ARC4(pSessionDesc->cipherAlgorithm)) {
memcpy(pSessionDesc->cipherPartialOpState,
pSessionDesc->cipherARC4InitialState,
LAC_CIPHER_ARC4_STATE_LEN_BYTES);
} else {
memcpy(pSessionDesc->cipherPartialOpState,
pRequest->pOpData->pIv,
pRequest->pOpData->ivLenInBytes);
}
}
/* Send to QAT */
status = icp_adf_transPutMsg(pService->trans_handle_sym_tx,
(void *)&(pRequest->qatMsg),
LAC_QAT_SYM_REQ_SZ_LW);
/* if fail to send request, we need to change
* nonBlockingOpsInProgress
* to CPA_TRUE
*/
if ((CPA_STATUS_SUCCESS != status) &&
(CPA_CY_SYM_PACKET_TYPE_FULL !=
pRequest->pOpData->packetType)) {
pSessionDesc->nonBlockingOpsInProgress = CPA_TRUE;
}
}
return status;
}

139
sys/dev/qat/qat_api/common/crypto/sym/lac_sym_stats.c Normal file
View File

@ -0,0 +1,139 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_stats.c Implementation of symmetric stats
*
* @ingroup LacSym
*
***************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "lac_mem_pools.h"
#include "icp_adf_transport.h"
#include "icp_accel_devices.h"
#include "icp_adf_debug.h"
#include "icp_qat_fw_la.h"
#include "lac_sym_qat.h"
#include "lac_sym_stats.h"
#include "lac_sal_types_crypto.h"
#include "sal_statistics.h"
/* Number of Symmetric Crypto statistics */
#define LAC_SYM_NUM_STATS (sizeof(CpaCySymStats64) / sizeof(Cpa64U))
CpaStatus
LacSym_StatsInit(CpaInstanceHandle instanceHandle)
{
CpaStatus status = CPA_STATUS_SUCCESS;
sal_crypto_service_t *pService = (sal_crypto_service_t *)instanceHandle;
pService->pLacSymStatsArr =
LAC_OS_MALLOC(LAC_SYM_NUM_STATS * sizeof(QatUtilsAtomic));
if (NULL != pService->pLacSymStatsArr) {
LAC_OS_BZERO((void *)LAC_CONST_VOLATILE_PTR_CAST(
pService->pLacSymStatsArr),
LAC_SYM_NUM_STATS * sizeof(QatUtilsAtomic));
} else {
status = CPA_STATUS_RESOURCE;
}
return status;
}
void
LacSym_StatsFree(CpaInstanceHandle instanceHandle)
{
sal_crypto_service_t *pService = (sal_crypto_service_t *)instanceHandle;
if (NULL != pService->pLacSymStatsArr) {
LAC_OS_FREE(pService->pLacSymStatsArr);
}
}
void
LacSym_StatsInc(Cpa32U offset, CpaInstanceHandle instanceHandle)
{
sal_crypto_service_t *pService = (sal_crypto_service_t *)instanceHandle;
if (CPA_TRUE ==
pService->generic_service_info.stats->bSymStatsEnabled) {
qatUtilsAtomicInc(
&pService->pLacSymStatsArr[offset / sizeof(Cpa64U)]);
}
}
void
LacSym_Stats32CopyGet(CpaInstanceHandle instanceHandle,
struct _CpaCySymStats *const pSymStats)
{
int i = 0;
sal_crypto_service_t *pService = (sal_crypto_service_t *)instanceHandle;
for (i = 0; i < LAC_SYM_NUM_STATS; i++) {
((Cpa32U *)pSymStats)[i] =
(Cpa32U)qatUtilsAtomicGet(&pService->pLacSymStatsArr[i]);
}
}
void
LacSym_Stats64CopyGet(CpaInstanceHandle instanceHandle,
CpaCySymStats64 *const pSymStats)
{
int i = 0;
sal_crypto_service_t *pService = (sal_crypto_service_t *)instanceHandle;
for (i = 0; i < LAC_SYM_NUM_STATS; i++) {
((Cpa64U *)pSymStats)[i] =
qatUtilsAtomicGet(&pService->pLacSymStatsArr[i]);
}
}
void
LacSym_StatsShow(CpaInstanceHandle instanceHandle)
{
CpaCySymStats64 symStats = { 0 };
LacSym_Stats64CopyGet(instanceHandle, &symStats);
QAT_UTILS_LOG(SEPARATOR BORDER
" Symmetric Stats " BORDER
"\n" SEPARATOR);
/* Session Info */
QAT_UTILS_LOG(BORDER " Sessions Initialized: %16llu " BORDER
"\n" BORDER
" Sessions Removed: %16llu " BORDER
"\n" BORDER
" Session Errors: %16llu " BORDER
"\n" SEPARATOR,
(unsigned long long)symStats.numSessionsInitialized,
(unsigned long long)symStats.numSessionsRemoved,
(unsigned long long)symStats.numSessionErrors);
/* Session info */
QAT_UTILS_LOG(
BORDER " Symmetric Requests: %16llu " BORDER "\n" BORDER
" Symmetric Request Errors: %16llu " BORDER "\n" BORDER
" Symmetric Completed: %16llu " BORDER "\n" BORDER
" Symmetric Completed Errors: %16llu " BORDER "\n" BORDER
" Symmetric Verify Failures: %16llu " BORDER
"\n" SEPARATOR,
(unsigned long long)symStats.numSymOpRequests,
(unsigned long long)symStats.numSymOpRequestErrors,
(unsigned long long)symStats.numSymOpCompleted,
(unsigned long long)symStats.numSymOpCompletedErrors,
(unsigned long long)symStats.numSymOpVerifyFailures);
}

227
sys/dev/qat/qat_api/common/crypto/sym/qat/lac_sym_qat.c Normal file
View File

@ -0,0 +1,227 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file lac_sym_qat.c Interfaces for populating the symmetric qat structures
*
* @ingroup LacSymQat
*
*****************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "icp_accel_devices.h"
#include "icp_adf_cfg.h"
#include "lac_log.h"
#include "lac_sym.h"
#include "lac_sym_qat.h"
#include "lac_sal_types_crypto.h"
#include "sal_string_parse.h"
#include "lac_sym_key.h"
#include "lac_sym_qat_hash_defs_lookup.h"
#include "lac_sym_qat_cipher.h"
#include "lac_sym_qat_hash.h"
#define EMBEDDED_CIPHER_KEY_MAX_SIZE 16
static void
LacSymQat_SymLogSliceHangError(icp_qat_fw_la_cmd_id_t symCmdId)
{
Cpa8U cmdId = symCmdId;
switch (cmdId) {
case ICP_QAT_FW_LA_CMD_CIPHER:
case ICP_QAT_FW_LA_CMD_CIPHER_PRE_COMP:
LAC_LOG_ERROR("slice hang detected on CPM cipher slice.");
break;
case ICP_QAT_FW_LA_CMD_AUTH:
case ICP_QAT_FW_LA_CMD_AUTH_PRE_COMP:
LAC_LOG_ERROR("slice hang detected on CPM auth slice.");
break;
case ICP_QAT_FW_LA_CMD_CIPHER_HASH:
case ICP_QAT_FW_LA_CMD_HASH_CIPHER:
case ICP_QAT_FW_LA_CMD_SSL3_KEY_DERIVE:
case ICP_QAT_FW_LA_CMD_TLS_V1_1_KEY_DERIVE:
case ICP_QAT_FW_LA_CMD_TLS_V1_2_KEY_DERIVE:
case ICP_QAT_FW_LA_CMD_MGF1:
default:
LAC_LOG_ERROR(
"slice hang detected on CPM cipher or auth slice.");
}
return;
}
/* sym crypto response handlers */
static sal_qat_resp_handler_func_t
respHandlerSymTbl[ICP_QAT_FW_LA_CMD_DELIMITER];
void
LacSymQat_SymRespHandler(void *pRespMsg)
{
Cpa8U lacCmdId = 0;
void *pOpaqueData = NULL;
icp_qat_fw_la_resp_t *pRespMsgFn = NULL;
Cpa8U opStatus = ICP_QAT_FW_COMN_STATUS_FLAG_OK;
Cpa8U comnErr = ERR_CODE_NO_ERROR;
pRespMsgFn = (icp_qat_fw_la_resp_t *)pRespMsg;
LAC_MEM_SHARED_READ_TO_PTR(pRespMsgFn->opaque_data, pOpaqueData);
lacCmdId = pRespMsgFn->comn_resp.cmd_id;
opStatus = pRespMsgFn->comn_resp.comn_status;
comnErr = pRespMsgFn->comn_resp.comn_error.s.comn_err_code;
/* log the slice hang and endpoint push/pull error inside the response
*/
if (ERR_CODE_SSM_ERROR == (Cpa8S)comnErr) {
LacSymQat_SymLogSliceHangError(lacCmdId);
} else if (ERR_CODE_ENDPOINT_ERROR == (Cpa8S)comnErr) {
LAC_LOG_ERROR("The PCIe End Point Push/Pull or"
" TI/RI Parity error detected.");
}
/* call the response message handler registered for the command ID */
respHandlerSymTbl[lacCmdId]((icp_qat_fw_la_cmd_id_t)lacCmdId,
pOpaqueData,
(icp_qat_fw_comn_flags)opStatus);
}
CpaStatus
LacSymQat_Init(CpaInstanceHandle instanceHandle)
{
CpaStatus status = CPA_STATUS_SUCCESS;
/* Initialise the Hash lookup table */
status = LacSymQat_HashLookupInit(instanceHandle);
return status;
}
void
LacSymQat_RespHandlerRegister(icp_qat_fw_la_cmd_id_t lacCmdId,
sal_qat_resp_handler_func_t pCbHandler)
{
if (lacCmdId >= ICP_QAT_FW_LA_CMD_DELIMITER) {
QAT_UTILS_LOG("Invalid Command ID\n");
return;
}
/* set the response handler for the command ID */
respHandlerSymTbl[lacCmdId] = pCbHandler;
}
void
LacSymQat_LaPacketCommandFlagSet(Cpa32U qatPacketType,
icp_qat_fw_la_cmd_id_t laCmdId,
CpaCySymCipherAlgorithm cipherAlgorithm,
Cpa16U *pLaCommandFlags,
Cpa32U ivLenInBytes)
{
/* For Chacha ciphers set command flag as partial none to proceed
* with stateless processing */
if (LAC_CIPHER_IS_CHACHA(cipherAlgorithm) ||
LAC_CIPHER_IS_SM4(cipherAlgorithm)) {
ICP_QAT_FW_LA_PARTIAL_SET(*pLaCommandFlags,
ICP_QAT_FW_LA_PARTIAL_NONE);
return;
}
ICP_QAT_FW_LA_PARTIAL_SET(*pLaCommandFlags, qatPacketType);
/* For ECB-mode ciphers, IV is NULL so update-state flag
* must be disabled always.
* For all other ciphers and auth
* update state is disabled for full packets and final partials */
if (((laCmdId != ICP_QAT_FW_LA_CMD_AUTH) &&
LAC_CIPHER_IS_ECB_MODE(cipherAlgorithm)) ||
(ICP_QAT_FW_LA_PARTIAL_NONE == qatPacketType) ||
(ICP_QAT_FW_LA_PARTIAL_END == qatPacketType)) {
ICP_QAT_FW_LA_UPDATE_STATE_SET(*pLaCommandFlags,
ICP_QAT_FW_LA_NO_UPDATE_STATE);
}
/* For first or middle partials set the update state command flag */
else {
ICP_QAT_FW_LA_UPDATE_STATE_SET(*pLaCommandFlags,
ICP_QAT_FW_LA_UPDATE_STATE);
if (laCmdId == ICP_QAT_FW_LA_CMD_AUTH) {
/* For hash only partial - verify and return auth result
* are
* disabled */
ICP_QAT_FW_LA_RET_AUTH_SET(
*pLaCommandFlags, ICP_QAT_FW_LA_NO_RET_AUTH_RES);
ICP_QAT_FW_LA_CMP_AUTH_SET(
*pLaCommandFlags, ICP_QAT_FW_LA_NO_CMP_AUTH_RES);
}
}
if ((LAC_CIPHER_IS_GCM(cipherAlgorithm)) &&
(LAC_CIPHER_IV_SIZE_GCM_12 == ivLenInBytes))
{
ICP_QAT_FW_LA_GCM_IV_LEN_FLAG_SET(
*pLaCommandFlags, ICP_QAT_FW_LA_GCM_IV_LEN_12_OCTETS);
}
}
void
LacSymQat_packetTypeGet(CpaCySymPacketType packetType,
CpaCySymPacketType packetState,
Cpa32U *pQatPacketType)
{
/* partial */
if (CPA_CY_SYM_PACKET_TYPE_PARTIAL == packetType) {
/* if the previous state was full, then this is the first packet
*/
if (CPA_CY_SYM_PACKET_TYPE_FULL == packetState) {
*pQatPacketType = ICP_QAT_FW_LA_PARTIAL_START;
} else {
*pQatPacketType = ICP_QAT_FW_LA_PARTIAL_MID;
}
}
/* final partial */
else if (CPA_CY_SYM_PACKET_TYPE_LAST_PARTIAL == packetType) {
*pQatPacketType = ICP_QAT_FW_LA_PARTIAL_END;
}
/* full packet - CPA_CY_SYM_PACKET_TYPE_FULL */
else {
*pQatPacketType = ICP_QAT_FW_LA_PARTIAL_NONE;
}
}
void
LacSymQat_LaSetDefaultFlags(icp_qat_fw_serv_specif_flags *laCmdFlags,
CpaCySymOp symOp)
{
ICP_QAT_FW_LA_PARTIAL_SET(*laCmdFlags, ICP_QAT_FW_LA_PARTIAL_NONE);
ICP_QAT_FW_LA_UPDATE_STATE_SET(*laCmdFlags,
ICP_QAT_FW_LA_NO_UPDATE_STATE);
if (symOp != CPA_CY_SYM_OP_CIPHER) {
ICP_QAT_FW_LA_RET_AUTH_SET(*laCmdFlags,
ICP_QAT_FW_LA_RET_AUTH_RES);
} else {
ICP_QAT_FW_LA_RET_AUTH_SET(*laCmdFlags,
ICP_QAT_FW_LA_NO_RET_AUTH_RES);
}
ICP_QAT_FW_LA_CMP_AUTH_SET(*laCmdFlags, ICP_QAT_FW_LA_NO_CMP_AUTH_RES);
ICP_QAT_FW_LA_GCM_IV_LEN_FLAG_SET(
*laCmdFlags, ICP_QAT_FW_LA_GCM_IV_LEN_NOT_12_OCTETS);
}

889
sys/dev/qat/qat_api/common/crypto/sym/qat/lac_sym_qat_cipher.c Normal file
View File

@ -0,0 +1,889 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_qat_cipher.c QAT-related support functions for Cipher
*
* @ingroup LacSymQat_Cipher
*
* @description Functions to support the QAT related operations for Cipher
***************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "icp_accel_devices.h"
#include "icp_adf_debug.h"
#include "lac_sym_qat.h"
#include "lac_sym_qat_cipher.h"
#include "lac_mem.h"
#include "lac_common.h"
#include "cpa_cy_sym.h"
#include "lac_sym_qat.h"
#include "lac_sym_cipher_defs.h"
#include "icp_qat_hw.h"
#include "icp_qat_fw_la.h"
/*****************************************************************************
* Internal data
*****************************************************************************/
typedef enum _icp_qat_hw_key_depend {
IS_KEY_DEP_NO = 0,
IS_KEY_DEP_YES,
} icp_qat_hw_key_depend;
/* LAC_CIPHER_IS_XTS_MODE */
static const uint8_t key_size_xts[] = {
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
ICP_QAT_HW_CIPHER_ALGO_AES128, // ICP_QAT_HW_AES_128_XTS_KEY_SZ
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
ICP_QAT_HW_CIPHER_ALGO_AES256 // ICP_QAT_HW_AES_256_XTS_KEY_SZ
};
/* LAC_CIPHER_IS_AES */
static const uint8_t key_size_aes[] = {
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
ICP_QAT_HW_CIPHER_ALGO_AES128, // ICP_QAT_HW_AES_128_KEY_SZ
0,
0,
0,
0,
0,
0,
0,
ICP_QAT_HW_CIPHER_ALGO_AES192, // ICP_QAT_HW_AES_192_KEY_SZ
0,
0,
0,
0,
0,
0,
0,
ICP_QAT_HW_CIPHER_ALGO_AES256 // ICP_QAT_HW_AES_256_KEY_SZ
};
/* LAC_CIPHER_IS_AES_F8 */
static const uint8_t key_size_f8[] = {
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
ICP_QAT_HW_CIPHER_ALGO_AES128, // ICP_QAT_HW_AES_128_F8_KEY_SZ
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
ICP_QAT_HW_CIPHER_ALGO_AES192, // ICP_QAT_HW_AES_192_F8_KEY_SZ
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
ICP_QAT_HW_CIPHER_ALGO_AES256 // ICP_QAT_HW_AES_256_F8_KEY_SZ
};
/* LAC_CIPHER_IS_SM4 */
static const uint8_t key_size_sm4[] = {
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
ICP_QAT_HW_CIPHER_ALGO_SM4 // ICP_QAT_HW_SM4_KEY_SZ
};
typedef struct _icp_qat_hw_cipher_info {
icp_qat_hw_cipher_algo_t algorithm;
icp_qat_hw_cipher_mode_t mode;
icp_qat_hw_cipher_convert_t key_convert[2];
icp_qat_hw_cipher_dir_t dir[2];
icp_qat_hw_key_depend isKeyLenDepend;
const uint8_t *pAlgByKeySize;
} icp_qat_hw_cipher_info;
static const icp_qat_hw_cipher_info icp_qat_alg_info[] =
{
/* CPA_CY_SYM_CIPHER_NULL */
{
ICP_QAT_HW_CIPHER_ALGO_NULL,
ICP_QAT_HW_CIPHER_ECB_MODE,
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_NO,
NULL,
},
/* CPA_CY_SYM_CIPHER_ARC4 */
{
ICP_QAT_HW_CIPHER_ALGO_ARC4,
ICP_QAT_HW_CIPHER_ECB_MODE,
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
/* Streaming ciphers are a special case. Decrypt = encrypt */
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_ENCRYPT },
IS_KEY_DEP_NO,
NULL,
},
/* CPA_CY_SYM_CIPHER_AES_ECB */
{
ICP_QAT_HW_CIPHER_ALGO_AES128,
ICP_QAT_HW_CIPHER_ECB_MODE,
/* AES decrypt key needs to be reversed. Instead of reversing the key
* at session registration, it is instead reversed on-the-fly by
* setting the KEY_CONVERT bit here
*/
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_KEY_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_YES,
key_size_aes,
},
/* CPA_CY_SYM_CIPHER_AES_CBC */
{
ICP_QAT_HW_CIPHER_ALGO_AES128,
ICP_QAT_HW_CIPHER_CBC_MODE,
/* AES decrypt key needs to be reversed. Instead of reversing the key
* at session registration, it is instead reversed on-the-fly by
* setting the KEY_CONVERT bit here
*/
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_KEY_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_YES,
key_size_aes,
},
/* CPA_CY_SYM_CIPHER_AES_CTR */
{
ICP_QAT_HW_CIPHER_ALGO_AES128,
ICP_QAT_HW_CIPHER_CTR_MODE,
/* AES decrypt key needs to be reversed. Instead of reversing the key
* at session registration, it is instead reversed on-the-fly by
* setting the KEY_CONVERT bit here
*/
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
/* Streaming ciphers are a special case. Decrypt = encrypt
* Overriding default values previously set for AES
*/
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_ENCRYPT },
IS_KEY_DEP_YES,
key_size_aes,
},
/* CPA_CY_SYM_CIPHER_AES_CCM */
{
ICP_QAT_HW_CIPHER_ALGO_AES128,
ICP_QAT_HW_CIPHER_CTR_MODE,
/* AES decrypt key needs to be reversed. Instead of reversing the key
* at session registration, it is instead reversed on-the-fly by
* setting the KEY_CONVERT bit here
*/
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
/* Streaming ciphers are a special case. Decrypt = encrypt
* Overriding default values previously set for AES
*/
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_ENCRYPT },
IS_KEY_DEP_YES,
key_size_aes,
},
/* CPA_CY_SYM_CIPHER_AES_GCM */
{
ICP_QAT_HW_CIPHER_ALGO_AES128,
ICP_QAT_HW_CIPHER_CTR_MODE,
/* AES decrypt key needs to be reversed. Instead of reversing the key
* at session registration, it is instead reversed on-the-fly by
* setting the KEY_CONVERT bit here
*/
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
/* Streaming ciphers are a special case. Decrypt = encrypt
* Overriding default values previously set for AES
*/
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_ENCRYPT },
IS_KEY_DEP_YES,
key_size_aes,
},
/* CPA_CY_SYM_CIPHER_DES_ECB */
{
ICP_QAT_HW_CIPHER_ALGO_DES,
ICP_QAT_HW_CIPHER_ECB_MODE,
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_NO,
NULL,
},
/* CPA_CY_SYM_CIPHER_DES_CBC */
{
ICP_QAT_HW_CIPHER_ALGO_DES,
ICP_QAT_HW_CIPHER_CBC_MODE,
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_NO,
NULL,
},
/* CPA_CY_SYM_CIPHER_3DES_ECB */
{
ICP_QAT_HW_CIPHER_ALGO_3DES,
ICP_QAT_HW_CIPHER_ECB_MODE,
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_NO,
NULL,
},
/* CPA_CY_SYM_CIPHER_3DES_CBC */
{
ICP_QAT_HW_CIPHER_ALGO_3DES,
ICP_QAT_HW_CIPHER_CBC_MODE,
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_NO,
NULL,
},
/* CPA_CY_SYM_CIPHER_3DES_CTR */
{
ICP_QAT_HW_CIPHER_ALGO_3DES,
ICP_QAT_HW_CIPHER_CTR_MODE,
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
/* Streaming ciphers are a special case. Decrypt = encrypt
* Overriding default values previously set for AES
*/
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_ENCRYPT },
IS_KEY_DEP_NO,
NULL,
},
/* CPA_CY_SYM_CIPHER_KASUMI_F8 */
{
ICP_QAT_HW_CIPHER_ALGO_KASUMI,
ICP_QAT_HW_CIPHER_F8_MODE,
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
/* Streaming ciphers are a special case. Decrypt = encrypt */
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_ENCRYPT },
IS_KEY_DEP_NO,
NULL,
},
/* CPA_CY_SYM_CIPHER_SNOW3G_UEA2 */
{
/* The KEY_CONVERT bit has to be set for Snow_3G operation */
ICP_QAT_HW_CIPHER_ALGO_SNOW_3G_UEA2,
ICP_QAT_HW_CIPHER_ECB_MODE,
{ ICP_QAT_HW_CIPHER_KEY_CONVERT, ICP_QAT_HW_CIPHER_KEY_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_NO,
NULL,
},
/* CPA_CY_SYM_CIPHER_AES_F8 */
{
ICP_QAT_HW_CIPHER_ALGO_AES128,
ICP_QAT_HW_CIPHER_F8_MODE,
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
/* Streaming ciphers are a special case. Decrypt = encrypt */
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_ENCRYPT },
IS_KEY_DEP_YES,
key_size_f8,
},
/* CPA_CY_SYM_CIPHER_AES_XTS */
{
ICP_QAT_HW_CIPHER_ALGO_AES128,
ICP_QAT_HW_CIPHER_XTS_MODE,
/* AES decrypt key needs to be reversed. Instead of reversing the key
* at session registration, it is instead reversed on-the-fly by
* setting the KEY_CONVERT bit here
*/
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_KEY_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_YES,
key_size_xts,
},
/* CPA_CY_SYM_CIPHER_ZUC_EEA3 */
{
ICP_QAT_HW_CIPHER_ALGO_ZUC_3G_128_EEA3,
ICP_QAT_HW_CIPHER_ECB_MODE,
{ ICP_QAT_HW_CIPHER_KEY_CONVERT, ICP_QAT_HW_CIPHER_KEY_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_NO,
NULL,
},
/* CPA_CY_SYM_CIPHER_CHACHA */
{
ICP_QAT_HW_CIPHER_ALGO_CHACHA20_POLY1305,
ICP_QAT_HW_CIPHER_CTR_MODE,
{ ICP_QAT_HW_CIPHER_KEY_CONVERT, ICP_QAT_HW_CIPHER_KEY_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_NO,
NULL,
},
/* CPA_CY_SYM_CIPHER_SM4_ECB */
{
ICP_QAT_HW_CIPHER_ALGO_SM4,
ICP_QAT_HW_CIPHER_ECB_MODE,
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_KEY_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_YES,
key_size_sm4,
},
/* CPA_CY_SYM_CIPHER_SM4_CBC */
{
ICP_QAT_HW_CIPHER_ALGO_SM4,
ICP_QAT_HW_CIPHER_CBC_MODE,
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_KEY_CONVERT },
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_DECRYPT },
IS_KEY_DEP_YES,
key_size_sm4,
},
/* CPA_CY_SYM_CIPHER_SM4_CTR */
{
ICP_QAT_HW_CIPHER_ALGO_SM4,
ICP_QAT_HW_CIPHER_CTR_MODE,
{ ICP_QAT_HW_CIPHER_NO_CONVERT, ICP_QAT_HW_CIPHER_NO_CONVERT },
/* Streaming ciphers are a special case. Decrypt = encrypt */
{ ICP_QAT_HW_CIPHER_ENCRYPT, ICP_QAT_HW_CIPHER_ENCRYPT },
IS_KEY_DEP_YES,
key_size_sm4,
},
};
/*****************************************************************************
* Internal functions
*****************************************************************************/
void
LacSymQat_CipherCtrlBlockWrite(icp_qat_la_bulk_req_ftr_t *pMsg,
Cpa32U cipherAlgorithm,
Cpa32U targetKeyLenInBytes,
icp_qat_fw_slice_t nextSlice,
Cpa8U cipherCfgOffsetInQuadWord)
{
icp_qat_fw_cipher_cd_ctrl_hdr_t *cd_ctrl =
(icp_qat_fw_cipher_cd_ctrl_hdr_t *)&(pMsg->cd_ctrl);
/* state_padding_sz is nonzero for f8 mode only */
cd_ctrl->cipher_padding_sz = 0;
/* Base Key is not passed down to QAT in the case of ARC4 or NULL */
if (LAC_CIPHER_IS_ARC4(cipherAlgorithm) ||
LAC_CIPHER_IS_NULL(cipherAlgorithm)) {
cd_ctrl->cipher_key_sz = 0;
} else if (LAC_CIPHER_IS_KASUMI(cipherAlgorithm)) {
cd_ctrl->cipher_key_sz =
LAC_BYTES_TO_QUADWORDS(ICP_QAT_HW_KASUMI_F8_KEY_SZ);
cd_ctrl->cipher_padding_sz =
ICP_QAT_HW_MODE_F8_NUM_REG_TO_CLEAR;
} else if (LAC_CIPHER_IS_SNOW3G_UEA2(cipherAlgorithm)) {
/* For Snow3G UEA2 content descriptor key size is
key size plus iv size */
cd_ctrl->cipher_key_sz =
LAC_BYTES_TO_QUADWORDS(ICP_QAT_HW_SNOW_3G_UEA2_KEY_SZ +
ICP_QAT_HW_SNOW_3G_UEA2_IV_SZ);
} else if (LAC_CIPHER_IS_AES_F8(cipherAlgorithm)) {
cd_ctrl->cipher_key_sz =
LAC_BYTES_TO_QUADWORDS(targetKeyLenInBytes);
cd_ctrl->cipher_padding_sz =
2 * ICP_QAT_HW_MODE_F8_NUM_REG_TO_CLEAR;
} else if (LAC_CIPHER_IS_ZUC_EEA3(cipherAlgorithm)) {
/* For ZUC EEA3 content descriptor key size is
key size plus iv size */
cd_ctrl->cipher_key_sz =
LAC_BYTES_TO_QUADWORDS(ICP_QAT_HW_ZUC_3G_EEA3_KEY_SZ +
ICP_QAT_HW_ZUC_3G_EEA3_IV_SZ);
} else {
cd_ctrl->cipher_key_sz =
LAC_BYTES_TO_QUADWORDS(targetKeyLenInBytes);
}
cd_ctrl->cipher_state_sz = LAC_BYTES_TO_QUADWORDS(
LacSymQat_CipherIvSizeBytesGet(cipherAlgorithm));
cd_ctrl->cipher_cfg_offset = cipherCfgOffsetInQuadWord;
ICP_QAT_FW_COMN_NEXT_ID_SET(cd_ctrl, nextSlice);
ICP_QAT_FW_COMN_CURR_ID_SET(cd_ctrl, ICP_QAT_FW_SLICE_CIPHER);
}
void
LacSymQat_CipherGetCfgData(lac_session_desc_t *pSession,
icp_qat_hw_cipher_algo_t *pAlgorithm,
icp_qat_hw_cipher_mode_t *pMode,
icp_qat_hw_cipher_dir_t *pDir,
icp_qat_hw_cipher_convert_t *pKey_convert)
{
CpaCySymCipherAlgorithm cipherAlgorithm = 0;
icp_qat_hw_cipher_dir_t cipherDirection = 0;
/* Set defaults */
*pKey_convert = ICP_QAT_HW_CIPHER_NO_CONVERT;
*pAlgorithm = ICP_QAT_HW_CIPHER_ALGO_NULL;
*pMode = ICP_QAT_HW_CIPHER_ECB_MODE;
*pDir = ICP_QAT_HW_CIPHER_ENCRYPT;
/* decrease since it's numbered from 1 instead of 0 */
cipherAlgorithm = pSession->cipherAlgorithm - 1;
cipherDirection =
pSession->cipherDirection == CPA_CY_SYM_CIPHER_DIRECTION_ENCRYPT ?
ICP_QAT_HW_CIPHER_ENCRYPT :
ICP_QAT_HW_CIPHER_DECRYPT;
*pAlgorithm = icp_qat_alg_info[cipherAlgorithm].algorithm;
*pMode = icp_qat_alg_info[cipherAlgorithm].mode;
*pDir = icp_qat_alg_info[cipherAlgorithm].dir[cipherDirection];
*pKey_convert =
icp_qat_alg_info[cipherAlgorithm].key_convert[cipherDirection];
if (IS_KEY_DEP_NO != icp_qat_alg_info[cipherAlgorithm].isKeyLenDepend) {
*pAlgorithm = icp_qat_alg_info[cipherAlgorithm]
.pAlgByKeySize[pSession->cipherKeyLenInBytes];
}
/* Set the mode */
if (LAC_CIPHER_IS_CTR_MODE(pSession->cipherAlgorithm)) {
*pMode = ICP_QAT_HW_CIPHER_CTR_MODE;
*pKey_convert = ICP_QAT_HW_CIPHER_NO_CONVERT;
/* CCP and AES_GCM single pass, despite being limited to
* CTR/AEAD mode,
* support both Encrypt/Decrypt modes - this is because of the
* differences in the hash computation/verification paths in
* encrypt/decrypt modes respectively.
* By default CCP is set as CTR Mode.Set AEAD Mode for AES_GCM.
*/
if (pSession->isSinglePass) {
if (LAC_CIPHER_IS_GCM(pSession->cipherAlgorithm))
*pMode = ICP_QAT_HW_CIPHER_AEAD_MODE;
if (cipherDirection == ICP_QAT_HW_CIPHER_DECRYPT)
*pDir = ICP_QAT_HW_CIPHER_DECRYPT;
}
}
}
void
LacSymQat_CipherHwBlockPopulateCfgData(lac_session_desc_t *pSession,
const void *pCipherHwBlock,
Cpa32U *pSizeInBytes)
{
icp_qat_hw_cipher_algo_t algorithm = ICP_QAT_HW_CIPHER_ALGO_NULL;
icp_qat_hw_cipher_mode_t mode = ICP_QAT_HW_CIPHER_ECB_MODE;
icp_qat_hw_cipher_dir_t dir = ICP_QAT_HW_CIPHER_ENCRYPT;
icp_qat_hw_cipher_convert_t key_convert;
icp_qat_hw_cipher_config_t *pCipherConfig =
(icp_qat_hw_cipher_config_t *)pCipherHwBlock;
Cpa32U aed_hash_cmp_length = 0;
*pSizeInBytes = 0;
LacSymQat_CipherGetCfgData(
pSession, &algorithm, &mode, &dir, &key_convert);
/* Build the cipher config into the hardware setup block */
if (pSession->isSinglePass) {
aed_hash_cmp_length = pSession->hashResultSize;
pCipherConfig->reserved = ICP_QAT_HW_CIPHER_CONFIG_BUILD_UPPER(
pSession->aadLenInBytes);
} else {
pCipherConfig->reserved = 0;
}
pCipherConfig->val = ICP_QAT_HW_CIPHER_CONFIG_BUILD(
mode, algorithm, key_convert, dir, aed_hash_cmp_length);
*pSizeInBytes = sizeof(icp_qat_hw_cipher_config_t);
}
void
LacSymQat_CipherHwBlockPopulateKeySetup(
const CpaCySymCipherSetupData *pCipherSetupData,
Cpa32U targetKeyLenInBytes,
const void *pCipherHwBlock,
Cpa32U *pSizeInBytes)
{
Cpa8U *pCipherKey = (Cpa8U *)pCipherHwBlock;
Cpa32U actualKeyLenInBytes = pCipherSetupData->cipherKeyLenInBytes;
*pSizeInBytes = 0;
/* Key is copied into content descriptor for all cases except for
* Arc4 and Null cipher */
if (!(LAC_CIPHER_IS_ARC4(pCipherSetupData->cipherAlgorithm) ||
LAC_CIPHER_IS_NULL(pCipherSetupData->cipherAlgorithm))) {
/* Set the Cipher key field in the cipher block */
memcpy(pCipherKey,
pCipherSetupData->pCipherKey,
actualKeyLenInBytes);
/* Pad the key with 0's if required */
if (0 < (targetKeyLenInBytes - actualKeyLenInBytes)) {
LAC_OS_BZERO(pCipherKey + actualKeyLenInBytes,
targetKeyLenInBytes - actualKeyLenInBytes);
}
*pSizeInBytes += targetKeyLenInBytes;
/* For Kasumi in F8 mode Cipher Key is concatenated with
* Cipher Key XOR-ed with Key Modifier (CK||CK^KM) */
if (LAC_CIPHER_IS_KASUMI(pCipherSetupData->cipherAlgorithm)) {
Cpa32U wordIndex = 0;
Cpa32U *pu32CipherKey =
(Cpa32U *)pCipherSetupData->pCipherKey;
Cpa32U *pTempKey =
(Cpa32U *)(pCipherKey + targetKeyLenInBytes);
/* XOR Key with KASUMI F8 key modifier at 4 bytes level
*/
for (wordIndex = 0; wordIndex <
LAC_BYTES_TO_LONGWORDS(targetKeyLenInBytes);
wordIndex++) {
pTempKey[wordIndex] = pu32CipherKey[wordIndex] ^
LAC_CIPHER_KASUMI_F8_KEY_MODIFIER_4_BYTES;
}
*pSizeInBytes += targetKeyLenInBytes;
/* also add padding for F8 */
*pSizeInBytes += LAC_QUADWORDS_TO_BYTES(
ICP_QAT_HW_MODE_F8_NUM_REG_TO_CLEAR);
LAC_OS_BZERO((Cpa8U *)pTempKey + targetKeyLenInBytes,
LAC_QUADWORDS_TO_BYTES(
ICP_QAT_HW_MODE_F8_NUM_REG_TO_CLEAR));
}
/* For AES in F8 mode Cipher Key is concatenated with
* Cipher Key XOR-ed with Key Mask (CK||CK^KM) */
else if (LAC_CIPHER_IS_AES_F8(
pCipherSetupData->cipherAlgorithm)) {
Cpa32U index = 0;
Cpa8U *pTempKey =
pCipherKey + (targetKeyLenInBytes / 2);
*pSizeInBytes += targetKeyLenInBytes;
/* XOR Key with key Mask */
for (index = 0; index < targetKeyLenInBytes; index++) {
pTempKey[index] =
pCipherKey[index] ^ pTempKey[index];
}
pTempKey = (pCipherKey + targetKeyLenInBytes);
/* also add padding for AES F8 */
*pSizeInBytes += 2 * targetKeyLenInBytes;
LAC_OS_BZERO(pTempKey, 2 * targetKeyLenInBytes);
} else if (LAC_CIPHER_IS_SNOW3G_UEA2(
pCipherSetupData->cipherAlgorithm)) {
/* For Snow3G zero area after the key for FW */
LAC_OS_BZERO(pCipherKey + targetKeyLenInBytes,
ICP_QAT_HW_SNOW_3G_UEA2_IV_SZ);
*pSizeInBytes += ICP_QAT_HW_SNOW_3G_UEA2_IV_SZ;
} else if (LAC_CIPHER_IS_ZUC_EEA3(
pCipherSetupData->cipherAlgorithm)) {
/* For ZUC zero area after the key for FW */
LAC_OS_BZERO(pCipherKey + targetKeyLenInBytes,
ICP_QAT_HW_ZUC_3G_EEA3_IV_SZ);
*pSizeInBytes += ICP_QAT_HW_ZUC_3G_EEA3_IV_SZ;
}
}
}
/*****************************************************************************
* External functions
*****************************************************************************/
Cpa8U
LacSymQat_CipherBlockSizeBytesGet(CpaCySymCipherAlgorithm cipherAlgorithm)
{
if (LAC_CIPHER_IS_ARC4(cipherAlgorithm)) {
return LAC_CIPHER_ARC4_BLOCK_LEN_BYTES;
} else if (LAC_CIPHER_IS_AES(cipherAlgorithm) ||
LAC_CIPHER_IS_AES_F8(cipherAlgorithm)) {
return ICP_QAT_HW_AES_BLK_SZ;
} else if (LAC_CIPHER_IS_DES(cipherAlgorithm)) {
return ICP_QAT_HW_DES_BLK_SZ;
} else if (LAC_CIPHER_IS_TRIPLE_DES(cipherAlgorithm)) {
return ICP_QAT_HW_3DES_BLK_SZ;
} else if (LAC_CIPHER_IS_KASUMI(cipherAlgorithm)) {
return ICP_QAT_HW_KASUMI_BLK_SZ;
} else if (LAC_CIPHER_IS_SNOW3G_UEA2(cipherAlgorithm)) {
return ICP_QAT_HW_SNOW_3G_BLK_SZ;
} else if (LAC_CIPHER_IS_ZUC_EEA3(cipherAlgorithm)) {
return ICP_QAT_HW_ZUC_3G_BLK_SZ;
} else if (LAC_CIPHER_IS_NULL(cipherAlgorithm)) {
return LAC_CIPHER_NULL_BLOCK_LEN_BYTES;
} else if (LAC_CIPHER_IS_CHACHA(cipherAlgorithm)) {
return ICP_QAT_HW_CHACHAPOLY_BLK_SZ;
} else if (LAC_CIPHER_IS_SM4(cipherAlgorithm)) {
return ICP_QAT_HW_SM4_BLK_SZ;
} else {
QAT_UTILS_LOG("Algorithm not supported in Cipher\n");
return 0;
}
}
Cpa32U
LacSymQat_CipherIvSizeBytesGet(CpaCySymCipherAlgorithm cipherAlgorithm)
{
if (CPA_CY_SYM_CIPHER_ARC4 == cipherAlgorithm) {
return LAC_CIPHER_ARC4_STATE_LEN_BYTES;
} else if (LAC_CIPHER_IS_KASUMI(cipherAlgorithm)) {
return ICP_QAT_HW_KASUMI_BLK_SZ;
} else if (LAC_CIPHER_IS_SNOW3G_UEA2(cipherAlgorithm)) {
return ICP_QAT_HW_SNOW_3G_UEA2_IV_SZ;
} else if (LAC_CIPHER_IS_ZUC_EEA3(cipherAlgorithm)) {
return ICP_QAT_HW_ZUC_3G_EEA3_IV_SZ;
} else if (LAC_CIPHER_IS_CHACHA(cipherAlgorithm)) {
return ICP_QAT_HW_CHACHAPOLY_IV_SZ;
} else if (LAC_CIPHER_IS_ECB_MODE(cipherAlgorithm)) {
return 0;
} else {
return (Cpa32U)LacSymQat_CipherBlockSizeBytesGet(
cipherAlgorithm);
}
}
inline CpaStatus
LacSymQat_CipherRequestParamsPopulate(icp_qat_fw_la_bulk_req_t *pReq,
Cpa32U cipherOffsetInBytes,
Cpa32U cipherLenInBytes,
Cpa64U ivBufferPhysAddr,
Cpa8U *pIvBufferVirt)
{
icp_qat_fw_la_cipher_req_params_t *pCipherReqParams;
icp_qat_fw_cipher_cd_ctrl_hdr_t *pCipherCdCtrlHdr;
icp_qat_fw_serv_specif_flags *pCipherSpecificFlags;
pCipherReqParams = (icp_qat_fw_la_cipher_req_params_t
*)((Cpa8U *)&(pReq->serv_specif_rqpars) +
ICP_QAT_FW_CIPHER_REQUEST_PARAMETERS_OFFSET);
pCipherCdCtrlHdr = (icp_qat_fw_cipher_cd_ctrl_hdr_t *)&(pReq->cd_ctrl);
pCipherSpecificFlags = &(pReq->comn_hdr.serv_specif_flags);
pCipherReqParams->cipher_offset = cipherOffsetInBytes;
pCipherReqParams->cipher_length = cipherLenInBytes;
/* Don't copy the buffer into the Msg if
* it's too big for the cipher_IV_array
* OR if the FW needs to update it
* OR if there's no buffer supplied
* OR if last partial
*/
if ((pCipherCdCtrlHdr->cipher_state_sz >
LAC_SYM_QAT_HASH_IV_REQ_MAX_SIZE_QW) ||
(ICP_QAT_FW_LA_UPDATE_STATE_GET(*pCipherSpecificFlags) ==
ICP_QAT_FW_LA_UPDATE_STATE) ||
(pIvBufferVirt == NULL) ||
(ICP_QAT_FW_LA_PARTIAL_GET(*pCipherSpecificFlags) ==
ICP_QAT_FW_LA_PARTIAL_END)) {
/* Populate the field with a ptr to the flat buffer */
pCipherReqParams->u.s.cipher_IV_ptr = ivBufferPhysAddr;
pCipherReqParams->u.s.resrvd1 = 0;
/* Set the flag indicating the field format */
ICP_QAT_FW_LA_CIPH_IV_FLD_FLAG_SET(
*pCipherSpecificFlags, ICP_QAT_FW_CIPH_IV_64BIT_PTR);
} else {
/* Populate the field with the contents of the buffer,
* zero field first as data may be smaller than the field */
memset(pCipherReqParams->u.cipher_IV_array,
0,
LAC_LONGWORDS_TO_BYTES(ICP_QAT_FW_NUM_LONGWORDS_4));
/* We force a specific compiler optimisation here. The length
* to
* be copied turns out to be always 16, and by coding a memcpy
* with
* a literal value the compiler will compile inline code (in
* fact,
* only two vector instructions) to effect the copy. This gives
* us
* a huge performance increase.
*/
unsigned long cplen =
LAC_QUADWORDS_TO_BYTES(pCipherCdCtrlHdr->cipher_state_sz);
if (cplen == 16)
memcpy(pCipherReqParams->u.cipher_IV_array,
pIvBufferVirt,
16);
else
memcpy(pCipherReqParams->u.cipher_IV_array,
pIvBufferVirt,
cplen);
/* Set the flag indicating the field format */
ICP_QAT_FW_LA_CIPH_IV_FLD_FLAG_SET(
*pCipherSpecificFlags, ICP_QAT_FW_CIPH_IV_16BYTE_DATA);
}
return CPA_STATUS_SUCCESS;
}
void
LacSymQat_CipherArc4StateInit(const Cpa8U *pKey,
Cpa32U keyLenInBytes,
Cpa8U *pArc4CipherState)
{
Cpa32U i = 0;
Cpa32U j = 0;
Cpa32U k = 0;
for (i = 0; i < LAC_CIPHER_ARC4_KEY_MATRIX_LEN_BYTES; ++i) {
pArc4CipherState[i] = (Cpa8U)i;
}
for (i = 0; i < LAC_CIPHER_ARC4_KEY_MATRIX_LEN_BYTES; ++i) {
Cpa8U swap = 0;
if (k >= keyLenInBytes)
k -= keyLenInBytes;
j = (j + pArc4CipherState[i] + pKey[k]);
if (j >= LAC_CIPHER_ARC4_KEY_MATRIX_LEN_BYTES)
j %= LAC_CIPHER_ARC4_KEY_MATRIX_LEN_BYTES;
++k;
/* Swap state[i] & state[j] */
swap = pArc4CipherState[i];
pArc4CipherState[i] = pArc4CipherState[j];
pArc4CipherState[j] = swap;
}
/* Initialise i & j values for QAT */
pArc4CipherState[LAC_CIPHER_ARC4_KEY_MATRIX_LEN_BYTES] = 0;
pArc4CipherState[LAC_CIPHER_ARC4_KEY_MATRIX_LEN_BYTES + 1] = 0;
}
/* Update the cipher_key_sz in the Request cache prepared and stored
* in the session */
void
LacSymQat_CipherXTSModeUpdateKeyLen(lac_session_desc_t *pSessionDesc,
Cpa32U newKeySizeInBytes)
{
icp_qat_fw_cipher_cd_ctrl_hdr_t *pCipherControlBlock = NULL;
pCipherControlBlock = (icp_qat_fw_cipher_cd_ctrl_hdr_t *)&(
pSessionDesc->reqCacheFtr.cd_ctrl);
pCipherControlBlock->cipher_key_sz =
LAC_BYTES_TO_QUADWORDS(newKeySizeInBytes);
}

942
sys/dev/qat/qat_api/common/crypto/sym/qat/lac_sym_qat_hash.c Normal file
View File

@ -0,0 +1,942 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_qat_hash.c
*
* @ingroup LacSymQatHash
*
* Implementation for populating QAT data structures for hash operation
***************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
#include "cpa_cy_sym.h"
#include "icp_accel_devices.h"
#include "icp_adf_debug.h"
#include "lac_log.h"
#include "lac_mem.h"
#include "lac_sym.h"
#include "lac_common.h"
#include "lac_sym_qat.h"
#include "lac_list.h"
#include "lac_sal_types.h"
#include "lac_sym_qat_hash.h"
#include "lac_sym_qat_hash_defs_lookup.h"
/**
* This structure contains pointers into the hash setup block of the
* security descriptor. As the hash setup block contains fields that
* are of variable length, pointers must be calculated to these fields
* and the hash setup block is populated using these pointers. */
typedef struct lac_hash_blk_ptrs_s {
icp_qat_hw_auth_setup_t *pInHashSetup;
/**< inner hash setup */
Cpa8U *pInHashInitState1;
/**< inner initial state 1 */
Cpa8U *pInHashInitState2;
/**< inner initial state 2 */
icp_qat_hw_auth_setup_t *pOutHashSetup;
/**< outer hash setup */
Cpa8U *pOutHashInitState1;
/**< outer hash initial state */
} lac_hash_blk_ptrs_t;
typedef struct lac_hash_blk_ptrs_optimised_s {
Cpa8U *pInHashInitState1;
/**< inner initial state 1 */
Cpa8U *pInHashInitState2;
/**< inner initial state 2 */
} lac_hash_blk_ptrs_optimised_t;
/**
* This function calculates the pointers into the hash setup block
* based on the control block
*
* @param[in] pHashControlBlock Pointer to hash control block
* @param[in] pHwBlockBase pointer to base of hardware block
* @param[out] pHashBlkPtrs structure containing pointers to
* various fields in the hash setup block
*
* @return void
*/
static void
LacSymQat_HashHwBlockPtrsInit(icp_qat_fw_auth_cd_ctrl_hdr_t *pHashControlBlock,
void *pHwBlockBase,
lac_hash_blk_ptrs_t *pHashBlkPtrs);
static void
LacSymQat_HashSetupBlockOptimisedFormatInit(
const CpaCySymHashSetupData *pHashSetupData,
icp_qat_fw_auth_cd_ctrl_hdr_t *pHashControlBlock,
void *pHwBlockBase,
icp_qat_hw_auth_mode_t qatHashMode,
lac_sym_qat_hash_precompute_info_t *pPrecompute,
lac_sym_qat_hash_defs_t *pHashDefs,
lac_sym_qat_hash_defs_t *pOuterHashDefs);
/**
* This function populates the hash setup block
*
* @param[in] pHashSetupData Pointer to the hash context
* @param[in] pHashControlBlock Pointer to hash control block
* @param[in] pHwBlockBase pointer to base of hardware block
* @param[in] qatHashMode QAT hash mode
* @param[in] pPrecompute For auth mode, this is the pointer
* to the precompute data. Otherwise this
* should be set to NULL
* @param[in] pHashDefs Pointer to Hash definitions
* @param[in] pOuterHashDefs Pointer to Outer Hash definitions.
* Required for nested hash mode only
*
* @return void
*/
static void
LacSymQat_HashSetupBlockInit(const CpaCySymHashSetupData *pHashSetupData,
icp_qat_fw_auth_cd_ctrl_hdr_t *pHashControlBlock,
void *pHwBlockBase,
icp_qat_hw_auth_mode_t qatHashMode,
lac_sym_qat_hash_precompute_info_t *pPrecompute,
lac_sym_qat_hash_defs_t *pHashDefs,
lac_sym_qat_hash_defs_t *pOuterHashDefs);
/** @ingroup LacSymQatHash */
void
LacSymQat_HashGetCfgData(CpaInstanceHandle pInstance,
icp_qat_hw_auth_mode_t qatHashMode,
CpaCySymHashMode apiHashMode,
CpaCySymHashAlgorithm apiHashAlgorithm,
icp_qat_hw_auth_algo_t *pQatAlgorithm,
CpaBoolean *pQatNested)
{
lac_sym_qat_hash_defs_t *pHashDefs = NULL;
LacSymQat_HashDefsLookupGet(pInstance, apiHashAlgorithm, &pHashDefs);
*pQatAlgorithm = pHashDefs->qatInfo->algoEnc;
if (IS_HASH_MODE_2(qatHashMode)) {
/* set bit for nested hashing */
*pQatNested = ICP_QAT_FW_AUTH_HDR_FLAG_DO_NESTED;
}
/* Nested hash in mode 0. */
else if (CPA_CY_SYM_HASH_MODE_NESTED == apiHashMode) {
/* set bit for nested hashing */
*pQatNested = ICP_QAT_FW_AUTH_HDR_FLAG_DO_NESTED;
}
/* mode0 - plain or mode1 - auth */
else {
*pQatNested = ICP_QAT_FW_AUTH_HDR_FLAG_NO_NESTED;
}
}
/** @ingroup LacSymQatHash */
void
LacSymQat_HashContentDescInit(icp_qat_la_bulk_req_ftr_t *pMsg,
CpaInstanceHandle instanceHandle,
const CpaCySymHashSetupData *pHashSetupData,
void *pHwBlockBase,
Cpa32U hwBlockOffsetInQuadWords,
icp_qat_fw_slice_t nextSlice,
icp_qat_hw_auth_mode_t qatHashMode,
CpaBoolean useSymConstantsTable,
CpaBoolean useOptimisedContentDesc,
lac_sym_qat_hash_precompute_info_t *pPrecompute,
Cpa32U *pHashBlkSizeInBytes)
{
icp_qat_fw_auth_cd_ctrl_hdr_t *cd_ctrl =
(icp_qat_fw_auth_cd_ctrl_hdr_t *)&(pMsg->cd_ctrl);
lac_sym_qat_hash_defs_t *pHashDefs = NULL;
lac_sym_qat_hash_defs_t *pOuterHashDefs = NULL;
Cpa32U hashSetupBlkSize = 0;
/* setup the offset in QuadWords into the hw blk */
cd_ctrl->hash_cfg_offset = hwBlockOffsetInQuadWords;
ICP_QAT_FW_COMN_NEXT_ID_SET(cd_ctrl, nextSlice);
ICP_QAT_FW_COMN_CURR_ID_SET(cd_ctrl, ICP_QAT_FW_SLICE_AUTH);
LacSymQat_HashDefsLookupGet(instanceHandle,
pHashSetupData->hashAlgorithm,
&pHashDefs);
/* Hmac in mode 2 TLS */
if (IS_HASH_MODE_2(qatHashMode)) {
/* Set bit for nested hashing.
* Make sure not to overwrite other flags in hash_flags byte.
*/
ICP_QAT_FW_HASH_FLAG_AUTH_HDR_NESTED_SET(
cd_ctrl->hash_flags, ICP_QAT_FW_AUTH_HDR_FLAG_DO_NESTED);
}
/* Nested hash in mode 0 */
else if (CPA_CY_SYM_HASH_MODE_NESTED == pHashSetupData->hashMode) {
/* Set bit for nested hashing.
* Make sure not to overwrite other flags in hash_flags byte.
*/
ICP_QAT_FW_HASH_FLAG_AUTH_HDR_NESTED_SET(
cd_ctrl->hash_flags, ICP_QAT_FW_AUTH_HDR_FLAG_DO_NESTED);
}
/* mode0 - plain or mode1 - auth */
else {
ICP_QAT_FW_HASH_FLAG_AUTH_HDR_NESTED_SET(
cd_ctrl->hash_flags, ICP_QAT_FW_AUTH_HDR_FLAG_NO_NESTED);
}
/* set the final digest size */
cd_ctrl->final_sz = pHashSetupData->digestResultLenInBytes;
/* set the state1 size */
cd_ctrl->inner_state1_sz =
LAC_ALIGN_POW2_ROUNDUP(pHashDefs->qatInfo->state1Length,
LAC_QUAD_WORD_IN_BYTES);
/* set the inner result size to the digest length */
cd_ctrl->inner_res_sz = pHashDefs->algInfo->digestLength;
/* set the state2 size - only for mode 1 Auth algos and AES CBC MAC */
if (IS_HASH_MODE_1(qatHashMode) ||
pHashSetupData->hashAlgorithm == CPA_CY_SYM_HASH_AES_CBC_MAC ||
pHashSetupData->hashAlgorithm == CPA_CY_SYM_HASH_ZUC_EIA3) {
cd_ctrl->inner_state2_sz =
LAC_ALIGN_POW2_ROUNDUP(pHashDefs->qatInfo->state2Length,
LAC_QUAD_WORD_IN_BYTES);
} else {
cd_ctrl->inner_state2_sz = 0;
}
cd_ctrl->inner_state2_offset = cd_ctrl->hash_cfg_offset +
LAC_BYTES_TO_QUADWORDS(sizeof(icp_qat_hw_auth_setup_t) +
cd_ctrl->inner_state1_sz);
/* size of inner part of hash setup block */
hashSetupBlkSize = sizeof(icp_qat_hw_auth_setup_t) +
cd_ctrl->inner_state1_sz + cd_ctrl->inner_state2_sz;
/* For nested hashing - Fill in the outer fields */
if (CPA_CY_SYM_HASH_MODE_NESTED == pHashSetupData->hashMode ||
IS_HASH_MODE_2(qatHashMode)) {
/* For nested - use the outer algorithm. This covers TLS and
* nested hash. For HMAC mode2 use inner algorithm again */
CpaCySymHashAlgorithm outerAlg =
(CPA_CY_SYM_HASH_MODE_NESTED == pHashSetupData->hashMode) ?
pHashSetupData->nestedModeSetupData.outerHashAlgorithm :
pHashSetupData->hashAlgorithm;
LacSymQat_HashDefsLookupGet(instanceHandle,
outerAlg,
&pOuterHashDefs);
/* outer config offset */
cd_ctrl->outer_config_offset = cd_ctrl->inner_state2_offset +
LAC_BYTES_TO_QUADWORDS(cd_ctrl->inner_state2_sz);
cd_ctrl->outer_state1_sz =
LAC_ALIGN_POW2_ROUNDUP(pOuterHashDefs->algInfo->stateSize,
LAC_QUAD_WORD_IN_BYTES);
/* outer result size */
cd_ctrl->outer_res_sz = pOuterHashDefs->algInfo->digestLength;
/* outer_prefix_offset will be the size of the inner prefix data
* plus the hash state storage size. */
/* The prefix buffer is part of the ReqParams, so this param
* will be
* setup where ReqParams are set up */
/* add on size of outer part of hash block */
hashSetupBlkSize +=
sizeof(icp_qat_hw_auth_setup_t) + cd_ctrl->outer_state1_sz;
} else {
cd_ctrl->outer_config_offset = 0;
cd_ctrl->outer_state1_sz = 0;
cd_ctrl->outer_res_sz = 0;
}
if (CPA_CY_SYM_HASH_SNOW3G_UIA2 == pHashSetupData->hashAlgorithm) {
/* add the size for the cipher config word, the key and the IV*/
hashSetupBlkSize += sizeof(icp_qat_hw_cipher_config_t) +
pHashSetupData->authModeSetupData.authKeyLenInBytes +
ICP_QAT_HW_SNOW_3G_UEA2_IV_SZ;
}
*pHashBlkSizeInBytes = hashSetupBlkSize;
if (useOptimisedContentDesc) {
LacSymQat_HashSetupBlockOptimisedFormatInit(pHashSetupData,
cd_ctrl,
pHwBlockBase,
qatHashMode,
pPrecompute,
pHashDefs,
pOuterHashDefs);
} else if (!useSymConstantsTable) {
/*****************************************************************************
* Populate Hash Setup block *
*****************************************************************************/
LacSymQat_HashSetupBlockInit(pHashSetupData,
cd_ctrl,
pHwBlockBase,
qatHashMode,
pPrecompute,
pHashDefs,
pOuterHashDefs);
}
}
/* This fn populates fields in both the CD ctrl block and the ReqParams block
* which describe the Hash ReqParams:
* cd_ctrl.outer_prefix_offset
* cd_ctrl.outer_prefix_sz
* req_params.inner_prefix_sz/aad_sz
* req_params.hash_state_sz
* req_params.auth_res_sz
*
*/
void
LacSymQat_HashSetupReqParamsMetaData(
icp_qat_la_bulk_req_ftr_t *pMsg,
CpaInstanceHandle instanceHandle,
const CpaCySymHashSetupData *pHashSetupData,
CpaBoolean hashStateBuffer,
icp_qat_hw_auth_mode_t qatHashMode,
CpaBoolean digestVerify)
{
icp_qat_fw_auth_cd_ctrl_hdr_t *cd_ctrl = NULL;
icp_qat_la_auth_req_params_t *pHashReqParams = NULL;
lac_sym_qat_hash_defs_t *pHashDefs = NULL;
cd_ctrl = (icp_qat_fw_auth_cd_ctrl_hdr_t *)&(pMsg->cd_ctrl);
pHashReqParams =
(icp_qat_la_auth_req_params_t *)(&(pMsg->serv_specif_rqpars));
LacSymQat_HashDefsLookupGet(instanceHandle,
pHashSetupData->hashAlgorithm,
&pHashDefs);
/* Hmac in mode 2 TLS */
if (IS_HASH_MODE_2(qatHashMode)) {
/* Inner and outer prefixes are the block length */
pHashReqParams->u2.inner_prefix_sz =
pHashDefs->algInfo->blockLength;
cd_ctrl->outer_prefix_sz = pHashDefs->algInfo->blockLength;
cd_ctrl->outer_prefix_offset = LAC_BYTES_TO_QUADWORDS(
LAC_ALIGN_POW2_ROUNDUP((pHashReqParams->u2.inner_prefix_sz),
LAC_QUAD_WORD_IN_BYTES));
}
/* Nested hash in mode 0 */
else if (CPA_CY_SYM_HASH_MODE_NESTED == pHashSetupData->hashMode) {
/* set inner and outer prefixes */
pHashReqParams->u2.inner_prefix_sz =
pHashSetupData->nestedModeSetupData.innerPrefixLenInBytes;
cd_ctrl->outer_prefix_sz =
pHashSetupData->nestedModeSetupData.outerPrefixLenInBytes;
cd_ctrl->outer_prefix_offset = LAC_BYTES_TO_QUADWORDS(
LAC_ALIGN_POW2_ROUNDUP((pHashReqParams->u2.inner_prefix_sz),
LAC_QUAD_WORD_IN_BYTES));
}
/* mode0 - plain or mode1 - auth */
else {
Cpa16U aadDataSize = 0;
/* For Auth Encrypt set the aad size */
if (CPA_CY_SYM_HASH_AES_CCM == pHashSetupData->hashAlgorithm) {
/* at the beginning of the buffer there is B0 block */
aadDataSize = LAC_HASH_AES_CCM_BLOCK_SIZE;
/* then, if there is some 'a' data, the buffer will
* store encoded
* length of 'a' and 'a' itself */
if (pHashSetupData->authModeSetupData.aadLenInBytes >
0) {
/* as the QAT API puts the requirement on the
* pAdditionalAuthData not to be bigger than 240
* bytes then we
* just need 2 bytes to store encoded length of
* 'a' */
aadDataSize += sizeof(Cpa16U);
aadDataSize += pHashSetupData->authModeSetupData
.aadLenInBytes;
}
/* round the aad size to the multiple of CCM block
* size.*/
pHashReqParams->u2.aad_sz =
LAC_ALIGN_POW2_ROUNDUP(aadDataSize,
LAC_HASH_AES_CCM_BLOCK_SIZE);
} else if (CPA_CY_SYM_HASH_AES_GCM ==
pHashSetupData->hashAlgorithm) {
aadDataSize =
pHashSetupData->authModeSetupData.aadLenInBytes;
/* round the aad size to the multiple of GCM hash block
* size. */
pHashReqParams->u2.aad_sz =
LAC_ALIGN_POW2_ROUNDUP(aadDataSize,
LAC_HASH_AES_GCM_BLOCK_SIZE);
} else {
pHashReqParams->u2.aad_sz = 0;
}
cd_ctrl->outer_prefix_sz = 0;
cd_ctrl->outer_prefix_offset = 0;
}
/* If there is a hash state prefix buffer */
if (CPA_TRUE == hashStateBuffer) {
/* Note, this sets up size for both aad and non-aad cases */
pHashReqParams->hash_state_sz = LAC_BYTES_TO_QUADWORDS(
LAC_ALIGN_POW2_ROUNDUP(pHashReqParams->u2.inner_prefix_sz,
LAC_QUAD_WORD_IN_BYTES) +
LAC_ALIGN_POW2_ROUNDUP(cd_ctrl->outer_prefix_sz,
LAC_QUAD_WORD_IN_BYTES));
} else {
pHashReqParams->hash_state_sz = 0;
}
if (CPA_TRUE == digestVerify) {
/* auth result size in bytes to be read in for a verify
* operation */
pHashReqParams->auth_res_sz =
pHashSetupData->digestResultLenInBytes;
} else {
pHashReqParams->auth_res_sz = 0;
}
pHashReqParams->resrvd1 = 0;
}
void
LacSymQat_HashHwBlockPtrsInit(icp_qat_fw_auth_cd_ctrl_hdr_t *cd_ctrl,
void *pHwBlockBase,
lac_hash_blk_ptrs_t *pHashBlkPtrs)
{
/* encoded offset for inner config is converted to a byte offset. */
pHashBlkPtrs->pInHashSetup =
(icp_qat_hw_auth_setup_t *)((Cpa8U *)pHwBlockBase +
(cd_ctrl->hash_cfg_offset *
LAC_QUAD_WORD_IN_BYTES));
pHashBlkPtrs->pInHashInitState1 = (Cpa8U *)pHashBlkPtrs->pInHashSetup +
sizeof(icp_qat_hw_auth_setup_t);
pHashBlkPtrs->pInHashInitState2 =
(Cpa8U *)(pHashBlkPtrs->pInHashInitState1) +
cd_ctrl->inner_state1_sz;
pHashBlkPtrs->pOutHashSetup =
(icp_qat_hw_auth_setup_t *)((Cpa8U *)(pHashBlkPtrs
->pInHashInitState2) +
cd_ctrl->inner_state2_sz);
pHashBlkPtrs->pOutHashInitState1 =
(Cpa8U *)(pHashBlkPtrs->pOutHashSetup) +
sizeof(icp_qat_hw_auth_setup_t);
}
static void
LacSymQat_HashSetupBlockInit(const CpaCySymHashSetupData *pHashSetupData,
icp_qat_fw_auth_cd_ctrl_hdr_t *pHashControlBlock,
void *pHwBlockBase,
icp_qat_hw_auth_mode_t qatHashMode,
lac_sym_qat_hash_precompute_info_t *pPrecompute,
lac_sym_qat_hash_defs_t *pHashDefs,
lac_sym_qat_hash_defs_t *pOuterHashDefs)
{
Cpa32U innerConfig = 0;
lac_hash_blk_ptrs_t hashBlkPtrs = { 0 };
Cpa32U aed_hash_cmp_length = 0;
LacSymQat_HashHwBlockPtrsInit(pHashControlBlock,
pHwBlockBase,
&hashBlkPtrs);
innerConfig = ICP_QAT_HW_AUTH_CONFIG_BUILD(
qatHashMode,
pHashDefs->qatInfo->algoEnc,
pHashSetupData->digestResultLenInBytes);
/* Set the Inner hash configuration */
hashBlkPtrs.pInHashSetup->auth_config.config = innerConfig;
hashBlkPtrs.pInHashSetup->auth_config.reserved = 0;
/* For mode 1 pre-computes for auth algorithms */
if (IS_HASH_MODE_1(qatHashMode) ||
CPA_CY_SYM_HASH_AES_CBC_MAC == pHashSetupData->hashAlgorithm ||
CPA_CY_SYM_HASH_ZUC_EIA3 == pHashSetupData->hashAlgorithm) {
/* for HMAC in mode 1 authCounter is the block size
* else the authCounter is 0. The firmware expects the counter
* to be
* big endian */
LAC_MEM_SHARED_WRITE_SWAP(
hashBlkPtrs.pInHashSetup->auth_counter.counter,
pHashDefs->qatInfo->authCounter);
/* state 1 is set to 0 for the following algorithms */
if ((CPA_CY_SYM_HASH_AES_XCBC ==
pHashSetupData->hashAlgorithm) ||
(CPA_CY_SYM_HASH_AES_CMAC ==
pHashSetupData->hashAlgorithm) ||
(CPA_CY_SYM_HASH_AES_CBC_MAC ==
pHashSetupData->hashAlgorithm) ||
(CPA_CY_SYM_HASH_KASUMI_F9 ==
pHashSetupData->hashAlgorithm) ||
(CPA_CY_SYM_HASH_SNOW3G_UIA2 ==
pHashSetupData->hashAlgorithm) ||
(CPA_CY_SYM_HASH_AES_CCM ==
pHashSetupData->hashAlgorithm) ||
(CPA_CY_SYM_HASH_AES_GMAC ==
pHashSetupData->hashAlgorithm) ||
(CPA_CY_SYM_HASH_AES_GCM ==
pHashSetupData->hashAlgorithm) ||
(CPA_CY_SYM_HASH_ZUC_EIA3 ==
pHashSetupData->hashAlgorithm)) {
LAC_OS_BZERO(hashBlkPtrs.pInHashInitState1,
pHashDefs->qatInfo->state1Length);
}
/* Pad remaining bytes of sha1 precomputes */
if (CPA_CY_SYM_HASH_SHA1 == pHashSetupData->hashAlgorithm) {
Cpa32U state1PadLen = 0;
Cpa32U state2PadLen = 0;
if (pHashControlBlock->inner_state1_sz >
pHashDefs->algInfo->stateSize) {
state1PadLen =
pHashControlBlock->inner_state1_sz -
pHashDefs->algInfo->stateSize;
}
if (pHashControlBlock->inner_state2_sz >
pHashDefs->algInfo->stateSize) {
state2PadLen =
pHashControlBlock->inner_state2_sz -
pHashDefs->algInfo->stateSize;
}
if (state1PadLen > 0) {
LAC_OS_BZERO(hashBlkPtrs.pInHashInitState1 +
pHashDefs->algInfo->stateSize,
state1PadLen);
}
if (state2PadLen > 0) {
LAC_OS_BZERO(hashBlkPtrs.pInHashInitState2 +
pHashDefs->algInfo->stateSize,
state2PadLen);
}
}
pPrecompute->state1Size = pHashDefs->qatInfo->state1Length;
pPrecompute->state2Size = pHashDefs->qatInfo->state2Length;
/* Set the destination for pre-compute state1 data to be written
*/
pPrecompute->pState1 = hashBlkPtrs.pInHashInitState1;
/* Set the destination for pre-compute state1 data to be written
*/
pPrecompute->pState2 = hashBlkPtrs.pInHashInitState2;
}
/* For digest and nested digest */
else {
Cpa32U padLen = pHashControlBlock->inner_state1_sz -
pHashDefs->algInfo->stateSize;
/* counter set to 0 */
hashBlkPtrs.pInHashSetup->auth_counter.counter = 0;
/* set the inner hash state 1 */
memcpy(hashBlkPtrs.pInHashInitState1,
pHashDefs->algInfo->initState,
pHashDefs->algInfo->stateSize);
if (padLen > 0) {
LAC_OS_BZERO(hashBlkPtrs.pInHashInitState1 +
pHashDefs->algInfo->stateSize,
padLen);
}
}
hashBlkPtrs.pInHashSetup->auth_counter.reserved = 0;
/* Fill in the outer part of the hash setup block */
if ((CPA_CY_SYM_HASH_MODE_NESTED == pHashSetupData->hashMode ||
IS_HASH_MODE_2(qatHashMode)) &&
(NULL != pOuterHashDefs)) {
Cpa32U outerConfig = ICP_QAT_HW_AUTH_CONFIG_BUILD(
qatHashMode,
pOuterHashDefs->qatInfo->algoEnc,
pHashSetupData->digestResultLenInBytes);
Cpa32U padLen = pHashControlBlock->outer_state1_sz -
pOuterHashDefs->algInfo->stateSize;
/* populate the auth config */
hashBlkPtrs.pOutHashSetup->auth_config.config = outerConfig;
hashBlkPtrs.pOutHashSetup->auth_config.reserved = 0;
/* outer Counter set to 0 */
hashBlkPtrs.pOutHashSetup->auth_counter.counter = 0;
hashBlkPtrs.pOutHashSetup->auth_counter.reserved = 0;
/* set outer hash state 1 */
memcpy(hashBlkPtrs.pOutHashInitState1,
pOuterHashDefs->algInfo->initState,
pOuterHashDefs->algInfo->stateSize);
if (padLen > 0) {
LAC_OS_BZERO(hashBlkPtrs.pOutHashInitState1 +
pOuterHashDefs->algInfo->stateSize,
padLen);
}
}
if (CPA_CY_SYM_HASH_SNOW3G_UIA2 == pHashSetupData->hashAlgorithm) {
icp_qat_hw_cipher_config_t *pCipherConfig =
(icp_qat_hw_cipher_config_t *)hashBlkPtrs.pOutHashSetup;
pCipherConfig->val = ICP_QAT_HW_CIPHER_CONFIG_BUILD(
ICP_QAT_HW_CIPHER_ECB_MODE,
ICP_QAT_HW_CIPHER_ALGO_SNOW_3G_UEA2,
ICP_QAT_HW_CIPHER_KEY_CONVERT,
ICP_QAT_HW_CIPHER_ENCRYPT,
aed_hash_cmp_length);
pCipherConfig->reserved = 0;
memcpy((Cpa8U *)pCipherConfig +
sizeof(icp_qat_hw_cipher_config_t),
pHashSetupData->authModeSetupData.authKey,
pHashSetupData->authModeSetupData.authKeyLenInBytes);
LAC_OS_BZERO(
(Cpa8U *)pCipherConfig +
sizeof(icp_qat_hw_cipher_config_t) +
pHashSetupData->authModeSetupData.authKeyLenInBytes,
ICP_QAT_HW_SNOW_3G_UEA2_IV_SZ);
} else if (CPA_CY_SYM_HASH_ZUC_EIA3 == pHashSetupData->hashAlgorithm) {
icp_qat_hw_cipher_config_t *pCipherConfig =
(icp_qat_hw_cipher_config_t *)hashBlkPtrs.pOutHashSetup;
pCipherConfig->val = ICP_QAT_HW_CIPHER_CONFIG_BUILD(
ICP_QAT_HW_CIPHER_ECB_MODE,
ICP_QAT_HW_CIPHER_ALGO_ZUC_3G_128_EEA3,
ICP_QAT_HW_CIPHER_KEY_CONVERT,
ICP_QAT_HW_CIPHER_ENCRYPT,
aed_hash_cmp_length);
pCipherConfig->reserved = 0;
memcpy((Cpa8U *)pCipherConfig +
sizeof(icp_qat_hw_cipher_config_t),
pHashSetupData->authModeSetupData.authKey,
pHashSetupData->authModeSetupData.authKeyLenInBytes);
LAC_OS_BZERO(
(Cpa8U *)pCipherConfig +
sizeof(icp_qat_hw_cipher_config_t) +
pHashSetupData->authModeSetupData.authKeyLenInBytes,
ICP_QAT_HW_ZUC_3G_EEA3_IV_SZ);
}
}
static void
LacSymQat_HashOpHwBlockPtrsInit(icp_qat_fw_auth_cd_ctrl_hdr_t *cd_ctrl,
void *pHwBlockBase,
lac_hash_blk_ptrs_optimised_t *pHashBlkPtrs)
{
pHashBlkPtrs->pInHashInitState1 = (((Cpa8U *)pHwBlockBase) + 16);
pHashBlkPtrs->pInHashInitState2 =
(Cpa8U *)(pHashBlkPtrs->pInHashInitState1) +
cd_ctrl->inner_state1_sz;
}
static void
LacSymQat_HashSetupBlockOptimisedFormatInit(
const CpaCySymHashSetupData *pHashSetupData,
icp_qat_fw_auth_cd_ctrl_hdr_t *pHashControlBlock,
void *pHwBlockBase,
icp_qat_hw_auth_mode_t qatHashMode,
lac_sym_qat_hash_precompute_info_t *pPrecompute,
lac_sym_qat_hash_defs_t *pHashDefs,
lac_sym_qat_hash_defs_t *pOuterHashDefs)
{
Cpa32U state1PadLen = 0;
Cpa32U state2PadLen = 0;
lac_hash_blk_ptrs_optimised_t pHashBlkPtrs = { 0 };
LacSymQat_HashOpHwBlockPtrsInit(pHashControlBlock,
pHwBlockBase,
&pHashBlkPtrs);
if (pHashControlBlock->inner_state1_sz >
pHashDefs->algInfo->stateSize) {
state1PadLen = pHashControlBlock->inner_state1_sz -
pHashDefs->algInfo->stateSize;
}
if (pHashControlBlock->inner_state2_sz >
pHashDefs->algInfo->stateSize) {
state2PadLen = pHashControlBlock->inner_state2_sz -
pHashDefs->algInfo->stateSize;
}
if (state1PadLen > 0) {
LAC_OS_BZERO(pHashBlkPtrs.pInHashInitState1 +
pHashDefs->algInfo->stateSize,
state1PadLen);
}
if (state2PadLen > 0) {
LAC_OS_BZERO(pHashBlkPtrs.pInHashInitState2 +
pHashDefs->algInfo->stateSize,
state2PadLen);
}
pPrecompute->state1Size = pHashDefs->qatInfo->state1Length;
pPrecompute->state2Size = pHashDefs->qatInfo->state2Length;
/* Set the destination for pre-compute state1 data to be written */
pPrecompute->pState1 = pHashBlkPtrs.pInHashInitState1;
/* Set the destination for pre-compute state1 data to be written */
pPrecompute->pState2 = pHashBlkPtrs.pInHashInitState2;
}
void
LacSymQat_HashStatePrefixAadBufferSizeGet(
icp_qat_la_bulk_req_ftr_t *pMsg,
lac_sym_qat_hash_state_buffer_info_t *pHashStateBuf)
{
const icp_qat_fw_auth_cd_ctrl_hdr_t *cd_ctrl;
icp_qat_la_auth_req_params_t *pHashReqParams;
cd_ctrl = (icp_qat_fw_auth_cd_ctrl_hdr_t *)&(pMsg->cd_ctrl);
pHashReqParams =
(icp_qat_la_auth_req_params_t *)(&(pMsg->serv_specif_rqpars));
/* hash state storage needed to support partial packets. Space reserved
* for this in all cases */
pHashStateBuf->stateStorageSzQuadWords = LAC_BYTES_TO_QUADWORDS(
sizeof(icp_qat_hw_auth_counter_t) + cd_ctrl->inner_state1_sz);
pHashStateBuf->prefixAadSzQuadWords = pHashReqParams->hash_state_sz;
}
void
LacSymQat_HashStatePrefixAadBufferPopulate(
lac_sym_qat_hash_state_buffer_info_t *pHashStateBuf,
icp_qat_la_bulk_req_ftr_t *pMsg,
Cpa8U *pInnerPrefixAad,
Cpa8U innerPrefixSize,
Cpa8U *pOuterPrefix,
Cpa8U outerPrefixSize)
{
const icp_qat_fw_auth_cd_ctrl_hdr_t *cd_ctrl =
(icp_qat_fw_auth_cd_ctrl_hdr_t *)&(pMsg->cd_ctrl);
icp_qat_la_auth_req_params_t *pHashReqParams =
(icp_qat_la_auth_req_params_t *)(&(pMsg->serv_specif_rqpars));
/*
* Let S be the supplied secret
* S1 = S/2 if S is even and (S/2 + 1) if S is odd.
* Set length S2 (inner prefix) = S1 and the start address
* of S2 is S[S1/2] i.e. if S is odd then S2 starts at the last byte of
* S1
* _____________________________________________________________
* | outer prefix | padding |
* |________________| |
* | |
* |____________________________________________________________|
* | inner prefix | padding |
* |________________| |
* | |
* |____________________________________________________________|
*
*/
if (NULL != pInnerPrefixAad) {
Cpa8U *pLocalInnerPrefix =
(Cpa8U *)(pHashStateBuf->pData) +
LAC_QUADWORDS_TO_BYTES(
pHashStateBuf->stateStorageSzQuadWords);
Cpa8U padding =
pHashReqParams->u2.inner_prefix_sz - innerPrefixSize;
/* copy the inner prefix or aad data */
memcpy(pLocalInnerPrefix, pInnerPrefixAad, innerPrefixSize);
/* Reset with zeroes any area reserved for padding in this block
*/
if (0 < padding) {
LAC_OS_BZERO(pLocalInnerPrefix + innerPrefixSize,
padding);
}
}
if (NULL != pOuterPrefix) {
Cpa8U *pLocalOuterPrefix =
(Cpa8U *)pHashStateBuf->pData +
LAC_QUADWORDS_TO_BYTES(
pHashStateBuf->stateStorageSzQuadWords +
cd_ctrl->outer_prefix_offset);
Cpa8U padding = LAC_QUADWORDS_TO_BYTES(
pHashStateBuf->prefixAadSzQuadWords) -
pHashReqParams->u2.inner_prefix_sz - outerPrefixSize;
/* copy the outer prefix */
memcpy(pLocalOuterPrefix, pOuterPrefix, outerPrefixSize);
/* Reset with zeroes any area reserved for padding in this block
*/
if (0 < padding) {
LAC_OS_BZERO(pLocalOuterPrefix + outerPrefixSize,
padding);
}
}
}
inline CpaStatus
LacSymQat_HashRequestParamsPopulate(
icp_qat_fw_la_bulk_req_t *pReq,
Cpa32U authOffsetInBytes,
Cpa32U authLenInBytes,
sal_service_t *pService,
lac_sym_qat_hash_state_buffer_info_t *pHashStateBuf,
Cpa32U packetType,
Cpa32U hashResultSize,
CpaBoolean digestVerify,
Cpa8U *pAuthResult,
CpaCySymHashAlgorithm alg,
void *hkdf_secret)
{
Cpa64U authResultPhys = 0;
icp_qat_fw_la_auth_req_params_t *pHashReqParams;
pHashReqParams = (icp_qat_fw_la_auth_req_params_t
*)((Cpa8U *)&(pReq->serv_specif_rqpars) +
ICP_QAT_FW_HASH_REQUEST_PARAMETERS_OFFSET);
pHashReqParams->auth_off = authOffsetInBytes;
pHashReqParams->auth_len = authLenInBytes;
/* Set the physical location of secret for HKDF */
if (NULL != hkdf_secret) {
LAC_MEM_SHARED_WRITE_VIRT_TO_PHYS_PTR_EXTERNAL(
(*pService), pHashReqParams->u1.aad_adr, hkdf_secret);
if (pHashReqParams->u1.aad_adr == 0) {
LAC_LOG_ERROR(
"Unable to get the physical address of the"
" HKDF secret\n");
return CPA_STATUS_FAIL;
}
}
/* For a Full packet or last partial need to set the digest result
* pointer
* and the auth result field */
if (NULL != pAuthResult) {
authResultPhys =
LAC_OS_VIRT_TO_PHYS_EXTERNAL((*pService),
(void *)pAuthResult);
if (authResultPhys == 0) {
LAC_LOG_ERROR(
"Unable to get the physical address of the"
" auth result\n");
return CPA_STATUS_FAIL;
}
pHashReqParams->auth_res_addr = authResultPhys;
} else {
pHashReqParams->auth_res_addr = 0;
}
if (CPA_TRUE == digestVerify) {
/* auth result size in bytes to be read in for a verify
* operation */
pHashReqParams->auth_res_sz = hashResultSize;
} else {
pHashReqParams->auth_res_sz = 0;
}
/* If there is a hash state prefix buffer */
if (NULL != pHashStateBuf) {
/* Only write the pointer to the buffer if the size is greater
* than 0
* this will be the case for plain and auth mode due to the
* state storage required for partial packets and for nested
* mode (when
* the prefix data is > 0) */
if ((pHashStateBuf->stateStorageSzQuadWords +
pHashStateBuf->prefixAadSzQuadWords) > 0) {
/* For the first partial packet, the QAT expects the
* pointer to the
* inner prefix even if there is no memory allocated for
* this. The
* QAT will internally calculate where to write the
* state back. */
if ((ICP_QAT_FW_LA_PARTIAL_START == packetType) ||
(ICP_QAT_FW_LA_PARTIAL_NONE == packetType)) {
// prefix_addr changed to auth_partial_st_prefix
pHashReqParams->u1.auth_partial_st_prefix =
((pHashStateBuf->pDataPhys) +
LAC_QUADWORDS_TO_BYTES(
pHashStateBuf
->stateStorageSzQuadWords));
} else {
pHashReqParams->u1.auth_partial_st_prefix =
pHashStateBuf->pDataPhys;
}
}
/* nested mode when the prefix data is 0 */
else {
pHashReqParams->u1.auth_partial_st_prefix = 0;
}
/* For middle & last partial, state size is the hash state
* storage
* if hash mode 2 this will include the prefix data */
if ((ICP_QAT_FW_LA_PARTIAL_MID == packetType) ||
(ICP_QAT_FW_LA_PARTIAL_END == packetType)) {
pHashReqParams->hash_state_sz =
(pHashStateBuf->stateStorageSzQuadWords +
pHashStateBuf->prefixAadSzQuadWords);
}
/* For full packets and first partials set the state size to
* that of
* the prefix/aad. prefix includes both the inner and outer
* prefix */
else {
pHashReqParams->hash_state_sz =
pHashStateBuf->prefixAadSzQuadWords;
}
} else {
pHashReqParams->u1.auth_partial_st_prefix = 0;
pHashReqParams->hash_state_sz = 0;
}
/* GMAC only */
if (CPA_CY_SYM_HASH_AES_GMAC == alg) {
pHashReqParams->hash_state_sz = 0;
pHashReqParams->u1.aad_adr = 0;
}
/* This field is only used by TLS requests */
/* In TLS case this is set after this function is called */
pHashReqParams->resrvd1 = 0;
return CPA_STATUS_SUCCESS;
}

491
sys/dev/qat/qat_api/common/crypto/sym/qat/lac_sym_qat_hash_defs_lookup.c Normal file
View File

@ -0,0 +1,491 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_sym_qat_hash_defs_lookup.c Hash Definitions Lookup
*
* @ingroup LacHashDefsLookup
***************************************************************************/
/*
*******************************************************************************
* Include public/global header files
*******************************************************************************
*/
#include "cpa.h"
/*
*******************************************************************************
* Include private header files
*******************************************************************************
*/
#include "lac_common.h"
#include "icp_accel_devices.h"
#include "icp_adf_debug.h"
#include "icp_adf_transport.h"
#include "lac_sym.h"
#include "icp_qat_fw_la.h"
#include "lac_sym_qat_hash_defs_lookup.h"
#include "lac_sal_types_crypto.h"
#include "lac_sym_hash_defs.h"
/* state size for xcbc mac consists of 3 * 16 byte keys */
#define LAC_SYM_QAT_XCBC_STATE_SIZE ((LAC_HASH_XCBC_MAC_BLOCK_SIZE)*3)
#define LAC_SYM_QAT_CMAC_STATE_SIZE ((LAC_HASH_CMAC_BLOCK_SIZE)*3)
/* This type is used for the mapping between the hash algorithm and
* the corresponding hash definitions structure */
typedef struct lac_sym_qat_hash_def_map_s {
CpaCySymHashAlgorithm hashAlgorithm;
/* hash algorithm */
lac_sym_qat_hash_defs_t hashDefs;
/* hash defintions pointers */
} lac_sym_qat_hash_def_map_t;
/*
*******************************************************************************
* Static Variables
*******************************************************************************
*/
/* initialisers as defined in FIPS and RFCS for digest operations */
/* md5 16 bytes - Initialiser state can be found in RFC 1321*/
static Cpa8U md5InitialState[LAC_HASH_MD5_STATE_SIZE] = {
0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
0xfe, 0xdc, 0xba, 0x98, 0x76, 0x54, 0x32, 0x10,
};
/* SHA1 - 20 bytes - Initialiser state can be found in FIPS stds 180-2 */
static Cpa8U sha1InitialState[LAC_HASH_SHA1_STATE_SIZE] = {
0x67, 0x45, 0x23, 0x01, 0xef, 0xcd, 0xab, 0x89, 0x98, 0xba,
0xdc, 0xfe, 0x10, 0x32, 0x54, 0x76, 0xc3, 0xd2, 0xe1, 0xf0
};
/* SHA 224 - 32 bytes - Initialiser state can be found in FIPS stds 180-2 */
static Cpa8U sha224InitialState[LAC_HASH_SHA224_STATE_SIZE] = {
0xc1, 0x05, 0x9e, 0xd8, 0x36, 0x7c, 0xd5, 0x07, 0x30, 0x70, 0xdd,
0x17, 0xf7, 0x0e, 0x59, 0x39, 0xff, 0xc0, 0x0b, 0x31, 0x68, 0x58,
0x15, 0x11, 0x64, 0xf9, 0x8f, 0xa7, 0xbe, 0xfa, 0x4f, 0xa4
};
/* SHA 256 - 32 bytes - Initialiser state can be found in FIPS stds 180-2 */
static Cpa8U sha256InitialState[LAC_HASH_SHA256_STATE_SIZE] =
{ 0x6a, 0x09, 0xe6, 0x67, 0xbb, 0x67, 0xae, 0x85, 0x3c, 0x6e, 0xf3,
0x72, 0xa5, 0x4f, 0xf5, 0x3a, 0x51, 0x0e, 0x52, 0x7f, 0x9b, 0x05,
0x68, 0x8c, 0x1f, 0x83, 0xd9, 0xab, 0x5b, 0xe0, 0xcd, 0x19 };
/* SHA 384 - 64 bytes - Initialiser state can be found in FIPS stds 180-2 */
static Cpa8U sha384InitialState[LAC_HASH_SHA384_STATE_SIZE] =
{ 0xcb, 0xbb, 0x9d, 0x5d, 0xc1, 0x05, 0x9e, 0xd8, 0x62, 0x9a, 0x29,
0x2a, 0x36, 0x7c, 0xd5, 0x07, 0x91, 0x59, 0x01, 0x5a, 0x30, 0x70,
0xdd, 0x17, 0x15, 0x2f, 0xec, 0xd8, 0xf7, 0x0e, 0x59, 0x39, 0x67,
0x33, 0x26, 0x67, 0xff, 0xc0, 0x0b, 0x31, 0x8e, 0xb4, 0x4a, 0x87,
0x68, 0x58, 0x15, 0x11, 0xdb, 0x0c, 0x2e, 0x0d, 0x64, 0xf9, 0x8f,
0xa7, 0x47, 0xb5, 0x48, 0x1d, 0xbe, 0xfa, 0x4f, 0xa4 };
/* SHA 512 - 64 bytes - Initialiser state can be found in FIPS stds 180-2 */
static Cpa8U sha512InitialState[LAC_HASH_SHA512_STATE_SIZE] =
{ 0x6a, 0x09, 0xe6, 0x67, 0xf3, 0xbc, 0xc9, 0x08, 0xbb, 0x67, 0xae,
0x85, 0x84, 0xca, 0xa7, 0x3b, 0x3c, 0x6e, 0xf3, 0x72, 0xfe, 0x94,
0xf8, 0x2b, 0xa5, 0x4f, 0xf5, 0x3a, 0x5f, 0x1d, 0x36, 0xf1, 0x51,
0x0e, 0x52, 0x7f, 0xad, 0xe6, 0x82, 0xd1, 0x9b, 0x05, 0x68, 0x8c,
0x2b, 0x3e, 0x6c, 0x1f, 0x1f, 0x83, 0xd9, 0xab, 0xfb, 0x41, 0xbd,
0x6b, 0x5b, 0xe0, 0xcd, 0x19, 0x13, 0x7e, 0x21, 0x79 };
/* SHA3 224 - 28 bytes */
static Cpa8U sha3_224InitialState[LAC_HASH_SHA3_224_STATE_SIZE] =
{ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
/* SHA3 256 - 32 bytes */
static Cpa8U sha3_256InitialState[LAC_HASH_SHA3_256_STATE_SIZE] =
{ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
/* SHA3 384 - 48 bytes */
static Cpa8U sha3_384InitialState[LAC_HASH_SHA3_384_STATE_SIZE] =
{ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
/* SHA3 512 - 64 bytes */
static Cpa8U sha3_512InitialState[LAC_HASH_SHA3_512_STATE_SIZE] =
{ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
/* SM3 - 32 bytes */
static Cpa8U sm3InitialState[LAC_HASH_SM3_STATE_SIZE] =
{ 0x73, 0x80, 0x16, 0x6f, 0x49, 0x14, 0xb2, 0xb9, 0x17, 0x24, 0x42,
0xd7, 0xda, 0x8a, 0x06, 0x00, 0xa9, 0x6f, 0x30, 0xbc, 0x16, 0x31,
0x38, 0xaa, 0xe3, 0x8d, 0xee, 0x4d, 0xb0, 0xfb, 0x0e, 0x4e };
/* Constants used in generating K1, K2, K3 from a Key for AES_XCBC_MAC
* State defined in RFC 3566 */
static Cpa8U aesXcbcKeySeed[LAC_SYM_QAT_XCBC_STATE_SIZE] = {
0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
0x01, 0x01, 0x01, 0x01, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02,
0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x03, 0x03, 0x03, 0x03,
0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03,
};
static Cpa8U aesCmacKeySeed[LAC_HASH_CMAC_BLOCK_SIZE] = { 0x00, 0x00, 0x00,
0x00, 0x00, 0x00,
0x00, 0x00, 0x00,
0x00, 0x00, 0x00,
0x00, 0x00, 0x00,
0x00 };
/* Hash Algorithm specific structure */
static lac_sym_qat_hash_alg_info_t md5Info = { LAC_HASH_MD5_DIGEST_SIZE,
LAC_HASH_MD5_BLOCK_SIZE,
md5InitialState,
LAC_HASH_MD5_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t sha1Info = { LAC_HASH_SHA1_DIGEST_SIZE,
LAC_HASH_SHA1_BLOCK_SIZE,
sha1InitialState,
LAC_HASH_SHA1_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t sha224Info = { LAC_HASH_SHA224_DIGEST_SIZE,
LAC_HASH_SHA224_BLOCK_SIZE,
sha224InitialState,
LAC_HASH_SHA224_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t sha256Info = { LAC_HASH_SHA256_DIGEST_SIZE,
LAC_HASH_SHA256_BLOCK_SIZE,
sha256InitialState,
LAC_HASH_SHA256_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t sha384Info = { LAC_HASH_SHA384_DIGEST_SIZE,
LAC_HASH_SHA384_BLOCK_SIZE,
sha384InitialState,
LAC_HASH_SHA384_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t sha512Info = { LAC_HASH_SHA512_DIGEST_SIZE,
LAC_HASH_SHA512_BLOCK_SIZE,
sha512InitialState,
LAC_HASH_SHA512_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t sha3_224Info =
{ LAC_HASH_SHA3_224_DIGEST_SIZE,
LAC_HASH_SHA3_224_BLOCK_SIZE,
sha3_224InitialState,
LAC_HASH_SHA3_224_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t sha3_256Info =
{ LAC_HASH_SHA3_256_DIGEST_SIZE,
LAC_HASH_SHA3_256_BLOCK_SIZE,
sha3_256InitialState,
LAC_HASH_SHA3_256_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t sha3_384Info =
{ LAC_HASH_SHA3_384_DIGEST_SIZE,
LAC_HASH_SHA3_384_BLOCK_SIZE,
sha3_384InitialState,
LAC_HASH_SHA3_384_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t sha3_512Info =
{ LAC_HASH_SHA3_512_DIGEST_SIZE,
LAC_HASH_SHA3_512_BLOCK_SIZE,
sha3_512InitialState,
LAC_HASH_SHA3_512_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t polyInfo = { LAC_HASH_POLY_DIGEST_SIZE,
LAC_HASH_POLY_BLOCK_SIZE,
NULL, /* intial state */
LAC_HASH_POLY_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t shake_128Info =
{ LAC_HASH_SHAKE_128_DIGEST_SIZE, LAC_HASH_SHAKE_128_BLOCK_SIZE, NULL, 0 };
static lac_sym_qat_hash_alg_info_t shake_256Info =
{ LAC_HASH_SHAKE_256_DIGEST_SIZE, LAC_HASH_SHAKE_256_BLOCK_SIZE, NULL, 0 };
static lac_sym_qat_hash_alg_info_t sm3Info = { LAC_HASH_SM3_DIGEST_SIZE,
LAC_HASH_SM3_BLOCK_SIZE,
sm3InitialState,
LAC_HASH_SM3_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t xcbcMacInfo =
{ LAC_HASH_XCBC_MAC_128_DIGEST_SIZE,
LAC_HASH_XCBC_MAC_BLOCK_SIZE,
aesXcbcKeySeed,
LAC_SYM_QAT_XCBC_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t aesCmacInfo =
{ LAC_HASH_CMAC_128_DIGEST_SIZE,
LAC_HASH_CMAC_BLOCK_SIZE,
aesCmacKeySeed,
LAC_SYM_QAT_CMAC_STATE_SIZE };
static lac_sym_qat_hash_alg_info_t aesCcmInfo = {
LAC_HASH_AES_CCM_DIGEST_SIZE,
LAC_HASH_AES_CCM_BLOCK_SIZE,
NULL, /* intial state */
0 /* state size */
};
static lac_sym_qat_hash_alg_info_t aesGcmInfo = {
LAC_HASH_AES_GCM_DIGEST_SIZE,
LAC_HASH_AES_GCM_BLOCK_SIZE,
NULL, /* initial state */
0 /* state size */
};
static lac_sym_qat_hash_alg_info_t kasumiF9Info = {
LAC_HASH_KASUMI_F9_DIGEST_SIZE,
LAC_HASH_KASUMI_F9_BLOCK_SIZE,
NULL, /* initial state */
0 /* state size */
};
static lac_sym_qat_hash_alg_info_t snow3gUia2Info = {
LAC_HASH_SNOW3G_UIA2_DIGEST_SIZE,
LAC_HASH_SNOW3G_UIA2_BLOCK_SIZE,
NULL, /* initial state */
0 /* state size */
};
static lac_sym_qat_hash_alg_info_t aesCbcMacInfo =
{ LAC_HASH_AES_CBC_MAC_DIGEST_SIZE,
LAC_HASH_AES_CBC_MAC_BLOCK_SIZE,
NULL,
0 };
static lac_sym_qat_hash_alg_info_t zucEia3Info = {
LAC_HASH_ZUC_EIA3_DIGEST_SIZE,
LAC_HASH_ZUC_EIA3_BLOCK_SIZE,
NULL, /* initial state */
0 /* state size */
};
/* Hash QAT specific structures */
static lac_sym_qat_hash_qat_info_t md5Config = { ICP_QAT_HW_AUTH_ALGO_MD5,
LAC_HASH_MD5_BLOCK_SIZE,
ICP_QAT_HW_MD5_STATE1_SZ,
ICP_QAT_HW_MD5_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t sha1Config = { ICP_QAT_HW_AUTH_ALGO_SHA1,
LAC_HASH_SHA1_BLOCK_SIZE,
ICP_QAT_HW_SHA1_STATE1_SZ,
ICP_QAT_HW_SHA1_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t sha224Config =
{ ICP_QAT_HW_AUTH_ALGO_SHA224,
LAC_HASH_SHA224_BLOCK_SIZE,
ICP_QAT_HW_SHA224_STATE1_SZ,
ICP_QAT_HW_SHA224_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t sha256Config =
{ ICP_QAT_HW_AUTH_ALGO_SHA256,
LAC_HASH_SHA256_BLOCK_SIZE,
ICP_QAT_HW_SHA256_STATE1_SZ,
ICP_QAT_HW_SHA256_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t sha384Config =
{ ICP_QAT_HW_AUTH_ALGO_SHA384,
LAC_HASH_SHA384_BLOCK_SIZE,
ICP_QAT_HW_SHA384_STATE1_SZ,
ICP_QAT_HW_SHA384_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t sha512Config =
{ ICP_QAT_HW_AUTH_ALGO_SHA512,
LAC_HASH_SHA512_BLOCK_SIZE,
ICP_QAT_HW_SHA512_STATE1_SZ,
ICP_QAT_HW_SHA512_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t sha3_224Config =
{ ICP_QAT_HW_AUTH_ALGO_SHA3_224,
LAC_HASH_SHA3_224_BLOCK_SIZE,
ICP_QAT_HW_SHA3_224_STATE1_SZ,
ICP_QAT_HW_SHA3_224_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t sha3_256Config =
{ ICP_QAT_HW_AUTH_ALGO_SHA3_256,
LAC_HASH_SHA3_256_BLOCK_SIZE,
ICP_QAT_HW_SHA3_256_STATE1_SZ,
ICP_QAT_HW_SHA3_256_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t sha3_384Config =
{ ICP_QAT_HW_AUTH_ALGO_SHA3_384,
LAC_HASH_SHA3_384_BLOCK_SIZE,
ICP_QAT_HW_SHA3_384_STATE1_SZ,
ICP_QAT_HW_SHA3_384_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t sha3_512Config =
{ ICP_QAT_HW_AUTH_ALGO_SHA3_512,
LAC_HASH_SHA3_512_BLOCK_SIZE,
ICP_QAT_HW_SHA3_512_STATE1_SZ,
ICP_QAT_HW_SHA3_512_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t shake_128Config =
{ ICP_QAT_HW_AUTH_ALGO_SHAKE_128, LAC_HASH_SHAKE_128_BLOCK_SIZE, 0, 0 };
static lac_sym_qat_hash_qat_info_t shake_256Config =
{ ICP_QAT_HW_AUTH_ALGO_SHAKE_256, LAC_HASH_SHAKE_256_BLOCK_SIZE, 0, 0 };
static lac_sym_qat_hash_qat_info_t polyConfig = { ICP_QAT_HW_AUTH_ALGO_POLY,
LAC_HASH_POLY_BLOCK_SIZE,
0,
0 };
static lac_sym_qat_hash_qat_info_t sm3Config = { ICP_QAT_HW_AUTH_ALGO_SM3,
LAC_HASH_SM3_BLOCK_SIZE,
ICP_QAT_HW_SM3_STATE1_SZ,
ICP_QAT_HW_SM3_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t xcbcMacConfig =
{ ICP_QAT_HW_AUTH_ALGO_AES_XCBC_MAC,
0,
ICP_QAT_HW_AES_XCBC_MAC_STATE1_SZ,
LAC_SYM_QAT_XCBC_STATE_SIZE };
static lac_sym_qat_hash_qat_info_t aesCmacConfig =
{ ICP_QAT_HW_AUTH_ALGO_AES_XCBC_MAC,
0,
ICP_QAT_HW_AES_XCBC_MAC_STATE1_SZ,
LAC_SYM_QAT_CMAC_STATE_SIZE };
static lac_sym_qat_hash_qat_info_t aesCcmConfig =
{ ICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC,
0,
ICP_QAT_HW_AES_CBC_MAC_STATE1_SZ,
ICP_QAT_HW_AES_CBC_MAC_KEY_SZ + ICP_QAT_HW_AES_CCM_CBC_E_CTR0_SZ };
static lac_sym_qat_hash_qat_info_t aesGcmConfig =
{ ICP_QAT_HW_AUTH_ALGO_GALOIS_128,
0,
ICP_QAT_HW_GALOIS_128_STATE1_SZ,
ICP_QAT_HW_GALOIS_H_SZ + ICP_QAT_HW_GALOIS_LEN_A_SZ +
ICP_QAT_HW_GALOIS_E_CTR0_SZ };
static lac_sym_qat_hash_qat_info_t kasumiF9Config =
{ ICP_QAT_HW_AUTH_ALGO_KASUMI_F9,
0,
ICP_QAT_HW_KASUMI_F9_STATE1_SZ,
ICP_QAT_HW_KASUMI_F9_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t snow3gUia2Config =
{ ICP_QAT_HW_AUTH_ALGO_SNOW_3G_UIA2,
0,
ICP_QAT_HW_SNOW_3G_UIA2_STATE1_SZ,
ICP_QAT_HW_SNOW_3G_UIA2_STATE2_SZ };
static lac_sym_qat_hash_qat_info_t aesCbcMacConfig =
{ ICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC,
0,
ICP_QAT_HW_AES_CBC_MAC_STATE1_SZ,
ICP_QAT_HW_AES_CBC_MAC_STATE1_SZ + ICP_QAT_HW_AES_CBC_MAC_STATE1_SZ };
static lac_sym_qat_hash_qat_info_t zucEia3Config =
{ ICP_QAT_HW_AUTH_ALGO_ZUC_3G_128_EIA3,
0,
ICP_QAT_HW_ZUC_3G_EIA3_STATE1_SZ,
ICP_QAT_HW_ZUC_3G_EIA3_STATE2_SZ };
/* Array of mappings between algorithm and info structure
* This array is used to populate the lookup table */
static lac_sym_qat_hash_def_map_t lacHashDefsMapping[] =
{ { CPA_CY_SYM_HASH_MD5, { &md5Info, &md5Config } },
{ CPA_CY_SYM_HASH_SHA1, { &sha1Info, &sha1Config } },
{ CPA_CY_SYM_HASH_SHA224, { &sha224Info, &sha224Config } },
{ CPA_CY_SYM_HASH_SHA256, { &sha256Info, &sha256Config } },
{ CPA_CY_SYM_HASH_SHA384, { &sha384Info, &sha384Config } },
{ CPA_CY_SYM_HASH_SHA512, { &sha512Info, &sha512Config } },
{ CPA_CY_SYM_HASH_SHA3_224, { &sha3_224Info, &sha3_224Config } },
{ CPA_CY_SYM_HASH_SHA3_256, { &sha3_256Info, &sha3_256Config } },
{ CPA_CY_SYM_HASH_SHA3_384, { &sha3_384Info, &sha3_384Config } },
{ CPA_CY_SYM_HASH_SHA3_512, { &sha3_512Info, &sha3_512Config } },
{ CPA_CY_SYM_HASH_SHAKE_128, { &shake_128Info, &shake_128Config } },
{ CPA_CY_SYM_HASH_SHAKE_256, { &shake_256Info, &shake_256Config } },
{ CPA_CY_SYM_HASH_POLY, { &polyInfo, &polyConfig } },
{ CPA_CY_SYM_HASH_SM3, { &sm3Info, &sm3Config } },
{ CPA_CY_SYM_HASH_AES_XCBC, { &xcbcMacInfo, &xcbcMacConfig } },
{ CPA_CY_SYM_HASH_AES_CMAC, { &aesCmacInfo, &aesCmacConfig } },
{ CPA_CY_SYM_HASH_AES_CCM, { &aesCcmInfo, &aesCcmConfig } },
{ CPA_CY_SYM_HASH_AES_GCM, { &aesGcmInfo, &aesGcmConfig } },
{ CPA_CY_SYM_HASH_KASUMI_F9, { &kasumiF9Info, &kasumiF9Config } },
{ CPA_CY_SYM_HASH_SNOW3G_UIA2, { &snow3gUia2Info, &snow3gUia2Config } },
{ CPA_CY_SYM_HASH_AES_GMAC, { &aesGcmInfo, &aesGcmConfig } },
{ CPA_CY_SYM_HASH_ZUC_EIA3, { &zucEia3Info, &zucEia3Config } },
{ CPA_CY_SYM_HASH_AES_CBC_MAC, { &aesCbcMacInfo, &aesCbcMacConfig } } };
/*
* LacSymQat_HashLookupInit
*/
CpaStatus
LacSymQat_HashLookupInit(CpaInstanceHandle instanceHandle)
{
Cpa32U entry = 0;
Cpa32U numEntries = 0;
Cpa32U arraySize = 0;
CpaStatus status = CPA_STATUS_SUCCESS;
CpaCySymHashAlgorithm hashAlg = CPA_CY_SYM_HASH_NONE;
sal_crypto_service_t *pService = (sal_crypto_service_t *)instanceHandle;
arraySize =
(CPA_CY_HASH_ALG_END + 1) * sizeof(lac_sym_qat_hash_defs_t *);
/* Size round up for performance */
arraySize = LAC_ALIGN_POW2_ROUNDUP(arraySize, LAC_64BYTE_ALIGNMENT);
pService->pLacHashLookupDefs = LAC_OS_MALLOC(arraySize);
if (NULL != pService->pLacHashLookupDefs) {
LAC_OS_BZERO(pService->pLacHashLookupDefs, arraySize);
numEntries = sizeof(lacHashDefsMapping) /
sizeof(lac_sym_qat_hash_def_map_t);
/* initialise the hash lookup definitions table so that the
* algorithm
* can be used to index into the table */
for (entry = 0; entry < numEntries; entry++) {
hashAlg = lacHashDefsMapping[entry].hashAlgorithm;
pService->pLacHashLookupDefs[hashAlg] =
&(lacHashDefsMapping[entry].hashDefs);
}
} else {
status = CPA_STATUS_RESOURCE;
}
return status;
}
/*
* LacSymQat_HashAlgLookupGet
*/
void
LacSymQat_HashAlgLookupGet(CpaInstanceHandle instanceHandle,
CpaCySymHashAlgorithm hashAlgorithm,
lac_sym_qat_hash_alg_info_t **ppHashAlgInfo)
{
sal_crypto_service_t *pService = (sal_crypto_service_t *)instanceHandle;
*ppHashAlgInfo = pService->pLacHashLookupDefs[hashAlgorithm]->algInfo;
}
/*
* LacSymQat_HashDefsLookupGet
*/
void
LacSymQat_HashDefsLookupGet(CpaInstanceHandle instanceHandle,
CpaCySymHashAlgorithm hashAlgorithm,
lac_sym_qat_hash_defs_t **ppHashDefsInfo)
{
sal_crypto_service_t *pService = (sal_crypto_service_t *)instanceHandle;
*ppHashDefsInfo = pService->pLacHashLookupDefs[hashAlgorithm];
}

196
sys/dev/qat/qat_api/common/crypto/sym/qat/lac_sym_qat_key.c Normal file
View File

@ -0,0 +1,196 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file lac_sym_qat_key.c Interfaces for populating the symmetric qat key
* structures
*
* @ingroup LacSymQatKey
*
*****************************************************************************/
#include "cpa.h"
#include "cpa_cy_key.h"
#include "lac_mem.h"
#include "icp_qat_fw_la.h"
#include "icp_accel_devices.h"
#include "icp_adf_debug.h"
#include "lac_list.h"
#include "lac_sal_types.h"
#include "lac_sym_qat_key.h"
#include "lac_sym_hash_defs.h"
void
LacSymQat_KeySslRequestPopulate(icp_qat_la_bulk_req_hdr_t *pKeyGenReqHdr,
icp_qat_fw_la_key_gen_common_t *pKeyGenReqMid,
Cpa32U generatedKeyLenInBytes,
Cpa32U labelLenInBytes,
Cpa32U secretLenInBytes,
Cpa32U iterations)
{
/* Rounded to nearest 8 byte boundary */
Cpa8U outLenRounded = 0;
outLenRounded = LAC_ALIGN_POW2_ROUNDUP(generatedKeyLenInBytes,
LAC_QUAD_WORD_IN_BYTES);
pKeyGenReqMid->u.secret_lgth_ssl = secretLenInBytes;
pKeyGenReqMid->u1.s1.output_lgth_ssl = outLenRounded;
pKeyGenReqMid->u1.s1.label_lgth_ssl = labelLenInBytes;
pKeyGenReqMid->u2.iter_count = iterations;
pKeyGenReqMid->u3.resrvd2 = 0;
pKeyGenReqMid->resrvd3 = 0;
/* Set up the common LA flags */
pKeyGenReqHdr->comn_hdr.service_cmd_id =
ICP_QAT_FW_LA_CMD_SSL3_KEY_DERIVE;
pKeyGenReqHdr->comn_hdr.resrvd1 = 0;
}
void
LacSymQat_KeyTlsRequestPopulate(
icp_qat_fw_la_key_gen_common_t *pKeyGenReqParams,
Cpa32U generatedKeyLenInBytes,
Cpa32U labelInfo, /* Generic name, can be num of labels or label length */
Cpa32U secretLenInBytes,
Cpa8U seedLenInBytes,
icp_qat_fw_la_cmd_id_t cmdId)
{
pKeyGenReqParams->u1.s3.output_lgth_tls =
LAC_ALIGN_POW2_ROUNDUP(generatedKeyLenInBytes,
LAC_QUAD_WORD_IN_BYTES);
/* For TLS u param of auth_req_params is set to secretLen */
pKeyGenReqParams->u.secret_lgth_tls = secretLenInBytes;
switch (cmdId) {
case ICP_QAT_FW_LA_CMD_HKDF_EXTRACT:
pKeyGenReqParams->u2.hkdf_ikm_length = secretLenInBytes;
pKeyGenReqParams->u3.resrvd2 = 0;
break;
case ICP_QAT_FW_LA_CMD_HKDF_EXPAND:
pKeyGenReqParams->u1.hkdf.info_length = labelInfo;
break;
case ICP_QAT_FW_LA_CMD_HKDF_EXTRACT_AND_EXPAND:
pKeyGenReqParams->u2.hkdf_ikm_length = secretLenInBytes;
pKeyGenReqParams->u1.hkdf.info_length = labelInfo;
break;
case ICP_QAT_FW_LA_CMD_HKDF_EXPAND_LABEL:
/* Num of Labels */
pKeyGenReqParams->u1.hkdf_label.num_labels = labelInfo;
pKeyGenReqParams->u3.hkdf_num_sublabels = 4; /* 4 subLabels */
break;
case ICP_QAT_FW_LA_CMD_HKDF_EXTRACT_AND_EXPAND_LABEL:
pKeyGenReqParams->u2.hkdf_ikm_length = secretLenInBytes;
/* Num of Labels */
pKeyGenReqParams->u1.hkdf_label.num_labels = labelInfo;
pKeyGenReqParams->u3.hkdf_num_sublabels = 4; /* 4 subLabels */
break;
default:
pKeyGenReqParams->u1.s3.label_lgth_tls = labelInfo;
pKeyGenReqParams->u2.tls_seed_length = seedLenInBytes;
pKeyGenReqParams->u3.resrvd2 = 0;
break;
}
pKeyGenReqParams->resrvd3 = 0;
}
void
LacSymQat_KeyMgfRequestPopulate(icp_qat_la_bulk_req_hdr_t *pKeyGenReqHdr,
icp_qat_fw_la_key_gen_common_t *pKeyGenReqMid,
Cpa8U seedLenInBytes,
Cpa16U maskLenInBytes,
Cpa8U hashLenInBytes)
{
pKeyGenReqHdr->comn_hdr.service_cmd_id = ICP_QAT_FW_LA_CMD_MGF1;
pKeyGenReqMid->u.mask_length =
LAC_ALIGN_POW2_ROUNDUP(maskLenInBytes, LAC_QUAD_WORD_IN_BYTES);
pKeyGenReqMid->u1.s2.hash_length = hashLenInBytes;
pKeyGenReqMid->u1.s2.seed_length = seedLenInBytes;
}
void
LacSymQat_KeySslKeyMaterialInputPopulate(
sal_service_t *pService,
icp_qat_fw_la_ssl_key_material_input_t *pSslKeyMaterialInput,
void *pSeed,
Cpa64U labelPhysAddr,
void *pSecret)
{
LAC_MEM_SHARED_WRITE_VIRT_TO_PHYS_PTR_EXTERNAL(
(*pService), pSslKeyMaterialInput->seed_addr, pSeed);
pSslKeyMaterialInput->label_addr = labelPhysAddr;
LAC_MEM_SHARED_WRITE_VIRT_TO_PHYS_PTR_EXTERNAL(
(*pService), pSslKeyMaterialInput->secret_addr, pSecret);
}
void
LacSymQat_KeyTlsKeyMaterialInputPopulate(
sal_service_t *pService,
icp_qat_fw_la_tls_key_material_input_t *pTlsKeyMaterialInput,
void *pSeed,
Cpa64U labelPhysAddr)
{
LAC_MEM_SHARED_WRITE_VIRT_TO_PHYS_PTR_EXTERNAL(
(*pService), pTlsKeyMaterialInput->seed_addr, pSeed);
pTlsKeyMaterialInput->label_addr = labelPhysAddr;
}
void
LacSymQat_KeyTlsHKDFKeyMaterialInputPopulate(
sal_service_t *pService,
icp_qat_fw_la_hkdf_key_material_input_t *pTlsKeyMaterialInput,
CpaCyKeyGenHKDFOpData *pKeyGenTlsOpData,
Cpa64U subLabelsPhysAddr,
icp_qat_fw_la_cmd_id_t cmdId)
{
switch (cmdId) {
case ICP_QAT_FW_LA_CMD_HKDF_EXTRACT:
LAC_MEM_SHARED_WRITE_VIRT_TO_PHYS_PTR_EXTERNAL(
(*pService),
pTlsKeyMaterialInput->ikm_addr,
pKeyGenTlsOpData->secret);
break;
case ICP_QAT_FW_LA_CMD_HKDF_EXPAND:
LAC_MEM_SHARED_WRITE_VIRT_TO_PHYS_PTR_EXTERNAL(
(*pService),
pTlsKeyMaterialInput->labels_addr,
pKeyGenTlsOpData->info);
break;
case ICP_QAT_FW_LA_CMD_HKDF_EXTRACT_AND_EXPAND:
LAC_MEM_SHARED_WRITE_VIRT_TO_PHYS_PTR_EXTERNAL(
(*pService),
pTlsKeyMaterialInput->ikm_addr,
pKeyGenTlsOpData->secret);
pTlsKeyMaterialInput->labels_addr =
pTlsKeyMaterialInput->ikm_addr +
((uint64_t)&pKeyGenTlsOpData->info -
(uint64_t)&pKeyGenTlsOpData->secret);
break;
case ICP_QAT_FW_LA_CMD_HKDF_EXPAND_LABEL:
pTlsKeyMaterialInput->sublabels_addr = subLabelsPhysAddr;
LAC_MEM_SHARED_WRITE_VIRT_TO_PHYS_PTR_EXTERNAL(
(*pService),
pTlsKeyMaterialInput->labels_addr,
pKeyGenTlsOpData->label);
break;
case ICP_QAT_FW_LA_CMD_HKDF_EXTRACT_AND_EXPAND_LABEL:
pTlsKeyMaterialInput->sublabels_addr = subLabelsPhysAddr;
LAC_MEM_SHARED_WRITE_VIRT_TO_PHYS_PTR_EXTERNAL(
(*pService),
pTlsKeyMaterialInput->ikm_addr,
pKeyGenTlsOpData->secret);
pTlsKeyMaterialInput->labels_addr =
pTlsKeyMaterialInput->ikm_addr +
((uint64_t)&pKeyGenTlsOpData->label -
(uint64_t)&pKeyGenTlsOpData->secret);
break;
default:
break;
}
}

1554
sys/dev/qat/qat_api/common/ctrl/sal_compression.c Normal file
View File

File diff suppressed because it is too large Load Diff

105
sys/dev/qat/qat_api/common/ctrl/sal_create_services.c Normal file
View File

@ -0,0 +1,105 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file sal_create_services.c
*
* @defgroup SalCtrl Service Access Layer Controller
*
* @ingroup SalCtrl
*
* @description
* This file contains the main function to create a specific service.
*
*****************************************************************************/
#include "cpa.h"
#include "lac_mem.h"
#include "lac_mem_pools.h"
#include "qat_utils.h"
#include "lac_list.h"
#include "icp_adf_transport.h"
#include "icp_accel_devices.h"
#include "icp_adf_debug.h"
#include "icp_qat_fw_la.h"
#include "lac_sym_qat.h"
#include "sal_types_compression.h"
#include "lac_sal_types_crypto.h"
#include "icp_adf_init.h"
#include "lac_sal.h"
#include "lac_sal_ctrl.h"
CpaStatus
SalCtrl_ServiceCreate(sal_service_type_t serviceType,
Cpa32U instance,
sal_service_t **ppInst)
{
sal_crypto_service_t *pCrypto_service = NULL;
sal_compression_service_t *pCompression_service = NULL;
switch ((sal_service_type_t)serviceType) {
case SAL_SERVICE_TYPE_CRYPTO_ASYM:
case SAL_SERVICE_TYPE_CRYPTO_SYM:
case SAL_SERVICE_TYPE_CRYPTO: {
pCrypto_service =
malloc(sizeof(sal_crypto_service_t), M_QAT, M_WAITOK);
/* Zero memory */
memset(pCrypto_service, 0, sizeof(sal_crypto_service_t));
pCrypto_service->generic_service_info.type =
(sal_service_type_t)serviceType;
pCrypto_service->generic_service_info.state =
SAL_SERVICE_STATE_UNINITIALIZED;
pCrypto_service->generic_service_info.instance = instance;
pCrypto_service->generic_service_info.init = SalCtrl_CryptoInit;
pCrypto_service->generic_service_info.start =
SalCtrl_CryptoStart;
pCrypto_service->generic_service_info.stop = SalCtrl_CryptoStop;
pCrypto_service->generic_service_info.shutdown =
SalCtrl_CryptoShutdown;
*(ppInst) = &(pCrypto_service->generic_service_info);
return CPA_STATUS_SUCCESS;
}
case SAL_SERVICE_TYPE_COMPRESSION: {
pCompression_service =
malloc(sizeof(sal_compression_service_t), M_QAT, M_WAITOK);
/* Zero memory */
memset(pCompression_service,
0,
sizeof(sal_compression_service_t));
pCompression_service->generic_service_info.type =
(sal_service_type_t)serviceType;
pCompression_service->generic_service_info.state =
SAL_SERVICE_STATE_UNINITIALIZED;
pCompression_service->generic_service_info.instance = instance;
pCompression_service->generic_service_info.init =
SalCtrl_CompressionInit;
pCompression_service->generic_service_info.start =
SalCtrl_CompressionStart;
pCompression_service->generic_service_info.stop =
SalCtrl_CompressionStop;
pCompression_service->generic_service_info.shutdown =
SalCtrl_CompressionShutdown;
*(ppInst) = &(pCompression_service->generic_service_info);
return CPA_STATUS_SUCCESS;
}
default: {
QAT_UTILS_LOG("Not a valid service type\n");
(*ppInst) = NULL;
return CPA_STATUS_FAIL;
}
}
}

1837
sys/dev/qat/qat_api/common/ctrl/sal_crypto.c Normal file
View File

File diff suppressed because it is too large Load Diff

1344
sys/dev/qat/qat_api/common/ctrl/sal_ctrl_services.c Normal file
View File

File diff suppressed because it is too large Load Diff

99
sys/dev/qat/qat_api/common/ctrl/sal_list.c Normal file
View File

@ -0,0 +1,99 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file sal_list.c
*
* @ingroup SalCtrl
*
* List implementations for SAL
*
*****************************************************************************/
#include "lac_mem.h"
#include "lac_list.h"
CpaStatus
SalList_add(sal_list_t **list, sal_list_t **tail, void *pObj)
{
sal_list_t *new_element = NULL;
if (NULL == *list) {
/* First element in list */
*list = malloc(sizeof(sal_list_t), M_QAT, M_WAITOK);
(*list)->next = NULL;
(*list)->pObj = pObj;
*tail = *list;
} else {
/* add to tail of the list */
new_element = malloc(sizeof(sal_list_t), M_QAT, M_WAITOK);
new_element->pObj = pObj;
new_element->next = NULL;
(*tail)->next = new_element;
*tail = new_element;
}
return CPA_STATUS_SUCCESS;
}
void *
SalList_getObject(sal_list_t *list)
{
if (list == NULL) {
return NULL;
}
return list->pObj;
}
void
SalList_delObject(sal_list_t **list)
{
if (*list == NULL) {
return;
}
(*list)->pObj = NULL;
return;
}
void *
SalList_next(sal_list_t *list)
{
return list->next;
}
void
SalList_free(sal_list_t **list)
{
sal_list_t *next_element = NULL;
void *pObj = NULL;
while (NULL != (*list)) {
next_element = SalList_next(*list);
pObj = SalList_getObject((*list));
LAC_OS_FREE(pObj);
LAC_OS_FREE(*list);
*list = next_element;
}
}
void
SalList_del(sal_list_t **head_list, sal_list_t **pre_list, sal_list_t *list)
{
void *pObj = NULL;
if ((NULL == *head_list) || (NULL == *pre_list) || (NULL == list)) {
return;
}
if (*head_list == list) { /* delete the first node in list */
*head_list = list->next;
} else {
(*pre_list)->next = list->next;
}
pObj = SalList_getObject(list);
LAC_OS_FREE(pObj);
LAC_OS_FREE(list);
return;
}

252
sys/dev/qat/qat_api/common/include/lac_buffer_desc.h Normal file
View File

@ -0,0 +1,252 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
***************************************************************************
* @file lac_buffer_desc.h
*
* @defgroup LacBufferDesc Buffer Descriptors
*
* @ingroup LacCommon
*
* Functions which handle updating a user supplied buffer with the QAT
* descriptor representation.
*
***************************************************************************/
/***************************************************************************/
#ifndef LAC_BUFFER_DESC_H
#define LAC_BUFFER_DESC_H
/***************************************************************************
* Include header files
***************************************************************************/
#include "cpa.h"
#include "icp_buffer_desc.h"
#include "cpa_cy_sym.h"
#include "lac_common.h"
/**
*******************************************************************************
* @ingroup LacBufferDesc
* Write the buffer descriptor in QAT friendly format.
*
* @description
* Updates the Meta Data associated with the pUserBufferList CpaBufferList
* This function will also return the (aligned) physical address
* associated with this CpaBufferList.
*
* @param[in] pUserBufferList A pointer to the buffer list to
* create the meta data for the QAT.
* @param[out] pBufferListAlignedPhyAddr The pointer to the aligned physical
* address.
* @param[in] isPhysicalAddress Type of address
* @param[in] pService Pointer to generic service
*
*****************************************************************************/
CpaStatus LacBuffDesc_BufferListDescWrite(const CpaBufferList *pUserBufferList,
Cpa64U *pBufferListAlignedPhyAddr,
CpaBoolean isPhysicalAddress,
sal_service_t *pService);
/**
*******************************************************************************
* @ingroup LacBufferDesc
* Write the buffer descriptor in QAT friendly format.
*
* @description
* Updates the Meta Data associated with the pUserBufferList CpaBufferList
* This function will also return the (aligned) physical address
* associated with this CpaBufferList. Zero length buffers are allowed.
* Should be used for CHA-CHA-POLY and GCM algorithms.
*
* @param[in] pUserBufferList A pointer to the buffer list to
* create the meta data for the QAT.
* @param[out] pBufferListAlignedPhyAddr The pointer to the aligned physical
* address.
* @param[in] isPhysicalAddress Type of address
* @param[in] pService Pointer to generic service
*
*****************************************************************************/
CpaStatus LacBuffDesc_BufferListDescWriteAndAllowZeroBuffer(
const CpaBufferList *pUserBufferList,
Cpa64U *pBufferListAlignedPhyAddr,
CpaBoolean isPhysicalAddress,
sal_service_t *pService);
/**
*******************************************************************************
* @ingroup LacBufferDesc
* Write the buffer descriptor in QAT friendly format.
*
* @description
* Updates the Meta Data associated with the PClientList CpaBufferList
* This function will also return the (aligned) physical address
* associated with this CpaBufferList and the total data length of the
* buffer list.
*
* @param[in] pUserBufferList A pointer to the buffer list to
* create the meta data for the QAT.
* @param[out] pBufListAlignedPhyAddr The pointer to the aligned physical
* address.
* @param[in] isPhysicalAddress Type of address
* @param[out] totalDataLenInBytes The pointer to the total data length
* of the buffer list
* @param[in] pService Pointer to generic service
*
*****************************************************************************/
CpaStatus
LacBuffDesc_BufferListDescWriteAndGetSize(const CpaBufferList *pUserBufferList,
Cpa64U *pBufListAlignedPhyAddr,
CpaBoolean isPhysicalAddress,
Cpa64U *totalDataLenInBytes,
sal_service_t *pService);
/**
*******************************************************************************
* @ingroup LacBufferDesc
* Ensure the CpaFlatBuffer is correctly formatted.
*
* @description
* Ensures the CpaFlatBuffer is correctly formatted
* This function will also return the total size of the buffers
* in the scatter gather list.
*
* @param[in] pUserFlatBuffer A pointer to the flat buffer to
* validate.
* @param[out] pPktSize The total size of the packet.
* @param[in] alignmentShiftExpected The expected alignment shift of each
* of the elements of the scatter gather
*
* @retval CPA_STATUS_INVALID_PARAM BufferList failed checks
* @retval CPA_STATUS_SUCCESS Function executed successfully
*
*****************************************************************************/
CpaStatus
LacBuffDesc_FlatBufferVerify(const CpaFlatBuffer *pUserFlatBuffer,
Cpa64U *pPktSize,
lac_aligment_shift_t alignmentShiftExpected);
/**
*******************************************************************************
* @ingroup LacBufferDesc
* Ensure the CpaFlatBuffer is correctly formatted.
* This function will allow a size of zero bytes to any of the Flat
* buffers.
*
* @description
* Ensures the CpaFlatBuffer is correctly formatted
* This function will also return the total size of the buffers
* in the scatter gather list.
*
* @param[in] pUserFlatBuffer A pointer to the flat buffer to
* validate.
* @param[out] pPktSize The total size of the packet.
* @param[in] alignmentShiftExpected The expected alignment shift of each
* of the elements of the scatter gather
*
* @retval CPA_STATUS_INVALID_PARAM BufferList failed checks
* @retval CPA_STATUS_SUCCESS Function executed successfully
*
*****************************************************************************/
CpaStatus
LacBuffDesc_FlatBufferVerifyNull(const CpaFlatBuffer *pUserFlatBuffer,
Cpa64U *pPktSize,
lac_aligment_shift_t alignmentShiftExpected);
/**
*******************************************************************************
* @ingroup LacBufferDesc
* Ensure the CpaBufferList is correctly formatted.
*
* @description
* Ensures the CpaBufferList pUserBufferList is correctly formatted
* including the user supplied metaData.
* This function will also return the total size of the buffers
* in the scatter gather list.
*
* @param[in] pUserBufferList A pointer to the buffer list to
* validate.
* @param[out] pPktSize The total size of the buffers in the
* scatter gather list.
* @param[in] alignmentShiftExpected The expected alignment shift of each
* of the elements of the scatter gather
* list.
* @retval CPA_STATUS_INVALID_PARAM BufferList failed checks
* @retval CPA_STATUS_SUCCESS Function executed successfully
*
*****************************************************************************/
CpaStatus
LacBuffDesc_BufferListVerify(const CpaBufferList *pUserBufferList,
Cpa64U *pPktSize,
lac_aligment_shift_t alignmentShiftExpected);
/**
*******************************************************************************
* @ingroup LacBufferDesc
* Ensure the CpaBufferList is correctly formatted.
*
* @description
* Ensures the CpaBufferList pUserBufferList is correctly formatted
* including the user supplied metaData.
* This function will also return the total size of the buffers
* in the scatter gather list.
*
* @param[in] pUserBufferList A pointer to the buffer list to
* validate.
* @param[out] pPktSize The total size of the buffers in the
* scatter gather list.
* @param[in] alignmentShiftExpected The expected alignment shift of each
* of the elements of the scatter gather
* list.
* @retval CPA_STATUS_INVALID_PARAM BufferList failed checks
* @retval CPA_STATUS_SUCCESS Function executed successfully
*
*****************************************************************************/
CpaStatus
LacBuffDesc_BufferListVerifyNull(const CpaBufferList *pUserBufferList,
Cpa64U *pPktSize,
lac_aligment_shift_t alignmentShiftExpected);
/**
*******************************************************************************
* @ingroup LacBufferDesc
* Get the total size of a CpaBufferList.
*
* @description
* This function returns the total size of the buffers
* in the scatter gather list.
*
* @param[in] pUserBufferList A pointer to the buffer list to
* calculate the total size for.
* @param[out] pPktSize The total size of the buffers in the
* scatter gather list.
*
*****************************************************************************/
void LacBuffDesc_BufferListTotalSizeGet(const CpaBufferList *pUserBufferList,
Cpa64U *pPktSize);
/**
*******************************************************************************
* @ingroup LacBufferDesc
* Zero some of the CpaBufferList.
*
* @description
* Zero a section of data within the CpaBufferList from an offset for
* a specific length.
*
* @param[in] pBuffList A pointer to the buffer list to
* zero an area of.
* @param[in] offset Number of bytes from start of buffer to where
* to start zeroing.
*
* @param[in] lenToZero Number of bytes that will be set to zero
* after the call to this function.
*****************************************************************************/
void LacBuffDesc_BufferListZeroFromOffset(CpaBufferList *pBuffList,
Cpa32U offset,
Cpa32U lenToZero);
#endif /* LAC_BUFFER_DESC_H */

847
sys/dev/qat/qat_api/common/include/lac_common.h Normal file
View File

@ -0,0 +1,847 @@
/* SPDX-License-Identifier: BSD-3-Clause */
/* Copyright(c) 2007-2022 Intel Corporation */
/* $FreeBSD$ */
/**
*****************************************************************************
* @file lac_common.h Common macros
*
* @defgroup Lac Look Aside Crypto LLD Doc
*
*****************************************************************************/
/**
*****************************************************************************
* @defgroup LacCommon LAC Common
* Common code for Lac which includes init/shutdown, memory, logging and
* hooks.
*
* @ingroup Lac
*
*****************************************************************************/
/***************************************************************************/
#ifndef LAC_COMMON_H
#define LAC_COMMON_H
/*
******************************************************************************
* Include public/global header files
******************************************************************************
*/
#include "cpa.h"
#include "qat_utils.h"
#include "cpa_cy_common.h"
#include "icp_adf_init.h"
#define LAC_ARCH_UINT uintptr_t
#define LAC_ARCH_INT intptr_t
/*
*****************************************************************************
* Max range values for some primitive param checking
*****************************************************************************
*/
/**< Maximum number of instances */
#define SAL_MAX_NUM_INSTANCES_PER_DEV 512
#define SAL_DEFAULT_RING_SIZE 256
/**< Default ring size */
#define SAL_64_CONCURR_REQUESTS 64
#define SAL_128_CONCURR_REQUESTS 128
#define SAL_256_CONCURR_REQUESTS 256
#define SAL_512_CONCURR_REQUESTS 512
#define SAL_1024_CONCURR_REQUESTS 1024
#define SAL_2048_CONCURR_REQUESTS 2048
#define SAL_4096_CONCURR_REQUESTS 4096
#define SAL_MAX_CONCURR_REQUESTS 65536
/**< Valid options for the num of concurrent requests per ring pair read
from the config file. These values are used to size the rings */
#define SAL_BATCH_SUBMIT_FREE_SPACE 2
/**< For data plane batch submissions ADF leaves 2 spaces free on the ring */
/*
******************************************************************************
* Some common settings for QA API queries
******************************************************************************
*/
#define SAL_INFO2_VENDOR_NAME "Intel(R)"
/**< @ingroup LacCommon
* Name of vendor of this driver */
#define SAL_INFO2_PART_NAME "%s with Intel(R) QuickAssist Technology"
/**< @ingroup LacCommon
*/
/*
********************************************************************************
* User process name defines and functions
********************************************************************************
*/
#define LAC_USER_PROCESS_NAME_MAX_LEN 32
/**< @ingroup LacCommon
* Max length of user process name */
#define LAC_KERNEL_PROCESS_NAME "KERNEL_QAT"
/**< @ingroup LacCommon
* Default name for kernel process */
/*
********************************************************************************
* response mode indicator from Config file
********************************************************************************
*/
#define SAL_RESP_POLL_CFG_FILE 1
#define SAL_RESP_EPOLL_CFG_FILE 2
/*
* @ingroup LacCommon
* @description
* This function sets the process name
*
* @context
* This functions is called from module_init or from user space process
* initialization function
*
* @assumptions
* None
* @sideEffects
* None
* @reentrant
* No
* @threadSafe
* No
*
* param[in] processName Process name to be set
*/
CpaStatus icpSetProcessName(const char *processName);
/*
* @ingroup LacCommon
* @description
* This function gets the process name
*
* @context
* This functions is called from LAC context
*
* @assumptions
* None
* @sideEffects
* None
* @reentrant
* Yes
* @threadSafe
* Yes
*
*/
char *icpGetProcessName(void);
/* Sections of the config file */
#define LAC_CFG_SECTION_GENERAL "GENERAL"
#define LAC_CFG_SECTION_INTERNAL "INTERNAL"
/*
********************************************************************************
* Debug Macros and settings
********************************************************************************
*/
#define SEPARATOR "+--------------------------------------------------+\n"
/**< @ingroup LacCommon
* separator used for printing stats to standard output*/
#define BORDER "|"
/**< @ingroup LacCommon
* separator used for printing stats to standard output*/
/**
*****************************************************************************
* @ingroup LacCommon
* Component state
*
* @description
* This enum is used to indicate the state that the component is in. Its
* purpose is to prevent components from being initialised or shutdown
* incorrectly.
*
*****************************************************************************/
typedef enum {
LAC_COMP_SHUT_DOWN = 0,
/**< Component in the Shut Down state */
LAC_COMP_SHUTTING_DOWN,
/**< Component in the Process of Shutting down */
LAC_COMP_INITIALISING,
/**< Component in the Process of being initialised */
LAC_COMP_INITIALISED,
/**< Component in the initialised state */
} lac_comp_state_t;
/**
*******************************************************************************
* @ingroup LacCommon
* This macro checks if a parameter is NULL
*
* @param[in] param Parameter
*
* @return CPA_STATUS_INVALID_PARAM Parameter is NULL
* @return void Parameter is not NULL
******************************************************************************/
#define LAC_CHECK_NULL_PARAM(param) \
do { \
if (NULL == (param)) { \
return CPA_STATUS_INVALID_PARAM; \
} \
} while (0)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro checks if a parameter is within a specified range
*
* @param[in] param Parameter
* @param[in] min Parameter must be greater than OR equal to
*min
* @param[in] max Parameter must be less than max
*
* @return CPA_STATUS_INVALID_PARAM Parameter is outside range
* @return void Parameter is within range
******************************************************************************/
#define LAC_CHECK_PARAM_RANGE(param, min, max) \
do { \
if (((param) < (min)) || ((param) >= (max))) { \
return CPA_STATUS_INVALID_PARAM; \
} \
} while (0)
/**
*******************************************************************************
* @ingroup LacCommon
* This checks if a param is 8 byte aligned.
*
******************************************************************************/
#define LAC_CHECK_8_BYTE_ALIGNMENT(param) \
do { \
if ((Cpa64U)param % 8 != 0) { \
return CPA_STATUS_INVALID_PARAM; \
} \
} while (0)
/**
*******************************************************************************
* @ingroup LacCommon
* This checks if a param is 64 byte aligned.
*
******************************************************************************/
#define LAC_CHECK_64_BYTE_ALIGNMENT(param) \
do { \
if ((LAC_ARCH_UINT)param % 64 != 0) { \
return CPA_STATUS_INVALID_PARAM; \
} \
} while (0)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro returns the size of the buffer list structure given the
* number of elements in the buffer list - note: only the sizeof the
* buffer list structure is returned.
*
* @param[in] numBuffers The number of flatbuffers in a buffer list
*
* @return size of the buffer list structure
******************************************************************************/
#define LAC_BUFFER_LIST_SIZE_GET(numBuffers) \
(sizeof(CpaBufferList) + (numBuffers * sizeof(CpaFlatBuffer)))
/**
*******************************************************************************
* @ingroup LacCommon
* This macro checks that a flatbuffer is valid i.e. that it is not
* null and the data it points to is not null
*
* @param[in] pFlatBuffer Pointer to flatbuffer
*
* @return CPA_STATUS_INVALID_PARAM Invalid flatbuffer pointer
* @return void flatbuffer is ok
******************************************************************************/
#define LAC_CHECK_FLAT_BUFFER(pFlatBuffer) \
do { \
LAC_CHECK_NULL_PARAM((pFlatBuffer)); \
LAC_CHECK_NULL_PARAM((pFlatBuffer)->pData); \
} while (0)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro verifies that the status is ok i.e. equal to CPA_STATUS_SUCCESS
*
* @param[in] status status we are checking
*
* @return void status is ok (CPA_STATUS_SUCCESS)
* @return status The value in the status parameter is an error one
*
******************************************************************************/
#define LAC_CHECK_STATUS(status) \
do { \
if (CPA_STATUS_SUCCESS != (status)) { \
return status; \
} \
} while (0)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro verifies that the Instance Handle is valid.
*
* @param[in] instanceHandle Instance Handle
*
* @return CPA_STATUS_INVALID_PARAM Parameter is NULL
* @return void Parameter is not NULL
*
******************************************************************************/
#define LAC_CHECK_INSTANCE_HANDLE(instanceHandle) \
do { \
if (NULL == (instanceHandle)) { \
return CPA_STATUS_INVALID_PARAM; \
} \
} while (0)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro copies a string from one location to another
*
* @param[out] pDestinationBuffer Pointer to destination buffer
* @param[in] pSource Pointer to source buffer
*
******************************************************************************/
#define LAC_COPY_STRING(pDestinationBuffer, pSource) \
do { \
memcpy(pDestinationBuffer, pSource, (sizeof(pSource) - 1)); \
pDestinationBuffer[(sizeof(pSource) - 1)] = '\0'; \
} while (0)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro fills a memory zone with ZEROES
*
* @param[in] pBuffer Pointer to buffer
* @param[in] count Buffer length
*
* @return void
*
******************************************************************************/
#define LAC_OS_BZERO(pBuffer, count) memset(pBuffer, 0, count);
/**
*******************************************************************************
* @ingroup LacCommon
* This macro calculates the position of the given member in a struct
* Only for use on a struct where all members are of equal size to map
* the struct member position to an array index
*
* @param[in] structType the struct
* @param[in] member the member of the given struct
*
******************************************************************************/
#define LAC_IDX_OF(structType, member) \
(offsetof(structType, member) / sizeof(((structType *)0)->member))
/*
********************************************************************************
* Alignment, Bid define and Bit Operation Macros
********************************************************************************
*/
#define LAC_BIT31_SET 0x80000000 /**< bit 31 == 1 */
#define LAC_BIT7_SET 0x80 /**< bit 7 == 1 */
#define LAC_BIT6_SET 0x40 /**< bit 6 == 1 */
#define LAC_BIT5_SET 0x20 /**< bit 5 == 1 */
#define LAC_BIT4_SET 0x10 /**< bit 4 == 1 */
#define LAC_BIT3_SET 0x08 /**< bit 3 == 1 */
#define LAC_BIT2_SET 0x04 /**< bit 2 == 1 */
#define LAC_BIT1_SET 0x02 /**< bit 1 == 1 */
#define LAC_BIT0_SET 0x01 /**< bit 0 == 1 */
#define LAC_NUM_BITS_IN_BYTE (8)
/**< @ingroup LacCommon
* Number of bits in a byte */
#define LAC_LONG_WORD_IN_BYTES (4)
/**< @ingroup LacCommon
* Number of bytes in an IA word */
#define LAC_QUAD_WORD_IN_BYTES (8)
/**< @ingroup LacCommon
* Number of bytes in a QUAD word */
#define LAC_QAT_MAX_MSG_SZ_LW (32)
/**< @ingroup LacCommon
* Maximum size in Long Words for a QAT message */
/**
*****************************************************************************
* @ingroup LacCommon
* Alignment shift requirements of a buffer.
*
* @description
* This enum is used to indicate the alignment shift of a buffer.
* All alignments are to power of 2
*
*****************************************************************************/
typedef enum lac_aligment_shift_s {
LAC_NO_ALIGNMENT_SHIFT = 0,
/**< No alignment shift (to a power of 2)*/
LAC_8BYTE_ALIGNMENT_SHIFT = 3,
/**< 8 byte alignment shift (to a power of 2)*/
LAC_16BYTE_ALIGNMENT_SHIFT = 4,
/**< 16 byte alignment shift (to a power of 2)*/
LAC_64BYTE_ALIGNMENT_SHIFT = 6,
/**< 64 byte alignment shift (to a power of 2)*/
LAC_4KBYTE_ALIGNMENT_SHIFT = 12,
/**< 4k byte alignment shift (to a power of 2)*/
} lac_aligment_shift_t;
/**
*****************************************************************************
* @ingroup LacCommon
* Alignment of a buffer.
*
* @description
* This enum is used to indicate the alignment requirements of a buffer.
*
*****************************************************************************/
typedef enum lac_aligment_s {
LAC_NO_ALIGNMENT = 0,
/**< No alignment */
LAC_1BYTE_ALIGNMENT = 1,
/**< 1 byte alignment */
LAC_8BYTE_ALIGNMENT = 8,
/**< 8 byte alignment*/
LAC_64BYTE_ALIGNMENT = 64,
/**< 64 byte alignment*/
LAC_4KBYTE_ALIGNMENT = 4096,
/**< 4k byte alignment */
} lac_aligment_t;
/**
*****************************************************************************
* @ingroup LacCommon
* Size of a buffer.
*
* @description
* This enum is used to indicate the required size.
* The buffer must be a multiple of the required size.
*
*****************************************************************************/
typedef enum lac_expected_size_s {
LAC_NO_LENGTH_REQUIREMENTS = 0,
/**< No requirement for size */
LAC_4KBYTE_MULTIPLE_REQUIRED = 4096,
/**< 4k multiple requirement for size */
} lac_expected_size_t;
#define LAC_OPTIMAL_ALIGNMENT_SHIFT LAC_64BYTE_ALIGNMENT_SHIFT
/**< @ingroup LacCommon
* optimal alignment to a power of 2 */
#define LAC_SHIFT_8 (1 << LAC_8BYTE_ALIGNMENT_SHIFT)
/**< shift by 8 bits */
#define LAC_SHIFT_24 \
((1 << LAC_8BYTE_ALIGNMENT_SHIFT) + (1 << LAC_16BYTE_ALIGNMENT_SHIFT))
/**< shift by 24 bits */
#define LAC_MAX_16_BIT_VALUE ((1 << 16) - 1)
/**< @ingroup LacCommon
* maximum value a 16 bit type can hold */
/**
*******************************************************************************
* @ingroup LacCommon
* This macro can be used to avoid an unused variable warning from the
* compiler
*
* @param[in] variable unused variable
*
******************************************************************************/
#define LAC_UNUSED_VARIABLE(x) (void)(x)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro checks if an address is aligned to the specified power of 2
* Returns 0 if alignment is ok, or non-zero otherwise
*
* @param[in] address the address we are checking
*
* @param[in] alignment the byte alignment to check (specified as power of 2)
*
******************************************************************************/
#define LAC_ADDRESS_ALIGNED(address, alignment) \
(!((LAC_ARCH_UINT)(address) & ((1 << (alignment)) - 1)))
/**
*******************************************************************************
* @ingroup LacCommon
* This macro rounds up a number to a be a multiple of the alignment when
* the alignment is a power of 2.
*
* @param[in] num Number
* @param[in] align Alignment (must be a power of 2)
*
******************************************************************************/
#define LAC_ALIGN_POW2_ROUNDUP(num, align) (((num) + (align)-1) & ~((align)-1))
/**
*******************************************************************************
* @ingroup LacCommon
* This macro generates a bit mask to select a particular bit
*
* @param[in] bitPos Bit position to select
*
******************************************************************************/
#define LAC_BIT(bitPos) (0x1 << (bitPos))
/**
*******************************************************************************
* @ingroup LacCommon
* This macro converts a size in bits to the equivalent size in bytes,
* using a bit shift to divide by 8
*
* @param[in] x size in bits
*
******************************************************************************/
#define LAC_BITS_TO_BYTES(x) ((x) >> 3)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro converts a size in bytes to the equivalent size in bits,
* using a bit shift to multiply by 8
*
* @param[in] x size in bytes
*
******************************************************************************/
#define LAC_BYTES_TO_BITS(x) ((x) << 3)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro converts a size in bytes to the equivalent size in longwords,
* using a bit shift to divide by 4
*
* @param[in] x size in bytes
*
******************************************************************************/
#define LAC_BYTES_TO_LONGWORDS(x) ((x) >> 2)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro converts a size in longwords to the equivalent size in bytes,
* using a bit shift to multiply by 4
*
* @param[in] x size in long words
*
******************************************************************************/
#define LAC_LONGWORDS_TO_BYTES(x) ((x) << 2)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro converts a size in bytes to the equivalent size in quadwords,
* using a bit shift to divide by 8
*
* @param[in] x size in bytes
*
******************************************************************************/
#define LAC_BYTES_TO_QUADWORDS(x) (((x) >> 3) + (((x) % 8) ? 1 : 0))
/**
*******************************************************************************
* @ingroup LacCommon
* This macro converts a size in quadwords to the equivalent size in bytes,
* using a bit shift to multiply by 8
*
* @param[in] x size in quad words
*
******************************************************************************/
#define LAC_QUADWORDS_TO_BYTES(x) ((x) << 3)
/******************************************************************************/
/*
*******************************************************************************
* Mutex Macros
*******************************************************************************
*/
/**
*******************************************************************************
* @ingroup LacCommon
* This macro tries to acquire a mutex and returns the status
*
* @param[in] pLock Pointer to Lock
* @param[in] timeout Timeout
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_RESOURCE Error with Mutex
******************************************************************************/
#define LAC_LOCK_MUTEX(pLock, timeout) \
((CPA_STATUS_SUCCESS != qatUtilsMutexLock((pLock), (timeout))) ? \
CPA_STATUS_RESOURCE : \
CPA_STATUS_SUCCESS)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro unlocks a mutex and returns the status
*
* @param[in] pLock Pointer to Lock
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_RESOURCE Error with Mutex
******************************************************************************/
#define LAC_UNLOCK_MUTEX(pLock) \
((CPA_STATUS_SUCCESS != qatUtilsMutexUnlock((pLock))) ? \
CPA_STATUS_RESOURCE : \
CPA_STATUS_SUCCESS)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro initialises a mutex and returns the status
*
* @param[in] pLock Pointer to Lock
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_RESOURCE Error with Mutex
******************************************************************************/
#define LAC_INIT_MUTEX(pLock) \
((CPA_STATUS_SUCCESS != qatUtilsMutexInit((pLock))) ? \
CPA_STATUS_RESOURCE : \
CPA_STATUS_SUCCESS)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro destroys a mutex and returns the status
*
* @param[in] pLock Pointer to Lock
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_RESOURCE Error with Mutex
******************************************************************************/
#define LAC_DESTROY_MUTEX(pLock) \
((CPA_STATUS_SUCCESS != qatUtilsMutexDestroy((pLock))) ? \
CPA_STATUS_RESOURCE : \
CPA_STATUS_SUCCESS)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro calls a trylock on a mutex
*
* @param[in] pLock Pointer to Lock
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_RESOURCE Error with Mutex
******************************************************************************/
#define LAC_TRYLOCK_MUTEX(pLock) \
((CPA_STATUS_SUCCESS != \
qatUtilsMutexTryLock((pLock), QAT_UTILS_WAIT_NONE)) ? \
CPA_STATUS_RESOURCE : \
CPA_STATUS_SUCCESS)
/*
*******************************************************************************
* Semaphore Macros
*******************************************************************************
*/
/**
*******************************************************************************
* @ingroup LacCommon
* This macro waits on a semaphore and returns the status
*
* @param[in] sid The semaphore
* @param[in] timeout Timeout
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_RESOURCE Error with semaphore
******************************************************************************/
#define LAC_WAIT_SEMAPHORE(sid, timeout) \
((CPA_STATUS_SUCCESS != qatUtilsSemaphoreWait(&sid, (timeout))) ? \
CPA_STATUS_RESOURCE : \
CPA_STATUS_SUCCESS)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro checks a semaphore and returns the status
*
* @param[in] sid The semaphore
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_RESOURCE Error with semaphore
******************************************************************************/
#define LAC_CHECK_SEMAPHORE(sid) \
((CPA_STATUS_SUCCESS != qatUtilsSemaphoreTryWait(&sid)) ? \
CPA_STATUS_RETRY : \
CPA_STATUS_SUCCESS)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro post a semaphore and returns the status
*
* @param[in] sid The semaphore
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_RESOURCE Error with semaphore
******************************************************************************/
#define LAC_POST_SEMAPHORE(sid) \
((CPA_STATUS_SUCCESS != qatUtilsSemaphorePost(&sid)) ? \
CPA_STATUS_RESOURCE : \
CPA_STATUS_SUCCESS)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro initialises a semaphore and returns the status
*
* @param[in] sid The semaphore
* @param[in] semValue Initial semaphore value
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_RESOURCE Error with semaphore
******************************************************************************/
#define LAC_INIT_SEMAPHORE(sid, semValue) \
((CPA_STATUS_SUCCESS != qatUtilsSemaphoreInit(&sid, semValue)) ? \
CPA_STATUS_RESOURCE : \
CPA_STATUS_SUCCESS)
/**
*******************************************************************************
* @ingroup LacCommon
* This macro destroys a semaphore and returns the status
*
* @param[in] sid The semaphore
*
* @retval CPA_STATUS_SUCCESS Function executed successfully.
* @retval CPA_STATUS_RESOURCE Error with semaphore
******************************************************************************/
#define LAC_DESTROY_SEMAPHORE(sid) \
((CPA_STATUS_SUCCESS != qatUtilsSemaphoreDestroy(&sid)) ? \
CPA_STATUS_RESOURCE : \
CPA_STATUS_SUCCESS)
/*
*******************************************************************************
* Spinlock Macros
*******************************************************************************
*/
typedef struct mtx *lac_lock_t;
#define LAC_SPINLOCK_INIT(lock) \
((CPA_STATUS_SUCCESS != qatUtilsLockInit(lock)) ? \
CPA_STATUS_RESOURCE : \
CPA_STATUS_SUCCESS)
#define LAC_SPINLOCK(lock) \
({ \
(void)qatUtilsLock(lock); \
CPA_STATUS_SUCCESS; \
})
#define LAC_SPINUNLOCK(lock) \
({ \
(void)qatUtilsUnlock(lock); \
CPA_STATUS_SUCCESS; \
})
#define LAC_SPINLOCK_DESTROY(lock) \
({ \
(void)qatUtilsLockDestroy(lock); \
CPA_STATUS_SUCCESS; \
})
#define LAC_CONST_PTR_CAST(castee) ((void *)(LAC_ARCH_UINT)(castee))
#define LAC_CONST_VOLATILE_PTR_CAST(castee) ((void *)(LAC_ARCH_UINT)(castee))
/* Type of ring */
#define SAL_RING_TYPE_NONE 0
#define SAL_RING_TYPE_A_SYM_HI 1
#define SAL_RING_TYPE_A_SYM_LO 2
#define SAL_RING_TYPE_A_ASYM 3
#define SAL_RING_TYPE_B_SYM_HI 4
#define SAL_RING_TYPE_B_SYM_LO 5
#define SAL_RING_TYPE_B_ASYM 6
#define SAL_RING_TYPE_DC 7
#define SAL_RING_TYPE_ADMIN 8
#define SAL_RING_TYPE_TRNG 9
/* Maps Ring Service to generic service type */
static inline icp_adf_ringInfoService_t
lac_getRingType(int type)
{
switch (type) {
case SAL_RING_TYPE_NONE:
return ICP_ADF_RING_SERVICE_0;
case SAL_RING_TYPE_A_SYM_HI:
return ICP_ADF_RING_SERVICE_1;
case SAL_RING_TYPE_A_SYM_LO:
return ICP_ADF_RING_SERVICE_2;
case SAL_RING_TYPE_A_ASYM:
return ICP_ADF_RING_SERVICE_3;
case SAL_RING_TYPE_B_SYM_HI:
return ICP_ADF_RING_SERVICE_4;
case SAL_RING_TYPE_B_SYM_LO:
return ICP_ADF_RING_SERVICE_5;
case SAL_RING_TYPE_B_ASYM:
return ICP_ADF_RING_SERVICE_6;
case SAL_RING_TYPE_DC:
return ICP_ADF_RING_SERVICE_7;
case SAL_RING_TYPE_ADMIN:
return ICP_ADF_RING_SERVICE_8;
case SAL_RING_TYPE_TRNG:
return ICP_ADF_RING_SERVICE_9;
default:
return ICP_ADF_RING_SERVICE_0;
}
return ICP_ADF_RING_SERVICE_0;
}
/* Maps generic service type to Ring Service type */
static inline int
lac_getServiceType(icp_adf_ringInfoService_t type)
{
switch (type) {
case ICP_ADF_RING_SERVICE_0:
return SAL_RING_TYPE_NONE;
case ICP_ADF_RING_SERVICE_1:
return SAL_RING_TYPE_A_SYM_HI;
case ICP_ADF_RING_SERVICE_2:
return SAL_RING_TYPE_A_SYM_LO;
case ICP_ADF_RING_SERVICE_3:
return SAL_RING_TYPE_A_ASYM;
case ICP_ADF_RING_SERVICE_4:
return SAL_RING_TYPE_B_SYM_HI;
case ICP_ADF_RING_SERVICE_5:
return SAL_RING_TYPE_B_SYM_LO;
case ICP_ADF_RING_SERVICE_6:
return SAL_RING_TYPE_B_ASYM;
case ICP_ADF_RING_SERVICE_7:
return SAL_RING_TYPE_DC;
case ICP_ADF_RING_SERVICE_8:
return SAL_RING_TYPE_ADMIN;
default:
return SAL_RING_TYPE_NONE;
}
return SAL_RING_TYPE_NONE;
}
#endif /* LAC_COMMON_H */

Some files were not shown because too many files have changed in this diff Show More