d/freebsd-nq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Introduce BLAKE3 checksums as an OpenZFS feature

Browse Source 
This commit adds BLAKE3 checksums to OpenZFS, it has similar
performance to Edon-R, but without the caveats around the latter.

Homepage of BLAKE3: https://github.com/BLAKE3-team/BLAKE3
Wikipedia: https://en.wikipedia.org/wiki/BLAKE_(hash_function)#BLAKE3

Short description of Wikipedia:

  BLAKE3 is a cryptographic hash function based on Bao and BLAKE2,
  created by Jack O'Connor, Jean-Philippe Aumasson, Samuel Neves, and
  Zooko Wilcox-O'Hearn. It was announced on January 9, 2020, at Real
  World Crypto. BLAKE3 is a single algorithm with many desirable
  features (parallelism, XOF, KDF, PRF and MAC), in contrast to BLAKE
  and BLAKE2, which are algorithm families with multiple variants.
  BLAKE3 has a binary tree structure, so it supports a practically
  unlimited degree of parallelism (both SIMD and multithreading) given
  enough input. The official Rust and C implementations are
  dual-licensed as public domain (CC0) and the Apache License.

Along with adding the BLAKE3 hash into the OpenZFS infrastructure a
new benchmarking file called chksum_bench was introduced.  When read
it reports the speed of the available checksum functions.

On Linux: cat /proc/spl/kstat/zfs/chksum_bench
On FreeBSD: sysctl kstat.zfs.misc.chksum_bench

This is an example output of an i3-1005G1 test system with Debian 11:

implementation      1k      4k     16k     64k    256k      1m      4m
edonr-generic     1196    1602    1761    1749    1762    1759    1751
skein-generic      546     591     608     615     619     612     616
sha256-generic     240     300     316     314     304     285     276
sha512-generic     353     441     467     476     472     467     426
blake3-generic     308     313     313     313     312     313     312
blake3-sse2        402    1289    1423    1446    1432    1458    1413
blake3-sse41       427    1470    1625    1704    1679    1607    1629
blake3-avx2        428    1920    3095    3343    3356    3318    3204
blake3-avx512      473    2687    4905    5836    5844    5643    5374

Output on Debian 5.10.0-10-amd64 system: (Ryzen 7 5800X)

implementation      1k      4k     16k     64k    256k      1m      4m
edonr-generic     1840    2458    2665    2719    2711    2723    2693
skein-generic      870     966     996     992    1003    1005    1009
sha256-generic     415     442     453     455     457     457     457
sha512-generic     608     690     711     718     719     720     721
blake3-generic     301     313     311     309     309     310     310
blake3-sse2        343    1865    2124    2188    2180    2181    2186
blake3-sse41       364    2091    2396    2509    2463    2482    2488
blake3-avx2        365    2590    4399    4971    4915    4802    4764

Output on Debian 5.10.0-9-powerpc64le system: (POWER 9)

implementation      1k      4k     16k     64k    256k      1m      4m
edonr-generic     1213    1703    1889    1918    1957    1902    1907
skein-generic      434     492     520     522     511     525     525
sha256-generic     167     183     187     188     188     187     188
sha512-generic     186     216     222     221     225     224     224
blake3-generic     153     152     154     153     151     153     153
blake3-sse2        391    1170    1366    1406    1428    1426    1414
blake3-sse41       352    1049    1212    1174    1262    1258    1259

Output on Debian 5.10.0-11-arm64 system: (Pi400)

implementation      1k      4k     16k     64k    256k      1m      4m
edonr-generic      487     603     629     639     643     641     641
skein-generic      271     299     303     308     309     309     307
sha256-generic     117     127     128     130     130     129     130
sha512-generic     145     165     170     172     173     174     175
blake3-generic      81      29      71      89      89      89      89
blake3-sse2        112     323     368     379     380     371     374
blake3-sse41       101     315     357     368     369     364     360

Structurally, the new code is mainly split into these parts:
- 1x cross platform generic c variant: blake3_generic.c
- 4x assembly for X86-64 (SSE2, SSE4.1, AVX2, AVX512)
- 2x assembly for ARMv8 (NEON converted from SSE2)
- 2x assembly for PPC64-LE (POWER8 converted from SSE2)
- one file for switching between the implementations

Note the PPC64 assembly requires the VSX instruction set and the
kfpu_begin() / kfpu_end() calls on PowerPC were updated accordingly.

Reviewed-by: Felix Dörre <felix@dogcraft.de>
Reviewed-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Co-authored-by: Rich Ercolani <rincebrain@gmail.com>
Closes #10058
Closes #12918
This commit is contained in:
Tino Reichardt 2022-06-09 00:55:57 +02:00 committed by GitHub
parent b9d98453f9
commit 985c33b132
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
53 changed files with 22804 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
AUTHORS
View File

@ -285,6 +285,7 @@ CONTRIBUTORS:
Tim Connors <tconnors@rather.puzzling.org>
Tim Crawford <tcrawford@datto.com>
Tim Haley <Tim.Haley@Sun.COM>
Tino Reichardt <milky-zfs@mcmilk.de>
Tobin Harding <me@tobin.cc>
Tom Caputi <tcaputi@datto.com>
Tom Matthews <tom@axiom-partners.com>

89
cmd/ztest.c
View File

@ -121,6 +121,7 @@
#include <sys/zfeature.h>
#include <sys/dsl_userhold.h>
#include <sys/abd.h>
#include <sys/blake3.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
@ -417,6 +418,7 @@ ztest_func_t ztest_device_removal;
ztest_func_t ztest_spa_checkpoint_create_discard;
ztest_func_t ztest_initialize;
ztest_func_t ztest_trim;
ztest_func_t ztest_blake3;
ztest_func_t ztest_fletcher;
ztest_func_t ztest_fletcher_incr;
ztest_func_t ztest_verify_dnode_bt;
@ -470,6 +472,7 @@ ztest_info_t ztest_info[] = {
ZTI_INIT(ztest_spa_checkpoint_create_discard, 1, &zopt_rarely),
ZTI_INIT(ztest_initialize, 1, &zopt_sometimes),
ZTI_INIT(ztest_trim, 1, &zopt_sometimes),
ZTI_INIT(ztest_blake3, 1, &zopt_rarely),
ZTI_INIT(ztest_fletcher, 1, &zopt_rarely),
ZTI_INIT(ztest_fletcher_incr, 1, &zopt_rarely),
ZTI_INIT(ztest_verify_dnode_bt, 1, &zopt_sometimes),
@ -6373,6 +6376,92 @@ ztest_reguid(ztest_ds_t *zd, uint64_t id)
VERIFY3U(load, ==, spa_load_guid(spa));
}
void
ztest_blake3(ztest_ds_t *zd, uint64_t id)
{
(void) zd, (void) id;
hrtime_t end = gethrtime() + NANOSEC;
zio_cksum_salt_t salt;
void *salt_ptr = &salt.zcs_bytes;
struct abd *abd_data, *abd_meta;
void *buf, *templ;
int i, *ptr;
uint32_t size;
BLAKE3_CTX ctx;
size = ztest_random_blocksize();
buf = umem_alloc(size, UMEM_NOFAIL);
abd_data = abd_alloc(size, B_FALSE);
abd_meta = abd_alloc(size, B_TRUE);
for (i = 0, ptr = buf; i < size / sizeof (*ptr); i++, ptr++)
*ptr = ztest_random(UINT_MAX);
memset(salt_ptr, 'A', 32);
abd_copy_from_buf_off(abd_data, buf, 0, size);
abd_copy_from_buf_off(abd_meta, buf, 0, size);
while (gethrtime() <= end) {
int run_count = 100;
zio_cksum_t zc_ref1, zc_ref2;
zio_cksum_t zc_res1, zc_res2;
void *ref1 = &zc_ref1;
void *ref2 = &zc_ref2;
void *res1 = &zc_res1;
void *res2 = &zc_res2;
/* BLAKE3_KEY_LEN = 32 */
VERIFY0(blake3_set_impl_name("generic"));
templ = abd_checksum_blake3_tmpl_init(&salt);
Blake3_InitKeyed(&ctx, salt_ptr);
Blake3_Update(&ctx, buf, size);
Blake3_Final(&ctx, ref1);
zc_ref2 = zc_ref1;
ZIO_CHECKSUM_BSWAP(&zc_ref2);
abd_checksum_blake3_tmpl_free(templ);
VERIFY0(blake3_set_impl_name("cycle"));
while (run_count-- > 0) {
/* Test current implementation */
Blake3_InitKeyed(&ctx, salt_ptr);
Blake3_Update(&ctx, buf, size);
Blake3_Final(&ctx, res1);
zc_res2 = zc_res1;
ZIO_CHECKSUM_BSWAP(&zc_res2);
VERIFY0(memcmp(ref1, res1, 32));
VERIFY0(memcmp(ref2, res2, 32));
/* Test ABD - data */
templ = abd_checksum_blake3_tmpl_init(&salt);
abd_checksum_blake3_native(abd_data, size,
templ, &zc_res1);
abd_checksum_blake3_byteswap(abd_data, size,
templ, &zc_res2);
VERIFY0(memcmp(ref1, res1, 32));
VERIFY0(memcmp(ref2, res2, 32));
/* Test ABD - metadata */
abd_checksum_blake3_native(abd_meta, size,
templ, &zc_res1);
abd_checksum_blake3_byteswap(abd_meta, size,
templ, &zc_res2);
abd_checksum_blake3_tmpl_free(templ);
VERIFY0(memcmp(ref1, res1, 32));
VERIFY0(memcmp(ref2, res2, 32));
}
}
abd_free(abd_data);
abd_free(abd_meta);
umem_free(buf, size);
}
void
ztest_fletcher(ztest_ds_t *zd, uint64_t id)
{

2
config/always-arch.m4
View File

@ -30,6 +30,8 @@ AC_DEFUN([ZFS_AC_CONFIG_ALWAYS_ARCH], [
;;
esac
AM_CONDITIONAL([TARGET_CPU_AARCH64], test $TARGET_CPU = aarch64)
AM_CONDITIONAL([TARGET_CPU_X86_64], test $TARGET_CPU = x86_64)
AM_CONDITIONAL([TARGET_CPU_POWERPC], test $TARGET_CPU = powerpc)
AM_CONDITIONAL([TARGET_CPU_SPARC64], test $TARGET_CPU = sparc64)
])

2
include/Makefile.am
View File

@ -23,6 +23,7 @@ COMMON_H = \
sys/avl.h \
sys/avl_impl.h \
sys/bitops.h \
sys/blake3.h \
sys/blkptr.h \
sys/bplist.h \
sys/bpobj.h \
@ -117,6 +118,7 @@ COMMON_H = \
sys/zfeature.h \
sys/zfs_acl.h \
sys/zfs_bootenv.h \
sys/zfs_chksum.h \
sys/zfs_context.h \
sys/zfs_debug.h \
sys/zfs_delay.h \

2
include/os/freebsd/spl/sys/ccompile.h
View File

@ -74,10 +74,12 @@ extern "C" {
#ifndef LOCORE
#ifndef HAVE_RPC_TYPES
#ifndef _KERNEL
typedef int bool_t;
typedef int enum_t;
#endif
#endif
#endif
#ifndef __cplusplus
#define __init

34
include/os/linux/kernel/linux/simd_powerpc.h
View File

@ -57,25 +57,45 @@
#include <sys/types.h>
#include <linux/version.h>
#define kfpu_allowed() 1
#define kfpu_begin() \
{ \
preempt_disable(); \
enable_kernel_altivec(); \
}
#define kfpu_allowed() 1
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 5, 0)
#define kfpu_end() \
{ \
disable_kernel_vsx(); \
disable_kernel_altivec(); \
preempt_enable(); \
}
#define kfpu_begin() \
{ \
preempt_disable(); \
enable_kernel_altivec(); \
enable_kernel_vsx(); \
}
#else
/* seems that before 4.5 no-one bothered disabling ... */
/* seems that before 4.5 no-one bothered */
#define kfpu_begin()
#define kfpu_end() preempt_enable()
#endif
#define kfpu_init() 0
#define kfpu_fini() ((void) 0)
static inline boolean_t
zfs_vsx_available(void)
{
boolean_t res;
#if defined(__powerpc64__)
u64 msr;
#else
u32 msr;
#endif
kfpu_begin();
__asm volatile("mfmsr %0" : "=r"(msr));
res = (msr & 0x800000) != 0;
kfpu_end();
return (res);
}
/*
* Check if AltiVec instruction set is available
*/

120
include/sys/blake3.h Normal file
View File

@ -0,0 +1,120 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Based on BLAKE3 v1.3.1, https://github.com/BLAKE3-team/BLAKE3
* Copyright (c) 2019-2020 Samuel Neves and Jack O'Connor
* Copyright (c) 2021 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#ifndef BLAKE3_H
#define BLAKE3_H
#ifdef _KERNEL
#include <sys/types.h>
#else
#include <stdint.h>
#include <stdlib.h>
#endif
#ifdef __cplusplus
extern "C" {
#endif
#define BLAKE3_KEY_LEN 32
#define BLAKE3_OUT_LEN 32
#define BLAKE3_MAX_DEPTH 54
#define BLAKE3_BLOCK_LEN 64
#define BLAKE3_CHUNK_LEN 1024
/*
* This struct is a private implementation detail.
* It has to be here because it's part of BLAKE3_CTX below.
*/
typedef struct {
uint32_t cv[8];
uint64_t chunk_counter;
uint8_t buf[BLAKE3_BLOCK_LEN];
uint8_t buf_len;
uint8_t blocks_compressed;
uint8_t flags;
} blake3_chunk_state_t;
typedef struct {
uint32_t key[8];
blake3_chunk_state_t chunk;
uint8_t cv_stack_len;
/*
* The stack size is MAX_DEPTH + 1 because we do lazy merging. For
* example, with 7 chunks, we have 3 entries in the stack. Adding an
* 8th chunk requires a 4th entry, rather than merging everything down
* to 1, because we don't know whether more input is coming. This is
* different from how the reference implementation does things.
*/
uint8_t cv_stack[(BLAKE3_MAX_DEPTH + 1) * BLAKE3_OUT_LEN];
/* const blake3_impl_ops_t *ops */
const void *ops;
} BLAKE3_CTX;
/* init the context for hash operation */
void Blake3_Init(BLAKE3_CTX *ctx);
/* init the context for a MAC and/or tree hash operation */
void Blake3_InitKeyed(BLAKE3_CTX *ctx, const uint8_t key[BLAKE3_KEY_LEN]);
/* process the input bytes */
void Blake3_Update(BLAKE3_CTX *ctx, const void *input, size_t input_len);
/* finalize the hash computation and output the result */
void Blake3_Final(const BLAKE3_CTX *ctx, uint8_t *out);
/* finalize the hash computation and output the result */
void Blake3_FinalSeek(const BLAKE3_CTX *ctx, uint64_t seek, uint8_t *out,
size_t out_len);
/* return number of supported implementations */
extern int blake3_get_impl_count(void);
/* return id of selected implementation */
extern int blake3_get_impl_id(void);
/* return name of selected implementation */
extern const char *blake3_get_impl_name(void);
/* setup id as fastest implementation */
extern void blake3_set_impl_fastest(uint32_t id);
/* set implementation by id */
extern void blake3_set_impl_id(uint32_t id);
/* set implementation by name */
extern int blake3_set_impl_name(const char *name);
/* set startup implementation */
extern void blake3_setup_impl(void);
#ifdef __cplusplus
}
#endif
#endif /* BLAKE3_H */

48
include/sys/zfs_chksum.h Normal file
View File

@ -0,0 +1,48 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Copyright (c) 2021 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#ifndef _ZFS_CHKSUM_H
#define _ZFS_CHKSUM_H
#ifdef _KERNEL
#include <sys/types.h>
#else
#include <stdint.h>
#include <stdlib.h>
#endif
#ifdef __cplusplus
extern "C" {
#endif
/* Benchmark the chksums of ZFS when the module is loading */
void chksum_init(void);
void chksum_fini(void);
#ifdef __cplusplus
}
#endif
#endif /* _ZFS_CHKSUM_H */

3
include/sys/zfs_ioctl.h
View File

@ -124,6 +124,7 @@ typedef enum drr_headertype {
* default use of "zfs send" won't encounter the bug mentioned above.
*/
#define DMU_BACKUP_FEATURE_SWITCH_TO_LARGE_BLOCKS (1 << 27)
#define DMU_BACKUP_FEATURE_BLAKE3 (1 << 28)
/*
* Mask of all supported backup features
@ -134,7 +135,7 @@ typedef enum drr_headertype {
DMU_BACKUP_FEATURE_COMPRESSED | DMU_BACKUP_FEATURE_LARGE_DNODE | \
DMU_BACKUP_FEATURE_RAW | DMU_BACKUP_FEATURE_HOLDS | \
DMU_BACKUP_FEATURE_REDACTED | DMU_BACKUP_FEATURE_SWITCH_TO_LARGE_BLOCKS | \
DMU_BACKUP_FEATURE_ZSTD)
DMU_BACKUP_FEATURE_ZSTD | DMU_BACKUP_FEATURE_BLAKE3)
/* Are all features in the given flag word currently supported? */
#define DMU_STREAM_SUPPORTED(x) (!((x) & ~DMU_BACKUP_FEATURE_MASK))

1
include/sys/zio.h
View File

@ -89,6 +89,7 @@ enum zio_checksum {
ZIO_CHECKSUM_SHA512,
ZIO_CHECKSUM_SKEIN,
ZIO_CHECKSUM_EDONR,
ZIO_CHECKSUM_BLAKE3,
ZIO_CHECKSUM_FUNCTIONS
};

12
include/sys/zio_checksum.h
View File

@ -21,7 +21,8 @@
/*
* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2014, 2016 by Delphix. All rights reserved.
* Copyright Saso Kiselkov 2013, All rights reserved.
* Copyright (c) 2013 Saso Kiselkov, All rights reserved.
* Copyright (c) 2021 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#ifndef _SYS_ZIO_CHECKSUM_H
@ -107,6 +108,8 @@ _SYS_ZIO_CHECKSUM_H zio_checksum_info_t
/*
* Checksum routines.
*/
/* SHA2 */
extern zio_checksum_t abd_checksum_SHA256;
extern zio_checksum_t abd_checksum_SHA512_native;
extern zio_checksum_t abd_checksum_SHA512_byteswap;
@ -123,6 +126,13 @@ extern zio_checksum_t abd_checksum_edonr_byteswap;
extern zio_checksum_tmpl_init_t abd_checksum_edonr_tmpl_init;
extern zio_checksum_tmpl_free_t abd_checksum_edonr_tmpl_free;
/* BLAKE3 */
extern zio_checksum_t abd_checksum_blake3_native;
extern zio_checksum_t abd_checksum_blake3_byteswap;
extern zio_checksum_tmpl_init_t abd_checksum_blake3_tmpl_init;
extern zio_checksum_tmpl_free_t abd_checksum_blake3_tmpl_free;
/* Fletcher 4 */
_SYS_ZIO_CHECKSUM_H zio_abd_checksum_func_t fletcher_4_abd_ops;
extern zio_checksum_t abd_fletcher_4_native;
extern zio_checksum_t abd_fletcher_4_byteswap;

1
include/zfeature_common.h
View File

@ -77,6 +77,7 @@ typedef enum spa_feature {
SPA_FEATURE_DRAID,
SPA_FEATURE_ZILSAXATTR,
SPA_FEATURE_HEAD_ERRLOG,
SPA_FEATURE_BLAKE3,
SPA_FEATURES
} spa_feature_t;

25
lib/libicp/Makefile.am
View File

@ -13,6 +13,10 @@ nodist_libicp_la_SOURCES = \
module/icp/algs/aes/aes_impl_x86-64.c \
module/icp/algs/aes/aes_impl.c \
module/icp/algs/aes/aes_modes.c \
module/icp/algs/blake3/blake3.c \
module/icp/algs/blake3/blake3_generic.c \
module/icp/algs/blake3/blake3_impl.c \
module/icp/algs/blake3/blake3_x86-64.c \
module/icp/algs/edonr/edonr.c \
module/icp/algs/modes/modes.c \
module/icp/algs/modes/cbc.c \
@ -36,15 +40,30 @@ nodist_libicp_la_SOURCES = \
module/icp/core/kcf_mech_tabs.c \
module/icp/core/kcf_prov_tabs.c
if TARGET_CPU_AARCH64
nodist_libicp_la_SOURCES += \
module/icp/asm-aarch64/blake3/b3_aarch64_sse2.S \
module/icp/asm-aarch64/blake3/b3_aarch64_sse41.S
endif
if TARGET_CPU_POWERPC
nodist_libicp_la_SOURCES += \
module/icp/asm-ppc64/blake3/b3_ppc64le_sse2.S \
module/icp/asm-ppc64/blake3/b3_ppc64le_sse41.S
endif
if TARGET_CPU_X86_64
nodist_libicp_la_SOURCES += \
module/icp/asm-x86_64/aes/aeskey.c
nodist_libicp_la_SOURCES += \
module/icp/asm-x86_64/aes/aeskey.c \
module/icp/asm-x86_64/aes/aes_amd64.S \
module/icp/asm-x86_64/aes/aes_aesni.S \
module/icp/asm-x86_64/modes/gcm_pclmulqdq.S \
module/icp/asm-x86_64/modes/aesni-gcm-x86_64.S \
module/icp/asm-x86_64/modes/ghash-x86_64.S \
module/icp/asm-x86_64/sha2/sha256_impl.S \
module/icp/asm-x86_64/sha2/sha512_impl.S
module/icp/asm-x86_64/sha2/sha512_impl.S \
module/icp/asm-x86_64/blake3/blake3_avx2.S \
module/icp/asm-x86_64/blake3/blake3_avx512.S \
module/icp/asm-x86_64/blake3/blake3_sse2.S \
module/icp/asm-x86_64/blake3/blake3_sse41.S
endif

18
lib/libspl/include/sys/simd.h
View File

@ -491,6 +491,24 @@ zfs_altivec_available(void)
#endif
return (has_altivec);
}
static inline boolean_t
zfs_vsx_available(void)
{
boolean_t has_vsx = B_FALSE;
#if defined(__ALTIVEC__) && !defined(__FreeBSD__)
sighandler_t savesig;
savesig = signal(SIGILL, sigillhandler);
if (setjmp(env)) {
signal(SIGILL, savesig);
has_vsx = B_FALSE;
} else {
__asm__ __volatile__("xssubsp 0,0,0\n");
signal(SIGILL, savesig);
has_vsx = B_TRUE;
}
#endif
return (has_vsx);
}
#else
#define kfpu_allowed() 0

9
lib/libzfs/libzfs.abi
View File

@ -583,7 +583,7 @@
<elf-symbol name='fletcher_4_superscalar_ops' size='64' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='libzfs_config_ops' size='16' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='sa_protocol_names' size='16' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='spa_feature_table' size='2016' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='spa_feature_table' size='2072' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='zfeature_checks_disable' size='4' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='zfs_deleg_perm_tab' size='512' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='zfs_history_event_names' size='328' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
@ -4770,8 +4770,8 @@
</function-decl>
</abi-instr>
<abi-instr address-size='64' path='module/zcommon/zfeature_common.c' language='LANG_C99'>
<array-type-def dimensions='1' type-id='83f29ca2' size-in-bits='16128' id='9d5e9e2e'>
<subrange length='36' type-id='7359adad' id='ae666bde'/>
<array-type-def dimensions='1' type-id='83f29ca2' size-in-bits='16576' id='9d5e9e2e'>
<subrange length='37' type-id='7359adad' id='ae666bde'/>
</array-type-def>
<enum-decl name='spa_feature' id='33ecb627'>
<underlying-type type-id='9cac1fee'/>
@ -4812,7 +4812,8 @@
<enumerator name='SPA_FEATURE_DRAID' value='33'/>
<enumerator name='SPA_FEATURE_ZILSAXATTR' value='34'/>
<enumerator name='SPA_FEATURE_HEAD_ERRLOG' value='35'/>
<enumerator name='SPA_FEATURES' value='36'/>
<enumerator name='SPA_FEATURE_BLAKE3' value='36'/>
<enumerator name='SPA_FEATURES' value='37'/>
</enum-decl>
<typedef-decl name='spa_feature_t' type-id='33ecb627' id='d6618c78'/>
<enum-decl name='zfeature_flags' id='6db816a4'>

2
lib/libzpool/Makefile.am
View File

@ -67,6 +67,7 @@ nodist_libzpool_la_SOURCES = \
module/zfs/abd.c \
module/zfs/aggsum.c \
module/zfs/arc.c \
module/zfs/blake3_zfs.c \
module/zfs/blkptr.c \
module/zfs/bplist.c \
module/zfs/bpobj.c \
@ -171,6 +172,7 @@ nodist_libzpool_la_SOURCES = \
module/zfs/zcp_synctask.c \
module/zfs/zfeature.c \
module/zfs/zfs_byteswap.c \
module/zfs/zfs_chksum.c \
module/zfs/zfs_fm.c \
module/zfs/zfs_fuid.c \
module/zfs/zfs_ratelimit.c \

7
man/man7/zfsprops.7
View File

@ -743,7 +743,7 @@ This property is not inherited.
.It Xo
.Sy checksum Ns = Ns Sy on Ns | Ns Sy off Ns | Ns Sy fletcher2 Ns | Ns
.Sy fletcher4 Ns | Ns Sy sha256 Ns | Ns Sy noparity Ns | Ns
.Sy sha512 Ns | Ns Sy skein Ns | Ns Sy edonr
.Sy sha512 Ns | Ns Sy skein Ns | Ns Sy edonr Ns | Ns Sy blake3
.Xc
Controls the checksum used to verify data integrity.
The default value is
@ -768,8 +768,9 @@ a recommended practice.
The
.Sy sha512 ,
.Sy skein ,
.Sy edonr ,
and
.Sy edonr
.Sy blake3
checksum algorithms require enabling the appropriate features on the pool.
.Pp
Please see
@ -984,7 +985,7 @@ mount options.
.It Xo
.Sy dedup Ns = Ns Sy off Ns | Ns Sy on Ns | Ns Sy verify Ns | Ns
.Sy sha256 Ns Oo , Ns Sy verify Oc Ns | Ns Sy sha512 Ns Oo , Ns Sy verify Oc Ns | Ns Sy skein Ns Oo , Ns Sy verify Oc Ns | Ns
.Sy edonr , Ns Sy verify
.Sy edonr , Ns Sy verify Ns | Ns Sy blake3 Ns Oo , Ns Sy verify Oc Ns
.Xc
Configures deduplication for a dataset.
The default value is

8
man/man7/zpool-features.7
View File

@ -326,6 +326,12 @@ while
.Sy freeing
is non-zero.
.
.feature org.openzfs blake3 no extensible_dataset
This feature enables the use of the BLAKE3 hash algorithm for checksum and dedup.
BLAKE3 is a secure hash algorithm focused on high performance.
.Pp
.checksum-spiel blake3
.
.feature com.delphix bookmarks yes extensible_dataset
This feature enables use of the
.Nm zfs Cm bookmark
@ -436,6 +442,8 @@ in ZFS, which means that the checksum is pre-seeded with a secret
to be checksummed.
Thus the produced checksums are unique to a given pool,
preventing hash collision attacks on systems with dedup.
.Pp
.checksum-spiel edonr
.
.feature com.delphix embedded_data no
This feature improves the performance and compression ratio of

32
module/Kbuild.in
View File

@ -75,6 +75,10 @@ ICP_OBJS := \
algs/aes/aes_impl.o \
algs/aes/aes_impl_generic.o \
algs/aes/aes_modes.o \
algs/blake3/blake3.o \
algs/blake3/blake3_generic.o \
algs/blake3/blake3_impl.o \
algs/blake3/blake3_x86-64.o \
algs/edonr/edonr.o \
algs/modes/cbc.o \
algs/modes/ccm.o \
@ -105,23 +109,44 @@ ICP_OBJS_X86_64 := \
asm-x86_64/aes/aes_aesni.o \
asm-x86_64/aes/aes_amd64.o \
asm-x86_64/aes/aeskey.o \
asm-x86_64/blake3/blake3_avx2.o \
asm-x86_64/blake3/blake3_avx512.o \
asm-x86_64/blake3/blake3_sse2.o \
asm-x86_64/blake3/blake3_sse41.o \
asm-x86_64/modes/aesni-gcm-x86_64.o \
asm-x86_64/modes/gcm_pclmulqdq.o \
asm-x86_64/modes/ghash-x86_64.o \
asm-x86_64/sha2/sha256_impl.o \
asm-x86_64/sha2/sha512_impl.o
ICP_OBJS_X86 := \
algs/aes/aes_impl_aesni.o \
algs/aes/aes_impl_x86-64.o \
algs/modes/gcm_pclmulqdq.o
ICP_OBJS_ARM64 := \
asm-aarch64/blake3/b3_aarch64_sse2.o \
asm-aarch64/blake3/b3_aarch64_sse41.o
ICP_OBJS_PPC_PPC64 := \
asm-ppc64/blake3/b3_ppc64le_sse2.o \
asm-ppc64/blake3/b3_ppc64le_sse41.o
zfs-objs += $(addprefix icp/,$(ICP_OBJS))
zfs-$(CONFIG_X86) += $(addprefix icp/,$(ICP_OBJS_X86))
zfs-$(CONFIG_X86_64) += $(addprefix icp/,$(ICP_OBJS_X86_64))
zfs-$(CONFIG_ARM64) += $(addprefix icp/,$(ICP_OBJS_ARM64))
zfs-$(CONFIG_PPC) += $(addprefix icp/,$(ICP_OBJS_PPC_PPC64))
zfs-$(CONFIG_PPC64) += $(addprefix icp/,$(ICP_OBJS_PPC_PPC64))
$(addprefix $(obj)/icp/,$(ICP_OBJS) $(ICP_OBJS_X86) $(ICP_OBJS_X86_64)) : asflags-y += -I$(icp_include)
$(addprefix $(obj)/icp/,$(ICP_OBJS) $(ICP_OBJS_X86) $(ICP_OBJS_X86_64)) : ccflags-y += -I$(icp_include)
$(addprefix $(obj)/icp/,$(ICP_OBJS) $(ICP_OBJS_X86) $(ICP_OBJS_X86_64) \
$(ICP_OBJS_ARM64) $(ICP_OBJS_PPC_PPC64)) : asflags-y += -I$(icp_include)
$(addprefix $(obj)/icp/,$(ICP_OBJS) $(ICP_OBJS_X86) $(ICP_OBJS_X86_64) \
$(ICP_OBJS_ARM64) $(ICP_OBJS_PPC_PPC64)) : ccflags-y += -I$(icp_include)
# Suppress objtool "can't find jump dest instruction at" warnings. They
# are caused by the constants which are defined in the text section of the
@ -129,6 +154,7 @@ $(addprefix $(obj)/icp/,$(ICP_OBJS) $(ICP_OBJS_X86) $(ICP_OBJS_X86_64)) : ccflag
# utility tries to interpret them as opcodes and obviously fails doing so.
OBJECT_FILES_NON_STANDARD_aesni-gcm-x86_64.o := y
OBJECT_FILES_NON_STANDARD_ghash-x86_64.o := y
# Suppress objtool "unsupported stack pointer realignment" warnings. We are
# not using a DRAP register while aligning the stack to a 64 byte boundary.
# See #6950 for the reasoning.
@ -261,6 +287,7 @@ ZFS_OBJS := \
abd.o \
aggsum.o \
arc.o \
blake3_zfs.o \
blkptr.o \
bplist.o \
bpobj.o \
@ -358,6 +385,7 @@ ZFS_OBJS := \
zcp_synctask.o \
zfeature.o \
zfs_byteswap.o \
zfs_chksum.o \
zfs_fm.o \
zfs_fuid.o \
zfs_ioctl.o \

34
module/Makefile.bsd
View File

@ -10,6 +10,10 @@ INCDIR=${.CURDIR:H}/include
KMOD= openzfs
.PATH: ${SRCDIR}/avl \
${SRCDIR}/icp/algs/blake3 \
${SRCDIR}/icp/asm-aarch64/blake3 \
${SRCDIR}/icp/asm-ppc64/blake3 \
${SRCDIR}/icp/asm-x86_64/blake3 \
${SRCDIR}/lua \
${SRCDIR}/nvpair \
${SRCDIR}/icp/algs/edonr \
@ -31,6 +35,7 @@ CFLAGS+= -I${INCDIR}/os/freebsd
CFLAGS+= -I${INCDIR}/os/freebsd/spl
CFLAGS+= -I${INCDIR}/os/freebsd/zfs
CFLAGS+= -I${SRCDIR}/zstd/include
CFLAGS+= -I${SRCDIR}/icp/include
CFLAGS+= -include ${INCDIR}/os/freebsd/spl/sys/ccompile.h
CFLAGS+= -D__KERNEL__ -DFREEBSD_NAMECACHE -DBUILDING_ZFS -D__BSD_VISIBLE=1 \
@ -38,7 +43,8 @@ CFLAGS+= -D__KERNEL__ -DFREEBSD_NAMECACHE -DBUILDING_ZFS -D__BSD_VISIBLE=1 \
-D_SYS_VMEM_H_ -DKDTRACE_HOOKS -DSMP -DCOMPAT_FREEBSD11
.if ${MACHINE_ARCH} == "amd64"
CFLAGS+= -DHAVE_AVX2 -DHAVE_AVX -D__x86_64 -DHAVE_SSE2 -DHAVE_AVX512F -DHAVE_SSSE3
CFLAGS+= -D__x86_64 -DHAVE_SSE2 -DHAVE_SSSE3 -DHAVE_SSE4_1 -DHAVE_SSE4_2 \
-DHAVE_AVX -DHAVE_AVX2 -DHAVE_AVX512F -DHAVE_AVX512VL
.endif
.if defined(WITH_DEBUG) && ${WITH_DEBUG} == "true"
@ -73,12 +79,32 @@ CFLAGS+= -DBITS_PER_LONG=64
SRCS= vnode_if.h device_if.h bus_if.h
# avl
#avl
SRCS+= avl.c
# icp
SRCS+= edonr.c
#icp/algs/blake3
SRCS+= blake3.c \
blake3_generic.c \
blake3_impl.c \
blake3_x86-64.c
#icp/asm-aarch64/blake3
SRCS+= b3_aarch64_sse2.S \
b3_aarch64_sse41.S
#icp/asm-ppc64/blake3
SRCS+= b3_ppc64le_sse2.S \
b3_ppc64le_sse41.S
#icp/asm-x86_64/blake3
SRCS+= blake3_avx2.S \
blake3_avx512.S \
blake3_sse2.S \
blake3_sse41.S
#lua
SRCS+= lapi.c \
lauxlib.c \
@ -189,6 +215,7 @@ SRCS+= zfeature_common.c \
SRCS+= abd.c \
aggsum.c \
arc.c \
blake3_zfs.c \
blkptr.c \
bplist.c \
bpobj.c \
@ -291,6 +318,7 @@ SRCS+= abd.c \
zcp_synctask.c \
zfeature.c \
zfs_byteswap.c \
zfs_chksum.c \
zfs_file_os.c \
zfs_fm.c \
zfs_fuid.c \
@ -337,8 +365,6 @@ SRCS+= zfs_zstd.c \
zstd_decompress.c \
zstd_decompress_block.c
beforeinstall:
.if ${MK_DEBUG_FILES} != "no"
mtree -eu \

732
module/icp/algs/blake3/blake3.c Normal file
View File

@ -0,0 +1,732 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Based on BLAKE3 v1.3.1, https://github.com/BLAKE3-team/BLAKE3
* Copyright (c) 2019-2020 Samuel Neves and Jack O'Connor
* Copyright (c) 2021-2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include <sys/zfs_context.h>
#include <sys/blake3.h>
#include "blake3_impl.h"
/*
* We need 1056 byte stack for blake3_compress_subtree_wide()
* - we define this pragma to make gcc happy
*/
#if defined(__GNUC__)
#pragma GCC diagnostic ignored "-Wframe-larger-than="
#endif
/* internal used */
typedef struct {
uint32_t input_cv[8];
uint64_t counter;
uint8_t block[BLAKE3_BLOCK_LEN];
uint8_t block_len;
uint8_t flags;
} output_t;
/* internal flags */
enum blake3_flags {
CHUNK_START = 1 << 0,
CHUNK_END = 1 << 1,
PARENT = 1 << 2,
ROOT = 1 << 3,
KEYED_HASH = 1 << 4,
DERIVE_KEY_CONTEXT = 1 << 5,
DERIVE_KEY_MATERIAL = 1 << 6,
};
/* internal start */
static void chunk_state_init(blake3_chunk_state_t *ctx,
const uint32_t key[8], uint8_t flags)
{
memcpy(ctx->cv, key, BLAKE3_KEY_LEN);
ctx->chunk_counter = 0;
memset(ctx->buf, 0, BLAKE3_BLOCK_LEN);
ctx->buf_len = 0;
ctx->blocks_compressed = 0;
ctx->flags = flags;
}
static void chunk_state_reset(blake3_chunk_state_t *ctx,
const uint32_t key[8], uint64_t chunk_counter)
{
memcpy(ctx->cv, key, BLAKE3_KEY_LEN);
ctx->chunk_counter = chunk_counter;
ctx->blocks_compressed = 0;
memset(ctx->buf, 0, BLAKE3_BLOCK_LEN);
ctx->buf_len = 0;
}
static size_t chunk_state_len(const blake3_chunk_state_t *ctx)
{
return (BLAKE3_BLOCK_LEN * (size_t)ctx->blocks_compressed) +
((size_t)ctx->buf_len);
}
static size_t chunk_state_fill_buf(blake3_chunk_state_t *ctx,
const uint8_t *input, size_t input_len)
{
size_t take = BLAKE3_BLOCK_LEN - ((size_t)ctx->buf_len);
if (take > input_len) {
take = input_len;
}
uint8_t *dest = ctx->buf + ((size_t)ctx->buf_len);
memcpy(dest, input, take);
ctx->buf_len += (uint8_t)take;
return (take);
}
static uint8_t chunk_state_maybe_start_flag(const blake3_chunk_state_t *ctx)
{
if (ctx->blocks_compressed == 0) {
return (CHUNK_START);
} else {
return (0);
}
}
static output_t make_output(const uint32_t input_cv[8],
const uint8_t *block, uint8_t block_len,
uint64_t counter, uint8_t flags)
{
output_t ret;
memcpy(ret.input_cv, input_cv, 32);
memcpy(ret.block, block, BLAKE3_BLOCK_LEN);
ret.block_len = block_len;
ret.counter = counter;
ret.flags = flags;
return (ret);
}
/*
* Chaining values within a given chunk (specifically the compress_in_place
* interface) are represented as words. This avoids unnecessary bytes<->words
* conversion overhead in the portable implementation. However, the hash_many
* interface handles both user input and parent node blocks, so it accepts
* bytes. For that reason, chaining values in the CV stack are represented as
* bytes.
*/
static void output_chaining_value(const blake3_impl_ops_t *ops,
const output_t *ctx, uint8_t cv[32])
{
uint32_t cv_words[8];
memcpy(cv_words, ctx->input_cv, 32);
ops->compress_in_place(cv_words, ctx->block, ctx->block_len,
ctx->counter, ctx->flags);
store_cv_words(cv, cv_words);
}
static void output_root_bytes(const blake3_impl_ops_t *ops, const output_t *ctx,
uint64_t seek, uint8_t *out, size_t out_len)
{
uint64_t output_block_counter = seek / 64;
size_t offset_within_block = seek % 64;
uint8_t wide_buf[64];
while (out_len > 0) {
ops->compress_xof(ctx->input_cv, ctx->block, ctx->block_len,
output_block_counter, ctx->flags | ROOT, wide_buf);
size_t available_bytes = 64 - offset_within_block;
size_t memcpy_len;
if (out_len > available_bytes) {
memcpy_len = available_bytes;
} else {
memcpy_len = out_len;
}
memcpy(out, wide_buf + offset_within_block, memcpy_len);
out += memcpy_len;
out_len -= memcpy_len;
output_block_counter += 1;
offset_within_block = 0;
}
}
static void chunk_state_update(const blake3_impl_ops_t *ops,
blake3_chunk_state_t *ctx, const uint8_t *input, size_t input_len)
{
if (ctx->buf_len > 0) {
size_t take = chunk_state_fill_buf(ctx, input, input_len);
input += take;
input_len -= take;
if (input_len > 0) {
ops->compress_in_place(ctx->cv, ctx->buf,
BLAKE3_BLOCK_LEN, ctx->chunk_counter,
ctx->flags|chunk_state_maybe_start_flag(ctx));
ctx->blocks_compressed += 1;
ctx->buf_len = 0;
memset(ctx->buf, 0, BLAKE3_BLOCK_LEN);
}
}
while (input_len > BLAKE3_BLOCK_LEN) {
ops->compress_in_place(ctx->cv, input, BLAKE3_BLOCK_LEN,
ctx->chunk_counter,
ctx->flags|chunk_state_maybe_start_flag(ctx));
ctx->blocks_compressed += 1;
input += BLAKE3_BLOCK_LEN;
input_len -= BLAKE3_BLOCK_LEN;
}
size_t take = chunk_state_fill_buf(ctx, input, input_len);
input += take;
input_len -= take;
}
static output_t chunk_state_output(const blake3_chunk_state_t *ctx)
{
uint8_t block_flags =
ctx->flags | chunk_state_maybe_start_flag(ctx) | CHUNK_END;
return (make_output(ctx->cv, ctx->buf, ctx->buf_len, ctx->chunk_counter,
block_flags));
}
static output_t parent_output(const uint8_t block[BLAKE3_BLOCK_LEN],
const uint32_t key[8], uint8_t flags)
{
return (make_output(key, block, BLAKE3_BLOCK_LEN, 0, flags | PARENT));
}
/*
* Given some input larger than one chunk, return the number of bytes that
* should go in the left subtree. This is the largest power-of-2 number of
* chunks that leaves at least 1 byte for the right subtree.
*/
static size_t left_len(size_t content_len)
{
/*
* Subtract 1 to reserve at least one byte for the right side.
* content_len
* should always be greater than BLAKE3_CHUNK_LEN.
*/
size_t full_chunks = (content_len - 1) / BLAKE3_CHUNK_LEN;
return (round_down_to_power_of_2(full_chunks) * BLAKE3_CHUNK_LEN);
}
/*
* Use SIMD parallelism to hash up to MAX_SIMD_DEGREE chunks at the same time
* on a single thread. Write out the chunk chaining values and return the
* number of chunks hashed. These chunks are never the root and never empty;
* those cases use a different codepath.
*/
static size_t compress_chunks_parallel(const blake3_impl_ops_t *ops,
const uint8_t *input, size_t input_len, const uint32_t key[8],
uint64_t chunk_counter, uint8_t flags, uint8_t *out)
{
const uint8_t *chunks_array[MAX_SIMD_DEGREE];
size_t input_position = 0;
size_t chunks_array_len = 0;
while (input_len - input_position >= BLAKE3_CHUNK_LEN) {
chunks_array[chunks_array_len] = &input[input_position];
input_position += BLAKE3_CHUNK_LEN;
chunks_array_len += 1;
}
ops->hash_many(chunks_array, chunks_array_len, BLAKE3_CHUNK_LEN /
BLAKE3_BLOCK_LEN, key, chunk_counter, B_TRUE, flags, CHUNK_START,
CHUNK_END, out);
/*
* Hash the remaining partial chunk, if there is one. Note that the
* empty chunk (meaning the empty message) is a different codepath.
*/
if (input_len > input_position) {
uint64_t counter = chunk_counter + (uint64_t)chunks_array_len;
blake3_chunk_state_t chunk_state;
chunk_state_init(&chunk_state, key, flags);
chunk_state.chunk_counter = counter;
chunk_state_update(ops, &chunk_state, &input[input_position],
input_len - input_position);
output_t output = chunk_state_output(&chunk_state);
output_chaining_value(ops, &output, &out[chunks_array_len *
BLAKE3_OUT_LEN]);
return (chunks_array_len + 1);
} else {
return (chunks_array_len);
}
}
/*
* Use SIMD parallelism to hash up to MAX_SIMD_DEGREE parents at the same time
* on a single thread. Write out the parent chaining values and return the
* number of parents hashed. (If there's an odd input chaining value left over,
* return it as an additional output.) These parents are never the root and
* never empty; those cases use a different codepath.
*/
static size_t compress_parents_parallel(const blake3_impl_ops_t *ops,
const uint8_t *child_chaining_values, size_t num_chaining_values,
const uint32_t key[8], uint8_t flags, uint8_t *out)
{
const uint8_t *parents_array[MAX_SIMD_DEGREE_OR_2];
size_t parents_array_len = 0;
while (num_chaining_values - (2 * parents_array_len) >= 2) {
parents_array[parents_array_len] = &child_chaining_values[2 *
parents_array_len * BLAKE3_OUT_LEN];
parents_array_len += 1;
}
ops->hash_many(parents_array, parents_array_len, 1, key, 0, B_FALSE,
flags | PARENT, 0, 0, out);
/* If there's an odd child left over, it becomes an output. */
if (num_chaining_values > 2 * parents_array_len) {
memcpy(&out[parents_array_len * BLAKE3_OUT_LEN],
&child_chaining_values[2 * parents_array_len *
BLAKE3_OUT_LEN], BLAKE3_OUT_LEN);
return (parents_array_len + 1);
} else {
return (parents_array_len);
}
}
/*
* The wide helper function returns (writes out) an array of chaining values
* and returns the length of that array. The number of chaining values returned
* is the dyanmically detected SIMD degree, at most MAX_SIMD_DEGREE. Or fewer,
* if the input is shorter than that many chunks. The reason for maintaining a
* wide array of chaining values going back up the tree, is to allow the
* implementation to hash as many parents in parallel as possible.
*
* As a special case when the SIMD degree is 1, this function will still return
* at least 2 outputs. This guarantees that this function doesn't perform the
* root compression. (If it did, it would use the wrong flags, and also we
* wouldn't be able to implement exendable ouput.) Note that this function is
* not used when the whole input is only 1 chunk long; that's a different
* codepath.
*
* Why not just have the caller split the input on the first update(), instead
* of implementing this special rule? Because we don't want to limit SIMD or
* multi-threading parallelism for that update().
*/
static size_t blake3_compress_subtree_wide(const blake3_impl_ops_t *ops,
const uint8_t *input, size_t input_len, const uint32_t key[8],
uint64_t chunk_counter, uint8_t flags, uint8_t *out)
{
/*
* Note that the single chunk case does *not* bump the SIMD degree up
* to 2 when it is 1. If this implementation adds multi-threading in
* the future, this gives us the option of multi-threading even the
* 2-chunk case, which can help performance on smaller platforms.
*/
if (input_len <= (size_t)(ops->degree * BLAKE3_CHUNK_LEN)) {
return (compress_chunks_parallel(ops, input, input_len, key,
chunk_counter, flags, out));
}
/*
* With more than simd_degree chunks, we need to recurse. Start by
* dividing the input into left and right subtrees. (Note that this is
* only optimal as long as the SIMD degree is a power of 2. If we ever
* get a SIMD degree of 3 or something, we'll need a more complicated
* strategy.)
*/
size_t left_input_len = left_len(input_len);
size_t right_input_len = input_len - left_input_len;
const uint8_t *right_input = &input[left_input_len];
uint64_t right_chunk_counter = chunk_counter +
(uint64_t)(left_input_len / BLAKE3_CHUNK_LEN);
/*
* Make space for the child outputs. Here we use MAX_SIMD_DEGREE_OR_2
* to account for the special case of returning 2 outputs when the
* SIMD degree is 1.
*/
uint8_t cv_array[2 * MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN];
size_t degree = ops->degree;
if (left_input_len > BLAKE3_CHUNK_LEN && degree == 1) {
/*
* The special case: We always use a degree of at least two,
* to make sure there are two outputs. Except, as noted above,
* at the chunk level, where we allow degree=1. (Note that the
* 1-chunk-input case is a different codepath.)
*/
degree = 2;
}
uint8_t *right_cvs = &cv_array[degree * BLAKE3_OUT_LEN];
/*
* Recurse! If this implementation adds multi-threading support in the
* future, this is where it will go.
*/
size_t left_n = blake3_compress_subtree_wide(ops, input, left_input_len,
key, chunk_counter, flags, cv_array);
size_t right_n = blake3_compress_subtree_wide(ops, right_input,
right_input_len, key, right_chunk_counter, flags, right_cvs);
/*
* The special case again. If simd_degree=1, then we'll have left_n=1
* and right_n=1. Rather than compressing them into a single output,
* return them directly, to make sure we always have at least two
* outputs.
*/
if (left_n == 1) {
memcpy(out, cv_array, 2 * BLAKE3_OUT_LEN);
return (2);
}
/* Otherwise, do one layer of parent node compression. */
size_t num_chaining_values = left_n + right_n;
return compress_parents_parallel(ops, cv_array,
num_chaining_values, key, flags, out);
}
/*
* Hash a subtree with compress_subtree_wide(), and then condense the resulting
* list of chaining values down to a single parent node. Don't compress that
* last parent node, however. Instead, return its message bytes (the
* concatenated chaining values of its children). This is necessary when the
* first call to update() supplies a complete subtree, because the topmost
* parent node of that subtree could end up being the root. It's also necessary
* for extended output in the general case.
*
* As with compress_subtree_wide(), this function is not used on inputs of 1
* chunk or less. That's a different codepath.
*/
static void compress_subtree_to_parent_node(const blake3_impl_ops_t *ops,
const uint8_t *input, size_t input_len, const uint32_t key[8],
uint64_t chunk_counter, uint8_t flags, uint8_t out[2 * BLAKE3_OUT_LEN])
{
uint8_t cv_array[MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN];
size_t num_cvs = blake3_compress_subtree_wide(ops, input, input_len,
key, chunk_counter, flags, cv_array);
/*
* If MAX_SIMD_DEGREE is greater than 2 and there's enough input,
* compress_subtree_wide() returns more than 2 chaining values. Condense
* them into 2 by forming parent nodes repeatedly.
*/
uint8_t out_array[MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN / 2];
while (num_cvs > 2) {
num_cvs = compress_parents_parallel(ops, cv_array, num_cvs, key,
flags, out_array);
memcpy(cv_array, out_array, num_cvs * BLAKE3_OUT_LEN);
}
memcpy(out, cv_array, 2 * BLAKE3_OUT_LEN);
}
static void hasher_init_base(BLAKE3_CTX *ctx, const uint32_t key[8],
uint8_t flags)
{
memcpy(ctx->key, key, BLAKE3_KEY_LEN);
chunk_state_init(&ctx->chunk, key, flags);
ctx->cv_stack_len = 0;
ctx->ops = blake3_impl_get_ops();
}
/*
* As described in hasher_push_cv() below, we do "lazy merging", delaying
* merges until right before the next CV is about to be added. This is
* different from the reference implementation. Another difference is that we
* aren't always merging 1 chunk at a time. Instead, each CV might represent
* any power-of-two number of chunks, as long as the smaller-above-larger
* stack order is maintained. Instead of the "count the trailing 0-bits"
* algorithm described in the spec, we use a "count the total number of
* 1-bits" variant that doesn't require us to retain the subtree size of the
* CV on top of the stack. The principle is the same: each CV that should
* remain in the stack is represented by a 1-bit in the total number of chunks
* (or bytes) so far.
*/
static void hasher_merge_cv_stack(BLAKE3_CTX *ctx, uint64_t total_len)
{
size_t post_merge_stack_len = (size_t)popcnt(total_len);
while (ctx->cv_stack_len > post_merge_stack_len) {
uint8_t *parent_node =
&ctx->cv_stack[(ctx->cv_stack_len - 2) * BLAKE3_OUT_LEN];
output_t output =
parent_output(parent_node, ctx->key, ctx->chunk.flags);
output_chaining_value(ctx->ops, &output, parent_node);
ctx->cv_stack_len -= 1;
}
}
/*
* In reference_impl.rs, we merge the new CV with existing CVs from the stack
* before pushing it. We can do that because we know more input is coming, so
* we know none of the merges are root.
*
* This setting is different. We want to feed as much input as possible to
* compress_subtree_wide(), without setting aside anything for the chunk_state.
* If the user gives us 64 KiB, we want to parallelize over all 64 KiB at once
* as a single subtree, if at all possible.
*
* This leads to two problems:
* 1) This 64 KiB input might be the only call that ever gets made to update.
* In this case, the root node of the 64 KiB subtree would be the root node
* of the whole tree, and it would need to be ROOT finalized. We can't
* compress it until we know.
* 2) This 64 KiB input might complete a larger tree, whose root node is
* similarly going to be the the root of the whole tree. For example, maybe
* we have 196 KiB (that is, 128 + 64) hashed so far. We can't compress the
* node at the root of the 256 KiB subtree until we know how to finalize it.
*
* The second problem is solved with "lazy merging". That is, when we're about
* to add a CV to the stack, we don't merge it with anything first, as the
* reference impl does. Instead we do merges using the *previous* CV that was
* added, which is sitting on top of the stack, and we put the new CV
* (unmerged) on top of the stack afterwards. This guarantees that we never
* merge the root node until finalize().
*
* Solving the first problem requires an additional tool,
* compress_subtree_to_parent_node(). That function always returns the top
* *two* chaining values of the subtree it's compressing. We then do lazy
* merging with each of them separately, so that the second CV will always
* remain unmerged. (That also helps us support extendable output when we're
* hashing an input all-at-once.)
*/
static void hasher_push_cv(BLAKE3_CTX *ctx, uint8_t new_cv[BLAKE3_OUT_LEN],
uint64_t chunk_counter)
{
hasher_merge_cv_stack(ctx, chunk_counter);
memcpy(&ctx->cv_stack[ctx->cv_stack_len * BLAKE3_OUT_LEN], new_cv,
BLAKE3_OUT_LEN);
ctx->cv_stack_len += 1;
}
void
Blake3_Init(BLAKE3_CTX *ctx)
{
hasher_init_base(ctx, BLAKE3_IV, 0);
}
void
Blake3_InitKeyed(BLAKE3_CTX *ctx, const uint8_t key[BLAKE3_KEY_LEN])
{
uint32_t key_words[8];
load_key_words(key, key_words);
hasher_init_base(ctx, key_words, KEYED_HASH);
}
static void
Blake3_Update2(BLAKE3_CTX *ctx, const void *input, size_t input_len)
{
/*
* Explicitly checking for zero avoids causing UB by passing a null
* pointer to memcpy. This comes up in practice with things like:
* std::vector<uint8_t> v;
* blake3_hasher_update(&hasher, v.data(), v.size());
*/
if (input_len == 0) {
return;
}
const uint8_t *input_bytes = (const uint8_t *)input;
/*
* If we have some partial chunk bytes in the internal chunk_state, we
* need to finish that chunk first.
*/
if (chunk_state_len(&ctx->chunk) > 0) {
size_t take = BLAKE3_CHUNK_LEN - chunk_state_len(&ctx->chunk);
if (take > input_len) {
take = input_len;
}
chunk_state_update(ctx->ops, &ctx->chunk, input_bytes, take);
input_bytes += take;
input_len -= take;
/*
* If we've filled the current chunk and there's more coming,
* finalize this chunk and proceed. In this case we know it's
* not the root.
*/
if (input_len > 0) {
output_t output = chunk_state_output(&ctx->chunk);
uint8_t chunk_cv[32];
output_chaining_value(ctx->ops, &output, chunk_cv);
hasher_push_cv(ctx, chunk_cv, ctx->chunk.chunk_counter);
chunk_state_reset(&ctx->chunk, ctx->key,
ctx->chunk.chunk_counter + 1);
} else {
return;
}
}
/*
* Now the chunk_state is clear, and we have more input. If there's
* more than a single chunk (so, definitely not the root chunk), hash
* the largest whole subtree we can, with the full benefits of SIMD
* (and maybe in the future, multi-threading) parallelism. Two
* restrictions:
* - The subtree has to be a power-of-2 number of chunks. Only
* subtrees along the right edge can be incomplete, and we don't know
* where the right edge is going to be until we get to finalize().
* - The subtree must evenly divide the total number of chunks up
* until this point (if total is not 0). If the current incomplete
* subtree is only waiting for 1 more chunk, we can't hash a subtree
* of 4 chunks. We have to complete the current subtree first.
* Because we might need to break up the input to form powers of 2, or
* to evenly divide what we already have, this part runs in a loop.
*/
while (input_len > BLAKE3_CHUNK_LEN) {
size_t subtree_len = round_down_to_power_of_2(input_len);
uint64_t count_so_far =
ctx->chunk.chunk_counter * BLAKE3_CHUNK_LEN;
/*
* Shrink the subtree_len until it evenly divides the count so
* far. We know that subtree_len itself is a power of 2, so we
* can use a bitmasking trick instead of an actual remainder
* operation. (Note that if the caller consistently passes
* power-of-2 inputs of the same size, as is hopefully
* typical, this loop condition will always fail, and
* subtree_len will always be the full length of the input.)
*
* An aside: We don't have to shrink subtree_len quite this
* much. For example, if count_so_far is 1, we could pass 2
* chunks to compress_subtree_to_parent_node. Since we'll get
* 2 CVs back, we'll still get the right answer in the end,
* and we might get to use 2-way SIMD parallelism. The problem
* with this optimization, is that it gets us stuck always
* hashing 2 chunks. The total number of chunks will remain
* odd, and we'll never graduate to higher degrees of
* parallelism. See
* https://github.com/BLAKE3-team/BLAKE3/issues/69.
*/
while ((((uint64_t)(subtree_len - 1)) & count_so_far) != 0) {
subtree_len /= 2;
}
/*
* The shrunken subtree_len might now be 1 chunk long. If so,
* hash that one chunk by itself. Otherwise, compress the
* subtree into a pair of CVs.
*/
uint64_t subtree_chunks = subtree_len / BLAKE3_CHUNK_LEN;
if (subtree_len <= BLAKE3_CHUNK_LEN) {
blake3_chunk_state_t chunk_state;
chunk_state_init(&chunk_state, ctx->key,
ctx->chunk.flags);
chunk_state.chunk_counter = ctx->chunk.chunk_counter;
chunk_state_update(ctx->ops, &chunk_state, input_bytes,
subtree_len);
output_t output = chunk_state_output(&chunk_state);
uint8_t cv[BLAKE3_OUT_LEN];
output_chaining_value(ctx->ops, &output, cv);
hasher_push_cv(ctx, cv, chunk_state.chunk_counter);
} else {
/*
* This is the high-performance happy path, though
* getting here depends on the caller giving us a long
* enough input.
*/
uint8_t cv_pair[2 * BLAKE3_OUT_LEN];
compress_subtree_to_parent_node(ctx->ops, input_bytes,
subtree_len, ctx->key, ctx-> chunk.chunk_counter,
ctx->chunk.flags, cv_pair);
hasher_push_cv(ctx, cv_pair, ctx->chunk.chunk_counter);
hasher_push_cv(ctx, &cv_pair[BLAKE3_OUT_LEN],
ctx->chunk.chunk_counter + (subtree_chunks / 2));
}
ctx->chunk.chunk_counter += subtree_chunks;
input_bytes += subtree_len;
input_len -= subtree_len;
}
/*
* If there's any remaining input less than a full chunk, add it to
* the chunk state. In that case, also do a final merge loop to make
* sure the subtree stack doesn't contain any unmerged pairs. The
* remaining input means we know these merges are non-root. This merge
* loop isn't strictly necessary here, because hasher_push_chunk_cv
* already does its own merge loop, but it simplifies
* blake3_hasher_finalize below.
*/
if (input_len > 0) {
chunk_state_update(ctx->ops, &ctx->chunk, input_bytes,
input_len);
hasher_merge_cv_stack(ctx, ctx->chunk.chunk_counter);
}
}
void
Blake3_Update(BLAKE3_CTX *ctx, const void *input, size_t todo)
{
size_t done = 0;
const uint8_t *data = input;
const size_t block_max = 1024 * 64;
/* max feed buffer to leave the stack size small */
while (todo != 0) {
size_t block = (todo >= block_max) ? block_max : todo;
Blake3_Update2(ctx, data + done, block);
done += block;
todo -= block;
}
}
void
Blake3_Final(const BLAKE3_CTX *ctx, uint8_t *out)
{
Blake3_FinalSeek(ctx, 0, out, BLAKE3_OUT_LEN);
}
void
Blake3_FinalSeek(const BLAKE3_CTX *ctx, uint64_t seek, uint8_t *out,
size_t out_len)
{
/*
* Explicitly checking for zero avoids causing UB by passing a null
* pointer to memcpy. This comes up in practice with things like:
* std::vector<uint8_t> v;
* blake3_hasher_finalize(&hasher, v.data(), v.size());
*/
if (out_len == 0) {
return;
}
/* If the subtree stack is empty, then the current chunk is the root. */
if (ctx->cv_stack_len == 0) {
output_t output = chunk_state_output(&ctx->chunk);
output_root_bytes(ctx->ops, &output, seek, out, out_len);
return;
}
/*
* If there are any bytes in the chunk state, finalize that chunk and
* do a roll-up merge between that chunk hash and every subtree in the
* stack. In this case, the extra merge loop at the end of
* blake3_hasher_update guarantees that none of the subtrees in the
* stack need to be merged with each other first. Otherwise, if there
* are no bytes in the chunk state, then the top of the stack is a
* chunk hash, and we start the merge from that.
*/
output_t output;
size_t cvs_remaining;
if (chunk_state_len(&ctx->chunk) > 0) {
cvs_remaining = ctx->cv_stack_len;
output = chunk_state_output(&ctx->chunk);
} else {
/* There are always at least 2 CVs in the stack in this case. */
cvs_remaining = ctx->cv_stack_len - 2;
output = parent_output(&ctx->cv_stack[cvs_remaining * 32],
ctx->key, ctx->chunk.flags);
}
while (cvs_remaining > 0) {
cvs_remaining -= 1;
uint8_t parent_block[BLAKE3_BLOCK_LEN];
memcpy(parent_block, &ctx->cv_stack[cvs_remaining * 32], 32);
output_chaining_value(ctx->ops, &output, &parent_block[32]);
output = parent_output(parent_block, ctx->key,
ctx->chunk.flags);
}
output_root_bytes(ctx->ops, &output, seek, out, out_len);
}

202
module/icp/algs/blake3/blake3_generic.c Normal file
View File

@ -0,0 +1,202 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Based on BLAKE3 v1.3.1, https://github.com/BLAKE3-team/BLAKE3
* Copyright (c) 2019-2020 Samuel Neves and Jack O'Connor
* Copyright (c) 2021-2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include <sys/zfs_context.h>
#include "blake3_impl.h"
#define rotr32(x, n) (((x) >> (n)) | ((x) << (32 - (n))))
static inline void g(uint32_t *state, size_t a, size_t b, size_t c, size_t d,
uint32_t x, uint32_t y)
{
state[a] = state[a] + state[b] + x;
state[d] = rotr32(state[d] ^ state[a], 16);
state[c] = state[c] + state[d];
state[b] = rotr32(state[b] ^ state[c], 12);
state[a] = state[a] + state[b] + y;
state[d] = rotr32(state[d] ^ state[a], 8);
state[c] = state[c] + state[d];
state[b] = rotr32(state[b] ^ state[c], 7);
}
static inline void round_fn(uint32_t state[16], const uint32_t *msg,
size_t round)
{
/* Select the message schedule based on the round. */
const uint8_t *schedule = BLAKE3_MSG_SCHEDULE[round];
/* Mix the columns. */
g(state, 0, 4, 8, 12, msg[schedule[0]], msg[schedule[1]]);
g(state, 1, 5, 9, 13, msg[schedule[2]], msg[schedule[3]]);
g(state, 2, 6, 10, 14, msg[schedule[4]], msg[schedule[5]]);
g(state, 3, 7, 11, 15, msg[schedule[6]], msg[schedule[7]]);
/* Mix the rows. */
g(state, 0, 5, 10, 15, msg[schedule[8]], msg[schedule[9]]);
g(state, 1, 6, 11, 12, msg[schedule[10]], msg[schedule[11]]);
g(state, 2, 7, 8, 13, msg[schedule[12]], msg[schedule[13]]);
g(state, 3, 4, 9, 14, msg[schedule[14]], msg[schedule[15]]);
}
static inline void compress_pre(uint32_t state[16], const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags)
{
uint32_t block_words[16];
block_words[0] = load32(block + 4 * 0);
block_words[1] = load32(block + 4 * 1);
block_words[2] = load32(block + 4 * 2);
block_words[3] = load32(block + 4 * 3);
block_words[4] = load32(block + 4 * 4);
block_words[5] = load32(block + 4 * 5);
block_words[6] = load32(block + 4 * 6);
block_words[7] = load32(block + 4 * 7);
block_words[8] = load32(block + 4 * 8);
block_words[9] = load32(block + 4 * 9);
block_words[10] = load32(block + 4 * 10);
block_words[11] = load32(block + 4 * 11);
block_words[12] = load32(block + 4 * 12);
block_words[13] = load32(block + 4 * 13);
block_words[14] = load32(block + 4 * 14);
block_words[15] = load32(block + 4 * 15);
state[0] = cv[0];
state[1] = cv[1];
state[2] = cv[2];
state[3] = cv[3];
state[4] = cv[4];
state[5] = cv[5];
state[6] = cv[6];
state[7] = cv[7];
state[8] = BLAKE3_IV[0];
state[9] = BLAKE3_IV[1];
state[10] = BLAKE3_IV[2];
state[11] = BLAKE3_IV[3];
state[12] = counter_low(counter);
state[13] = counter_high(counter);
state[14] = (uint32_t)block_len;
state[15] = (uint32_t)flags;
round_fn(state, &block_words[0], 0);
round_fn(state, &block_words[0], 1);
round_fn(state, &block_words[0], 2);
round_fn(state, &block_words[0], 3);
round_fn(state, &block_words[0], 4);
round_fn(state, &block_words[0], 5);
round_fn(state, &block_words[0], 6);
}
static inline void blake3_compress_in_place_generic(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags)
{
uint32_t state[16];
compress_pre(state, cv, block, block_len, counter, flags);
cv[0] = state[0] ^ state[8];
cv[1] = state[1] ^ state[9];
cv[2] = state[2] ^ state[10];
cv[3] = state[3] ^ state[11];
cv[4] = state[4] ^ state[12];
cv[5] = state[5] ^ state[13];
cv[6] = state[6] ^ state[14];
cv[7] = state[7] ^ state[15];
}
static inline void hash_one_generic(const uint8_t *input, size_t blocks,
const uint32_t key[8], uint64_t counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t out[BLAKE3_OUT_LEN])
{
uint32_t cv[8];
memcpy(cv, key, BLAKE3_KEY_LEN);
uint8_t block_flags = flags | flags_start;
while (blocks > 0) {
if (blocks == 1) {
block_flags |= flags_end;
}
blake3_compress_in_place_generic(cv, input, BLAKE3_BLOCK_LEN,
counter, block_flags);
input = &input[BLAKE3_BLOCK_LEN];
blocks -= 1;
block_flags = flags;
}
store_cv_words(out, cv);
}
static inline void blake3_compress_xof_generic(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64])
{
uint32_t state[16];
compress_pre(state, cv, block, block_len, counter, flags);
store32(&out[0 * 4], state[0] ^ state[8]);
store32(&out[1 * 4], state[1] ^ state[9]);
store32(&out[2 * 4], state[2] ^ state[10]);
store32(&out[3 * 4], state[3] ^ state[11]);
store32(&out[4 * 4], state[4] ^ state[12]);
store32(&out[5 * 4], state[5] ^ state[13]);
store32(&out[6 * 4], state[6] ^ state[14]);
store32(&out[7 * 4], state[7] ^ state[15]);
store32(&out[8 * 4], state[8] ^ cv[0]);
store32(&out[9 * 4], state[9] ^ cv[1]);
store32(&out[10 * 4], state[10] ^ cv[2]);
store32(&out[11 * 4], state[11] ^ cv[3]);
store32(&out[12 * 4], state[12] ^ cv[4]);
store32(&out[13 * 4], state[13] ^ cv[5]);
store32(&out[14 * 4], state[14] ^ cv[6]);
store32(&out[15 * 4], state[15] ^ cv[7]);
}
static inline void blake3_hash_many_generic(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8], uint64_t counter,
boolean_t increment_counter, uint8_t flags, uint8_t flags_start,
uint8_t flags_end, uint8_t *out)
{
while (num_inputs > 0) {
hash_one_generic(inputs[0], blocks, key, counter, flags,
flags_start, flags_end, out);
if (increment_counter) {
counter += 1;
}
inputs += 1;
num_inputs -= 1;
out = &out[BLAKE3_OUT_LEN];
}
}
static inline boolean_t blake3_is_generic_supported(void)
{
return (B_TRUE);
}
const blake3_impl_ops_t blake3_generic_impl = {
.compress_in_place = blake3_compress_in_place_generic,
.compress_xof = blake3_compress_xof_generic,
.hash_many = blake3_hash_many_generic,
.is_supported = blake3_is_generic_supported,
.degree = 4,
.name = "generic"
};

256
module/icp/algs/blake3/blake3_impl.c Normal file
View File

@ -0,0 +1,256 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Copyright (c) 2021-2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include <sys/zfs_context.h>
#include <sys/zio_checksum.h>
#include "blake3_impl.h"
static const blake3_impl_ops_t *const blake3_impls[] = {
&blake3_generic_impl,
#if defined(__aarch64__) || \
(defined(__x86_64) && defined(HAVE_SSE2)) || \
(defined(__PPC64__) && defined(__LITTLE_ENDIAN__))
&blake3_sse2_impl,
#endif
#if defined(__aarch64__) || \
(defined(__x86_64) && defined(HAVE_SSE4_1)) || \
(defined(__PPC64__) && defined(__LITTLE_ENDIAN__))
&blake3_sse41_impl,
#endif
#if defined(__x86_64) && defined(HAVE_SSE4_1) && defined(HAVE_AVX2)
&blake3_avx2_impl,
#endif
#if defined(__x86_64) && defined(HAVE_AVX512F) && defined(HAVE_AVX512VL)
&blake3_avx512_impl,
#endif
};
/* this pointer holds current ops for implementation */
static const blake3_impl_ops_t *blake3_selected_impl = &blake3_generic_impl;
/* special implementation selections */
#define IMPL_FASTEST (UINT32_MAX)
#define IMPL_CYCLE (UINT32_MAX-1)
#define IMPL_USER (UINT32_MAX-2)
#define IMPL_PARAM (UINT32_MAX-3)
#define IMPL_READ(i) (*(volatile uint32_t *) &(i))
static uint32_t icp_blake3_impl = IMPL_FASTEST;
#define BLAKE3_IMPL_NAME_MAX 16
/* id of fastest implementation */
static uint32_t blake3_fastest_id = 0;
/* currently used id */
static uint32_t blake3_current_id = 0;
/* id of module parameter (-1 == unused) */
static int blake3_param_id = -1;
/* return number of supported implementations */
int
blake3_get_impl_count(void)
{
static int impls = 0;
int i;
if (impls)
return (impls);
for (i = 0; i < ARRAY_SIZE(blake3_impls); i++) {
if (!blake3_impls[i]->is_supported()) continue;
impls++;
}
return (impls);
}
/* return id of selected implementation */
int
blake3_get_impl_id(void)
{
return (blake3_current_id);
}
/* return name of selected implementation */
const char *
blake3_get_impl_name(void)
{
return (blake3_selected_impl->name);
}
/* setup id as fastest implementation */
void
blake3_set_impl_fastest(uint32_t id)
{
blake3_fastest_id = id;
}
/* set implementation by id */
void
blake3_set_impl_id(uint32_t id)
{
int i, cid;
/* select fastest */
if (id == IMPL_FASTEST)
id = blake3_fastest_id;
/* select next or first */
if (id == IMPL_CYCLE)
id = (++blake3_current_id) % blake3_get_impl_count();
/* 0..N for the real impl */
for (i = 0, cid = 0; i < ARRAY_SIZE(blake3_impls); i++) {
if (!blake3_impls[i]->is_supported()) continue;
if (cid == id) {
blake3_current_id = cid;
blake3_selected_impl = blake3_impls[i];
return;
}
cid++;
}
}
/* set implementation by name */
int
blake3_set_impl_name(const char *name)
{
int i, cid;
if (strcmp(name, "fastest") == 0) {
atomic_swap_32(&icp_blake3_impl, IMPL_FASTEST);
blake3_set_impl_id(IMPL_FASTEST);
return (0);
} else if (strcmp(name, "cycle") == 0) {
atomic_swap_32(&icp_blake3_impl, IMPL_CYCLE);
blake3_set_impl_id(IMPL_CYCLE);
return (0);
}
for (i = 0, cid = 0; i < ARRAY_SIZE(blake3_impls); i++) {
if (!blake3_impls[i]->is_supported()) continue;
if (strcmp(name, blake3_impls[i]->name) == 0) {
if (icp_blake3_impl == IMPL_PARAM) {
blake3_param_id = cid;
return (0);
}
blake3_selected_impl = blake3_impls[i];
blake3_current_id = cid;
return (0);
}
cid++;
}
return (-EINVAL);
}
/* setup implementation */
void
blake3_setup_impl(void)
{
switch (IMPL_READ(icp_blake3_impl)) {
case IMPL_PARAM:
blake3_set_impl_id(blake3_param_id);
atomic_swap_32(&icp_blake3_impl, IMPL_USER);
break;
case IMPL_FASTEST:
blake3_set_impl_id(IMPL_FASTEST);
break;
case IMPL_CYCLE:
blake3_set_impl_id(IMPL_CYCLE);
break;
default:
blake3_set_impl_id(blake3_current_id);
break;
}
}
/* return selected implementation */
const blake3_impl_ops_t *
blake3_impl_get_ops(void)
{
/* each call to ops will cycle */
if (icp_blake3_impl == IMPL_CYCLE)
blake3_set_impl_id(IMPL_CYCLE);
return (blake3_selected_impl);
}
#if defined(_KERNEL) && defined(__linux__)
static int
icp_blake3_impl_set(const char *name, zfs_kernel_param_t *kp)
{
char req_name[BLAKE3_IMPL_NAME_MAX];
size_t i;
/* sanitize input */
i = strnlen(name, BLAKE3_IMPL_NAME_MAX);
if (i == 0 || i >= BLAKE3_IMPL_NAME_MAX)
return (-EINVAL);
strlcpy(req_name, name, BLAKE3_IMPL_NAME_MAX);
while (i > 0 && isspace(req_name[i-1]))
i--;
req_name[i] = '\0';
atomic_swap_32(&icp_blake3_impl, IMPL_PARAM);
return (blake3_set_impl_name(req_name));
}
static int
icp_blake3_impl_get(char *buffer, zfs_kernel_param_t *kp)
{
int i, cid, cnt = 0;
char *fmt;
/* cycling */
fmt = (icp_blake3_impl == IMPL_CYCLE) ? "[cycle] " : "cycle ";
cnt += sprintf(buffer + cnt, fmt);
/* fastest one */
fmt = (icp_blake3_impl == IMPL_FASTEST) ? "[fastest] " : "fastest ";
cnt += sprintf(buffer + cnt, fmt);
/* user selected */
for (i = 0, cid = 0; i < ARRAY_SIZE(blake3_impls); i++) {
if (!blake3_impls[i]->is_supported()) continue;
fmt = (icp_blake3_impl == IMPL_USER &&
cid == blake3_current_id) ? "[%s] " : "%s ";
cnt += sprintf(buffer + cnt, fmt, blake3_impls[i]->name);
cid++;
}
buffer[cnt] = 0;
return (cnt);
}
module_param_call(icp_blake3_impl, icp_blake3_impl_set, icp_blake3_impl_get,
NULL, 0644);
MODULE_PARM_DESC(icp_blake3_impl, "Select BLAKE3 implementation.");
#endif

213
module/icp/algs/blake3/blake3_impl.h Normal file
View File

@ -0,0 +1,213 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Based on BLAKE3 v1.3.1, https://github.com/BLAKE3-team/BLAKE3
* Copyright (c) 2019-2020 Samuel Neves and Jack O'Connor
* Copyright (c) 2021-2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#ifndef BLAKE3_IMPL_H
#define BLAKE3_IMPL_H
#ifdef __cplusplus
extern "C" {
#endif
#include <sys/types.h>
#include <sys/blake3.h>
#include <sys/simd.h>
/*
* Methods used to define BLAKE3 assembler implementations
*/
typedef void (*blake3_compress_in_place_f)(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN],
uint8_t block_len, uint64_t counter,
uint8_t flags);
typedef void (*blake3_compress_xof_f)(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]);
typedef void (*blake3_hash_many_f)(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out);
typedef boolean_t (*blake3_is_supported_f)(void);
typedef struct blake3_impl_ops {
blake3_compress_in_place_f compress_in_place;
blake3_compress_xof_f compress_xof;
blake3_hash_many_f hash_many;
blake3_is_supported_f is_supported;
int degree;
const char *name;
} blake3_impl_ops_t;
/* Return selected BLAKE3 implementation ops */
extern const blake3_impl_ops_t *blake3_impl_get_ops(void);
extern const blake3_impl_ops_t blake3_generic_impl;
#if defined(__aarch64__) || \
(defined(__x86_64) && defined(HAVE_SSE2)) || \
(defined(__PPC64__) && defined(__LITTLE_ENDIAN__))
extern const blake3_impl_ops_t blake3_sse2_impl;
#endif
#if defined(__aarch64__) || \
(defined(__x86_64) && defined(HAVE_SSE4_1)) || \
(defined(__PPC64__) && defined(__LITTLE_ENDIAN__))
extern const blake3_impl_ops_t blake3_sse41_impl;
#endif
#if defined(__x86_64) && defined(HAVE_SSE4_1) && defined(HAVE_AVX2)
extern const blake3_impl_ops_t blake3_avx2_impl;
#endif
#if defined(__x86_64) && defined(HAVE_AVX512F) && defined(HAVE_AVX512VL)
extern const blake3_impl_ops_t blake3_avx512_impl;
#endif
#if defined(__x86_64)
#define MAX_SIMD_DEGREE 16
#else
#define MAX_SIMD_DEGREE 4
#endif
#define MAX_SIMD_DEGREE_OR_2 (MAX_SIMD_DEGREE > 2 ? MAX_SIMD_DEGREE : 2)
static const uint32_t BLAKE3_IV[8] = {
0x6A09E667UL, 0xBB67AE85UL, 0x3C6EF372UL, 0xA54FF53AUL,
0x510E527FUL, 0x9B05688CUL, 0x1F83D9ABUL, 0x5BE0CD19UL};
static const uint8_t BLAKE3_MSG_SCHEDULE[7][16] = {
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15},
{2, 6, 3, 10, 7, 0, 4, 13, 1, 11, 12, 5, 9, 14, 15, 8},
{3, 4, 10, 12, 13, 2, 7, 14, 6, 5, 9, 0, 11, 15, 8, 1},
{10, 7, 12, 9, 14, 3, 13, 15, 4, 0, 11, 2, 5, 8, 1, 6},
{12, 13, 9, 11, 15, 10, 14, 8, 7, 2, 5, 3, 0, 1, 6, 4},
{9, 14, 11, 5, 8, 12, 15, 1, 13, 3, 0, 10, 2, 6, 4, 7},
{11, 15, 5, 0, 1, 9, 8, 6, 14, 10, 2, 12, 3, 4, 7, 13},
};
/* Find index of the highest set bit */
static inline unsigned int highest_one(uint64_t x) {
#if defined(__GNUC__) || defined(__clang__)
return (63 ^ __builtin_clzll(x));
#elif defined(_MSC_VER) && defined(IS_X86_64)
unsigned long index;
_BitScanReverse64(&index, x);
return (index);
#elif defined(_MSC_VER) && defined(IS_X86_32)
if (x >> 32) {
unsigned long index;
_BitScanReverse(&index, x >> 32);
return (32 + index);
} else {
unsigned long index;
_BitScanReverse(&index, x);
return (index);
}
#else
unsigned int c = 0;
if (x & 0xffffffff00000000ULL) { x >>= 32; c += 32; }
if (x & 0x00000000ffff0000ULL) { x >>= 16; c += 16; }
if (x & 0x000000000000ff00ULL) { x >>= 8; c += 8; }
if (x & 0x00000000000000f0ULL) { x >>= 4; c += 4; }
if (x & 0x000000000000000cULL) { x >>= 2; c += 2; }
if (x & 0x0000000000000002ULL) { c += 1; }
return (c);
#endif
}
/* Count the number of 1 bits. */
static inline unsigned int popcnt(uint64_t x) {
unsigned int count = 0;
while (x != 0) {
count += 1;
x &= x - 1;
}
return (count);
}
/*
* Largest power of two less than or equal to x.
* As a special case, returns 1 when x is 0.
*/
static inline uint64_t round_down_to_power_of_2(uint64_t x) {
return (1ULL << highest_one(x | 1));
}
static inline uint32_t counter_low(uint64_t counter) {
return ((uint32_t)counter);
}
static inline uint32_t counter_high(uint64_t counter) {
return ((uint32_t)(counter >> 32));
}
static inline uint32_t load32(const void *src) {
const uint8_t *p = (const uint8_t *)src;
return ((uint32_t)(p[0]) << 0) | ((uint32_t)(p[1]) << 8) |
((uint32_t)(p[2]) << 16) | ((uint32_t)(p[3]) << 24);
}
static inline void load_key_words(const uint8_t key[BLAKE3_KEY_LEN],
uint32_t key_words[8]) {
key_words[0] = load32(&key[0 * 4]);
key_words[1] = load32(&key[1 * 4]);
key_words[2] = load32(&key[2 * 4]);
key_words[3] = load32(&key[3 * 4]);
key_words[4] = load32(&key[4 * 4]);
key_words[5] = load32(&key[5 * 4]);
key_words[6] = load32(&key[6 * 4]);
key_words[7] = load32(&key[7 * 4]);
}
static inline void store32(void *dst, uint32_t w) {
uint8_t *p = (uint8_t *)dst;
p[0] = (uint8_t)(w >> 0);
p[1] = (uint8_t)(w >> 8);
p[2] = (uint8_t)(w >> 16);
p[3] = (uint8_t)(w >> 24);
}
static inline void store_cv_words(uint8_t bytes_out[32], uint32_t cv_words[8]) {
store32(&bytes_out[0 * 4], cv_words[0]);
store32(&bytes_out[1 * 4], cv_words[1]);
store32(&bytes_out[2 * 4], cv_words[2]);
store32(&bytes_out[3 * 4], cv_words[3]);
store32(&bytes_out[4 * 4], cv_words[4]);
store32(&bytes_out[5 * 4], cv_words[5]);
store32(&bytes_out[6 * 4], cv_words[6]);
store32(&bytes_out[7 * 4], cv_words[7]);
}
#ifdef __cplusplus
}
#endif
#endif /* BLAKE3_IMPL_H */

248
module/icp/algs/blake3/blake3_x86-64.c Normal file
View File

@ -0,0 +1,248 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Copyright (c) 2021-2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include "blake3_impl.h"
#if defined(__aarch64__) || \
(defined(__x86_64) && defined(HAVE_SSE2)) || \
(defined(__PPC64__) && defined(__LITTLE_ENDIAN__))
extern void zfs_blake3_compress_in_place_sse2(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags);
extern void zfs_blake3_compress_xof_sse2(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]);
extern void zfs_blake3_hash_many_sse2(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out);
static void blake3_compress_in_place_sse2(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags) {
kfpu_begin();
zfs_blake3_compress_in_place_sse2(cv, block, block_len, counter,
flags);
kfpu_end();
}
static void blake3_compress_xof_sse2(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]) {
kfpu_begin();
zfs_blake3_compress_xof_sse2(cv, block, block_len, counter, flags,
out);
kfpu_end();
}
static void blake3_hash_many_sse2(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out) {
kfpu_begin();
zfs_blake3_hash_many_sse2(inputs, num_inputs, blocks, key, counter,
increment_counter, flags, flags_start, flags_end, out);
kfpu_end();
}
static boolean_t blake3_is_sse2_supported(void)
{
#if defined(__x86_64)
return (kfpu_allowed() && zfs_sse2_available());
#elif defined(__PPC64__)
return (kfpu_allowed() && zfs_vsx_available());
#else
return (kfpu_allowed());
#endif
}
const blake3_impl_ops_t blake3_sse2_impl = {
.compress_in_place = blake3_compress_in_place_sse2,
.compress_xof = blake3_compress_xof_sse2,
.hash_many = blake3_hash_many_sse2,
.is_supported = blake3_is_sse2_supported,
.degree = 4,
.name = "sse2"
};
#endif
#if defined(__aarch64__) || \
(defined(__x86_64) && defined(HAVE_SSE2)) || \
(defined(__PPC64__) && defined(__LITTLE_ENDIAN__))
extern void zfs_blake3_compress_in_place_sse41(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags);
extern void zfs_blake3_compress_xof_sse41(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]);
extern void zfs_blake3_hash_many_sse41(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out);
static void blake3_compress_in_place_sse41(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags) {
kfpu_begin();
zfs_blake3_compress_in_place_sse41(cv, block, block_len, counter,
flags);
kfpu_end();
}
static void blake3_compress_xof_sse41(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]) {
kfpu_begin();
zfs_blake3_compress_xof_sse41(cv, block, block_len, counter, flags,
out);
kfpu_end();
}
static void blake3_hash_many_sse41(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out) {
kfpu_begin();
zfs_blake3_hash_many_sse41(inputs, num_inputs, blocks, key, counter,
increment_counter, flags, flags_start, flags_end, out);
kfpu_end();
}
static boolean_t blake3_is_sse41_supported(void)
{
#if defined(__x86_64)
return (kfpu_allowed() && zfs_sse4_1_available());
#elif defined(__PPC64__)
return (kfpu_allowed() && zfs_vsx_available());
#else
return (kfpu_allowed());
#endif
}
const blake3_impl_ops_t blake3_sse41_impl = {
.compress_in_place = blake3_compress_in_place_sse41,
.compress_xof = blake3_compress_xof_sse41,
.hash_many = blake3_hash_many_sse41,
.is_supported = blake3_is_sse41_supported,
.degree = 4,
.name = "sse41"
};
#endif
#if defined(__x86_64) && defined(HAVE_SSE4_1) && defined(HAVE_AVX2)
extern void zfs_blake3_hash_many_avx2(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out);
static void blake3_hash_many_avx2(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out) {
kfpu_begin();
zfs_blake3_hash_many_avx2(inputs, num_inputs, blocks, key, counter,
increment_counter, flags, flags_start, flags_end, out);
kfpu_end();
}
static boolean_t blake3_is_avx2_supported(void)
{
return (kfpu_allowed() && zfs_sse4_1_available() &&
zfs_avx2_available());
}
const blake3_impl_ops_t blake3_avx2_impl = {
.compress_in_place = blake3_compress_in_place_sse41,
.compress_xof = blake3_compress_xof_sse41,
.hash_many = blake3_hash_many_avx2,
.is_supported = blake3_is_avx2_supported,
.degree = 8,
.name = "avx2"
};
#endif
#if defined(__x86_64) && defined(HAVE_AVX512F) && defined(HAVE_AVX512VL)
extern void zfs_blake3_compress_in_place_avx512(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags);
extern void zfs_blake3_compress_xof_avx512(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]);
extern void zfs_blake3_hash_many_avx512(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out);
static void blake3_compress_in_place_avx512(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags) {
kfpu_begin();
zfs_blake3_compress_in_place_avx512(cv, block, block_len, counter,
flags);
kfpu_end();
}
static void blake3_compress_xof_avx512(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]) {
kfpu_begin();
zfs_blake3_compress_xof_avx512(cv, block, block_len, counter, flags,
out);
kfpu_end();
}
static void blake3_hash_many_avx512(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out) {
kfpu_begin();
zfs_blake3_hash_many_avx512(inputs, num_inputs, blocks, key, counter,
increment_counter, flags, flags_start, flags_end, out);
kfpu_end();
}
static boolean_t blake3_is_avx512_supported(void)
{
return (kfpu_allowed() && zfs_avx512f_available() &&
zfs_avx512vl_available());
}
const blake3_impl_ops_t blake3_avx512_impl = {
.compress_in_place = blake3_compress_in_place_avx512,
.compress_xof = blake3_compress_xof_avx512,
.hash_many = blake3_hash_many_avx512,
.is_supported = blake3_is_avx512_supported,
.degree = 16,
.name = "avx512"
};
#endif

2450
module/icp/asm-aarch64/blake3/b3_aarch64_sse2.S Normal file
View File

File diff suppressed because it is too large Load Diff

2463
module/icp/asm-aarch64/blake3/b3_aarch64_sse41.S Normal file
View File

File diff suppressed because it is too large Load Diff

2823
module/icp/asm-ppc64/blake3/b3_ppc64le_sse2.S Normal file
View File

File diff suppressed because it is too large Load Diff

3064
module/icp/asm-ppc64/blake3/b3_ppc64le_sse41.S Normal file
View File

File diff suppressed because it is too large Load Diff

1845
module/icp/asm-x86_64/blake3/blake3_avx2.S Normal file
View File

File diff suppressed because it is too large Load Diff

2618
module/icp/asm-x86_64/blake3/blake3_avx512.S Normal file
View File

File diff suppressed because it is too large Load Diff

2323
module/icp/asm-x86_64/blake3/blake3_sse2.S Normal file
View File

File diff suppressed because it is too large Load Diff

2058
module/icp/asm-x86_64/blake3/blake3_sse41.S Normal file
View File

File diff suppressed because it is too large Load Diff

31
module/zcommon/zfeature_common.c
View File

@ -696,16 +696,15 @@ zpool_feature_init(void)
ZFEATURE_FLAG_MOS, ZFEATURE_TYPE_BOOLEAN, NULL, sfeatures);
{
static const spa_feature_t zilsaxattr_deps[] = {
SPA_FEATURE_EXTENSIBLE_DATASET,
SPA_FEATURE_NONE
};
zfeature_register(SPA_FEATURE_ZILSAXATTR,
"org.openzfs:zilsaxattr", "zilsaxattr",
"Support for xattr=sa extended attribute logging in ZIL.",
ZFEATURE_FLAG_PER_DATASET | ZFEATURE_FLAG_READONLY_COMPAT,
ZFEATURE_TYPE_BOOLEAN, zilsaxattr_deps, sfeatures);
static const spa_feature_t zilsaxattr_deps[] = {
SPA_FEATURE_EXTENSIBLE_DATASET,
SPA_FEATURE_NONE
};
zfeature_register(SPA_FEATURE_ZILSAXATTR,
"org.openzfs:zilsaxattr", "zilsaxattr",
"Support for xattr=sa extended attribute logging in ZIL.",
ZFEATURE_FLAG_PER_DATASET | ZFEATURE_FLAG_READONLY_COMPAT,
ZFEATURE_TYPE_BOOLEAN, zilsaxattr_deps, sfeatures);
}
zfeature_register(SPA_FEATURE_HEAD_ERRLOG,
@ -714,6 +713,18 @@ zpool_feature_init(void)
ZFEATURE_FLAG_ACTIVATE_ON_ENABLE, ZFEATURE_TYPE_BOOLEAN, NULL,
sfeatures);
{
static const spa_feature_t blake3_deps[] = {
SPA_FEATURE_EXTENSIBLE_DATASET,
SPA_FEATURE_NONE
};
zfeature_register(SPA_FEATURE_BLAKE3,
"org.openzfs:blake3", "blake3",
"BLAKE3 hash algorithm.",
ZFEATURE_FLAG_PER_DATASET, ZFEATURE_TYPE_BOOLEAN,
blake3_deps, sfeatures);
}
zfs_mod_list_supported_free(sfeatures);
}

8
module/zcommon/zfs_prop.c
View File

@ -84,6 +84,7 @@ zfs_prop_init(void)
{ "sha512", ZIO_CHECKSUM_SHA512 },
{ "skein", ZIO_CHECKSUM_SKEIN },
{ "edonr", ZIO_CHECKSUM_EDONR },
{ "blake3", ZIO_CHECKSUM_BLAKE3 },
{ NULL }
};
@ -102,6 +103,9 @@ zfs_prop_init(void)
ZIO_CHECKSUM_SKEIN | ZIO_CHECKSUM_VERIFY },
{ "edonr,verify",
ZIO_CHECKSUM_EDONR | ZIO_CHECKSUM_VERIFY },
{ "blake3", ZIO_CHECKSUM_BLAKE3 },
{ "blake3,verify",
ZIO_CHECKSUM_BLAKE3 | ZIO_CHECKSUM_VERIFY },
{ NULL }
};
@ -394,12 +398,12 @@ zfs_prop_init(void)
ZIO_CHECKSUM_DEFAULT, PROP_INHERIT, ZFS_TYPE_FILESYSTEM |
ZFS_TYPE_VOLUME,
"on | off | fletcher2 | fletcher4 | sha256 | sha512 | skein"
" | edonr",
" | edonr | blake3",
"CHECKSUM", checksum_table, sfeatures);
zprop_register_index(ZFS_PROP_DEDUP, "dedup", ZIO_CHECKSUM_OFF,
PROP_INHERIT, ZFS_TYPE_FILESYSTEM | ZFS_TYPE_VOLUME,
"on | off | verify | sha256[,verify] | sha512[,verify] | "
"skein[,verify] | edonr,verify",
"skein[,verify] | edonr,verify | blake3[,verify]",
"DEDUP", dedup_table, sfeatures);
zprop_register_index(ZFS_PROP_COMPRESSION, "compression",
ZIO_COMPRESS_DEFAULT, PROP_INHERIT,

113
module/zfs/blake3_zfs.c Normal file
View File

@ -0,0 +1,113 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://opensource.org/licenses/CDDL-1.0.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Copyright 2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include <sys/zfs_context.h>
#include <sys/zio_checksum.h>
#include <sys/blake3.h>
#include <sys/abd.h>
static int
blake3_incremental(void *buf, size_t size, void *arg)
{
BLAKE3_CTX *ctx = arg;
Blake3_Update(ctx, buf, size);
return (0);
}
/*
* Computes a native 256-bit BLAKE3 MAC checksum. Please note that this
* function requires the presence of a ctx_template that should be allocated
* using abd_checksum_blake3_tmpl_init.
*/
void
abd_checksum_blake3_native(abd_t *abd, uint64_t size, const void *ctx_template,
zio_cksum_t *zcp)
{
BLAKE3_CTX *ctx;
ctx = kmem_alloc(sizeof (*ctx), KM_NOSLEEP);
ASSERT(ctx != 0);
ASSERT(ctx_template != 0);
memcpy(ctx, ctx_template, sizeof (*ctx));
(void) abd_iterate_func(abd, 0, size, blake3_incremental, ctx);
Blake3_Final(ctx, (uint8_t *)zcp);
memset(ctx, 0, sizeof (*ctx));
kmem_free(ctx, sizeof (*ctx));
}
/*
* Byteswapped version of abd_checksum_blake3_native. This just invokes
* the native checksum function and byteswaps the resulting checksum (since
* BLAKE3 is internally endian-insensitive).
*/
void
abd_checksum_blake3_byteswap(abd_t *abd, uint64_t size,
const void *ctx_template, zio_cksum_t *zcp)
{
zio_cksum_t tmp;
ASSERT(ctx_template != 0);
abd_checksum_blake3_native(abd, size, ctx_template, &tmp);
zcp->zc_word[0] = BSWAP_64(tmp.zc_word[0]);
zcp->zc_word[1] = BSWAP_64(tmp.zc_word[1]);
zcp->zc_word[2] = BSWAP_64(tmp.zc_word[2]);
zcp->zc_word[3] = BSWAP_64(tmp.zc_word[3]);
}
/*
* Allocates a BLAKE3 MAC template suitable for using in BLAKE3 MAC checksum
* computations and returns a pointer to it.
*/
void *
abd_checksum_blake3_tmpl_init(const zio_cksum_salt_t *salt)
{
BLAKE3_CTX *ctx;
ASSERT(sizeof (salt->zcs_bytes) == 32);
/* init reference object */
ctx = kmem_zalloc(sizeof (*ctx), KM_SLEEP);
Blake3_InitKeyed(ctx, salt->zcs_bytes);
return (ctx);
}
/*
* Frees a BLAKE3 context template previously allocated using
* zio_checksum_blake3_tmpl_init.
*/
void
abd_checksum_blake3_tmpl_free(void *ctx_template)
{
BLAKE3_CTX *ctx = ctx_template;
memset(ctx, 0, sizeof (*ctx));
kmem_free(ctx, sizeof (*ctx));
}

3
module/zfs/spa_misc.c
View File

@ -30,6 +30,7 @@
*/
#include <sys/zfs_context.h>
#include <sys/zfs_chksum.h>
#include <sys/spa_impl.h>
#include <sys/zio.h>
#include <sys/zio_checksum.h>
@ -2417,6 +2418,7 @@ spa_init(spa_mode_t mode)
vdev_raidz_math_init();
vdev_file_init();
zfs_prop_init();
chksum_init();
zpool_prop_init();
zpool_feature_init();
spa_config_load();
@ -2438,6 +2440,7 @@ spa_fini(void)
vdev_cache_stat_fini();
vdev_mirror_stat_fini();
vdev_raidz_math_fini();
chksum_fini();
zil_fini();
dmu_fini();
zio_fini();

316
module/zfs/zfs_chksum.c Normal file
View File

@ -0,0 +1,316 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Copyright (c) 2021 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include <sys/types.h>
#include <sys/spa.h>
#include <sys/zio_checksum.h>
#include <sys/zfs_context.h>
#include <sys/zfs_chksum.h>
#include <sys/blake3.h>
static kstat_t *chksum_kstat = NULL;
typedef struct {
const char *name;
const char *impl;
uint64_t bs1k;
uint64_t bs4k;
uint64_t bs16k;
uint64_t bs64k;
uint64_t bs256k;
uint64_t bs1m;
uint64_t bs4m;
zio_cksum_salt_t salt;
zio_checksum_t *(func);
zio_checksum_tmpl_init_t *(init);
zio_checksum_tmpl_free_t *(free);
} chksum_stat_t;
static int chksum_stat_cnt = 0;
static chksum_stat_t *chksum_stat_data = 0;
/*
* i3-1005G1 test output:
*
* implementation 1k 4k 16k 64k 256k 1m 4m
* fletcher-4 5421 15001 26468 32555 34720 32801 18847
* edonr-generic 1196 1602 1761 1749 1762 1759 1751
* skein-generic 546 591 608 615 619 612 616
* sha256-generic 246 270 274 274 277 275 276
* sha256-avx 262 296 304 307 307 307 306
* sha256-sha-ni 769 1072 1172 1220 1219 1232 1228
* sha256-openssl 240 300 316 314 304 285 276
* sha512-generic 333 374 385 392 391 393 392
* sha512-openssl 353 441 467 476 472 467 426
* sha512-avx 362 444 473 475 479 476 478
* sha512-avx2 394 500 530 538 543 545 542
* blake3-generic 308 313 313 313 312 313 312
* blake3-sse2 402 1289 1423 1446 1432 1458 1413
* blake3-sse41 427 1470 1625 1704 1679 1607 1629
* blake3-avx2 428 1920 3095 3343 3356 3318 3204
* blake3-avx512 473 2687 4905 5836 5844 5643 5374
*/
static int
chksum_stat_kstat_headers(char *buf, size_t size)
{
ssize_t off = 0;
off += snprintf(buf + off, size, "%-23s", "implementation");
off += snprintf(buf + off, size - off, "%8s", "1k");
off += snprintf(buf + off, size - off, "%8s", "4k");
off += snprintf(buf + off, size - off, "%8s", "16k");
off += snprintf(buf + off, size - off, "%8s", "64k");
off += snprintf(buf + off, size - off, "%8s", "256k");
off += snprintf(buf + off, size - off, "%8s", "1m");
(void) snprintf(buf + off, size - off, "%8s\n", "4m");
return (0);
}
static int
chksum_stat_kstat_data(char *buf, size_t size, void *data)
{
chksum_stat_t *cs;
ssize_t off = 0;
char b[24];
cs = (chksum_stat_t *)data;
snprintf(b, 23, "%s-%s", cs->name, cs->impl);
off += snprintf(buf + off, size - off, "%-23s", b);
off += snprintf(buf + off, size - off, "%8llu",
(u_longlong_t)cs->bs1k);
off += snprintf(buf + off, size - off, "%8llu",
(u_longlong_t)cs->bs4k);
off += snprintf(buf + off, size - off, "%8llu",
(u_longlong_t)cs->bs16k);
off += snprintf(buf + off, size - off, "%8llu",
(u_longlong_t)cs->bs64k);
off += snprintf(buf + off, size - off, "%8llu",
(u_longlong_t)cs->bs256k);
off += snprintf(buf + off, size - off, "%8llu",
(u_longlong_t)cs->bs1m);
(void) snprintf(buf + off, size - off, "%8llu\n",
(u_longlong_t)cs->bs4m);
return (0);
}
static void *
chksum_stat_kstat_addr(kstat_t *ksp, loff_t n)
{
if (n < chksum_stat_cnt)
ksp->ks_private = (void *)(chksum_stat_data + n);
else
ksp->ks_private = NULL;
return (ksp->ks_private);
}
static void
chksum_run(chksum_stat_t *cs, abd_t *abd, void *ctx, int round,
uint64_t *result)
{
hrtime_t start;
uint64_t run_bw, run_time_ns, run_count = 0, size = 0;
uint32_t l, loops = 0;
zio_cksum_t zcp;
switch (round) {
case 1: /* 1k */
size = 1<<10; loops = 128; break;
case 2: /* 2k */
size = 1<<12; loops = 64; break;
case 3: /* 4k */
size = 1<<14; loops = 32; break;
case 4: /* 16k */
size = 1<<16; loops = 16; break;
case 5: /* 256k */
size = 1<<18; loops = 8; break;
case 6: /* 1m */
size = 1<<20; loops = 4; break;
case 7: /* 4m */
size = 1<<22; loops = 1; break;
}
kpreempt_disable();
start = gethrtime();
do {
for (l = 0; l < loops; l++, run_count++)
cs->func(abd, size, ctx, &zcp);
run_time_ns = gethrtime() - start;
} while (run_time_ns < MSEC2NSEC(1));
kpreempt_enable();
run_bw = size * run_count * NANOSEC;
run_bw /= run_time_ns; /* B/s */
*result = run_bw/1024/1024; /* MiB/s */
}
static void
chksum_benchit(chksum_stat_t *cs)
{
abd_t *abd;
void *ctx = 0;
void *salt = &cs->salt.zcs_bytes;
/* allocate test memory via default abd interface */
abd = abd_alloc_linear(1<<22, B_FALSE);
memset(salt, 0, sizeof (cs->salt.zcs_bytes));
if (cs->init) {
ctx = cs->init(&cs->salt);
}
chksum_run(cs, abd, ctx, 1, &cs->bs1k);
chksum_run(cs, abd, ctx, 2, &cs->bs4k);
chksum_run(cs, abd, ctx, 3, &cs->bs16k);
chksum_run(cs, abd, ctx, 4, &cs->bs64k);
chksum_run(cs, abd, ctx, 5, &cs->bs256k);
chksum_run(cs, abd, ctx, 6, &cs->bs1m);
chksum_run(cs, abd, ctx, 7, &cs->bs4m);
/* free up temp memory */
if (cs->free) {
cs->free(ctx);
}
abd_free(abd);
}
/*
* Initialize and benchmark all supported implementations.
*/
static void
chksum_benchmark(void)
{
#ifndef _KERNEL
/* we need the benchmark only for the kernel module */
return;
#endif
chksum_stat_t *cs;
int cbid = 0, id;
uint64_t max = 0;
/* space for the benchmark times */
chksum_stat_cnt = 4;
chksum_stat_cnt += blake3_get_impl_count();
chksum_stat_data = (chksum_stat_t *)kmem_zalloc(
sizeof (chksum_stat_t) * chksum_stat_cnt, KM_SLEEP);
/* edonr */
cs = &chksum_stat_data[cbid++];
cs->init = abd_checksum_edonr_tmpl_init;
cs->func = abd_checksum_edonr_native;
cs->free = abd_checksum_edonr_tmpl_free;
cs->name = "edonr";
cs->impl = "generic";
chksum_benchit(cs);
/* skein */
cs = &chksum_stat_data[cbid++];
cs->init = abd_checksum_skein_tmpl_init;
cs->func = abd_checksum_skein_native;
cs->free = abd_checksum_skein_tmpl_free;
cs->name = "skein";
cs->impl = "generic";
chksum_benchit(cs);
/* sha256 */
cs = &chksum_stat_data[cbid++];
cs->init = 0;
cs->func = abd_checksum_SHA256;
cs->free = 0;
cs->name = "sha256";
cs->impl = "generic";
chksum_benchit(cs);
/* sha512 */
cs = &chksum_stat_data[cbid++];
cs->init = 0;
cs->func = abd_checksum_SHA512_native;
cs->free = 0;
cs->name = "sha512";
cs->impl = "generic";
chksum_benchit(cs);
/* blake3 */
for (id = 0; id < blake3_get_impl_count(); id++) {
blake3_set_impl_id(id);
cs = &chksum_stat_data[cbid++];
cs->init = abd_checksum_blake3_tmpl_init;
cs->func = abd_checksum_blake3_native;
cs->free = abd_checksum_blake3_tmpl_free;
cs->name = "blake3";
cs->impl = blake3_get_impl_name();
chksum_benchit(cs);
if (cs->bs256k > max) {
max = cs->bs256k;
blake3_set_impl_fastest(id);
}
}
}
void
chksum_init(void)
{
/* Benchmark supported implementations */
chksum_benchmark();
/* Install kstats for all implementations */
chksum_kstat = kstat_create("zfs", 0, "chksum_bench", "misc",
KSTAT_TYPE_RAW, 0, KSTAT_FLAG_VIRTUAL);
if (chksum_kstat != NULL) {
chksum_kstat->ks_data = NULL;
chksum_kstat->ks_ndata = UINT32_MAX;
kstat_set_raw_ops(chksum_kstat,
chksum_stat_kstat_headers,
chksum_stat_kstat_data,
chksum_stat_kstat_addr);
kstat_install(chksum_kstat);
}
/* setup implementations */
blake3_setup_impl();
}
void
chksum_fini(void)
{
if (chksum_kstat != NULL) {
kstat_delete(chksum_kstat);
chksum_kstat = NULL;
}
if (chksum_stat_cnt) {
kmem_free(chksum_stat_data,
sizeof (chksum_stat_t) * chksum_stat_cnt);
chksum_stat_cnt = 0;
chksum_stat_data = 0;
}
}

6
module/zfs/zio_checksum.c
View File

@ -195,6 +195,10 @@ zio_checksum_info_t zio_checksum_table[ZIO_CHECKSUM_FUNCTIONS] = {
abd_checksum_edonr_tmpl_init, abd_checksum_edonr_tmpl_free,
ZCHECKSUM_FLAG_METADATA | ZCHECKSUM_FLAG_SALTED |
ZCHECKSUM_FLAG_NOPWRITE, "edonr"},
{{abd_checksum_blake3_native, abd_checksum_blake3_byteswap},
abd_checksum_blake3_tmpl_init, abd_checksum_blake3_tmpl_free,
ZCHECKSUM_FLAG_METADATA | ZCHECKSUM_FLAG_DEDUP |
ZCHECKSUM_FLAG_SALTED | ZCHECKSUM_FLAG_NOPWRITE, "blake3"},
};
/*
@ -207,6 +211,8 @@ zio_checksum_to_feature(enum zio_checksum cksum)
VERIFY((cksum & ~ZIO_CHECKSUM_MASK) == 0);
switch (cksum) {
case ZIO_CHECKSUM_BLAKE3:
return (SPA_FEATURE_BLAKE3);
case ZIO_CHECKSUM_SHA512:
return (SPA_FEATURE_SHA512);
case ZIO_CHECKSUM_SKEIN:

4
tests/runfiles/common.run
View File

@ -113,8 +113,8 @@ tests = ['tst.destroy_fs', 'tst.destroy_snap', 'tst.get_count_and_limit',
tags = ['functional', 'channel_program', 'synctask_core']
[tests/functional/checksum]
tests = ['run_edonr_test', 'run_sha2_test', 'run_skein_test', 'filetest_001_pos',
'filetest_002_pos']
tests = ['run_edonr_test', 'run_sha2_test', 'run_skein_test', 'run_blake3_test',
'filetest_001_pos', 'filetest_002_pos']
tags = ['functional', 'checksum']
[tests/functional/clean_mirror]

1
tests/zfs-tests/cmd/.gitignore vendored
View File

@ -42,6 +42,7 @@
/ereports
/zfs_diff-socket
/dosmode_readonly_write
/blake3_test
/edonr_test
/skein_test
/sha2_test

6
tests/zfs-tests/cmd/Makefile.am
View File

@ -98,15 +98,19 @@ scripts_zfs_tests_bin_PROGRAMS += %D%/ereports
libzfs.la
scripts_zfs_tests_bin_PROGRAMS += %D%/edonr_test %D%/skein_test %D%/sha2_test
scripts_zfs_tests_bin_PROGRAMS += %D%/edonr_test %D%/skein_test \
%D%/sha2_test %D%/blake3_test
%C%_skein_test_SOURCES = %D%/checksum/skein_test.c
%C%_sha2_test_SOURCES = %D%/checksum/sha2_test.c
%C%_edonr_test_SOURCES = %D%/checksum/edonr_test.c
%C%_blake3_test_SOURCES = %D%/checksum/blake3_test.c
%C%_skein_test_LDADD = \
libicp.la \
libspl.la \
libspl_assert.la
%C%_sha2_test_LDADD = $(%C%_skein_test_LDADD)
%C%_edonr_test_LDADD = $(%C%_skein_test_LDADD)
%C%_blake3_test_LDADD = $(%C%_skein_test_LDADD)
if BUILD_LINUX

575
tests/zfs-tests/cmd/checksum/blake3_test.c Normal file
View File

@ -0,0 +1,575 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Copyright (c) 2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include <stdarg.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <sys/time.h>
#include <sys/blake3.h>
/*
* set it to a define for debugging
*/
#undef BLAKE3_DEBUG
/*
* C version of:
* https://github.com/BLAKE3-team/BLAKE3/tree/master/test_vectors
*/
typedef struct {
/* input length for this entry */
const int input_len;
/* hash value */
const char *hash;
/* salted hash value */
const char *shash;
} blake3_test_t;
/* BLAKE3 is variable here */
#define TEST_DIGEST_LEN 262
/*
* key for the keyed hashing
*/
static const char *salt = "whats the Elvish word for friend";
static blake3_test_t TestArray[] = {
{
0,
"af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262e0"
"0f03e7b69af26b7faaf09fcd333050338ddfe085b8cc869ca98b206c08243a26f5"
"487789e8f660afe6c99ef9e0c52b92e7393024a80459cf91f476f9ffdbda7001c2"
"2e159b402631f277ca96f2defdf1078282314e763699a31c5363165421cce14d",
"92b2b75604ed3c761f9d6f62392c8a9227ad0ea3f09573e783f1498a4ed60d26b1"
"8171a2f22a4b94822c701f107153dba24918c4bae4d2945c20ece13387627d3b73"
"cbf97b797d5e59948c7ef788f54372df45e45e4293c7dc18c1d41144a9758be589"
"60856be1eabbe22c2653190de560ca3b2ac4aa692a9210694254c371e851bc8f",
},
{
1,
"2d3adedff11b61f14c886e35afa036736dcd87a74d27b5c1510225d0f592e213c3"
"a6cb8bf623e20cdb535f8d1a5ffb86342d9c0b64aca3bce1d31f60adfa137b358a"
"d4d79f97b47c3d5e79f179df87a3b9776ef8325f8329886ba42f07fb138bb502f4"
"081cbcec3195c5871e6c23e2cc97d3c69a613eba131e5f1351f3f1da786545e5",
"6d7878dfff2f485635d39013278ae14f1454b8c0a3a2d34bc1ab38228a80c95b65"
"68c0490609413006fbd428eb3fd14e7756d90f73a4725fad147f7bf70fd61c4e0c"
"f7074885e92b0e3f125978b4154986d4fb202a3f331a3fb6cf349a3a70e49990f9"
"8fe4289761c8602c4e6ab1138d31d3b62218078b2f3ba9a88e1d08d0dd4cea11",
},
{
2,
"7b7015bb92cf0b318037702a6cdd81dee41224f734684c2c122cd6359cb1ee63d8"
"386b22e2ddc05836b7c1bb693d92af006deb5ffbc4c70fb44d0195d0c6f252faac"
"61659ef86523aa16517f87cb5f1340e723756ab65efb2f91964e14391de2a43226"
"3a6faf1d146937b35a33621c12d00be8223a7f1919cec0acd12097ff3ab00ab1",
"5392ddae0e0a69d5f40160462cbd9bd889375082ff224ac9c758802b7a6fd20a9f"
"fbf7efd13e989a6c246f96d3a96b9d279f2c4e63fb0bdff633957acf50ee1a5f65"
"8be144bab0f6f16500dee4aa5967fc2c586d85a04caddec90fffb7633f46a60786"
"024353b9e5cebe277fcd9514217fee2267dcda8f7b31697b7c54fab6a939bf8f",
},
{
3,
"e1be4d7a8ab5560aa4199eea339849ba8e293d55ca0a81006726d184519e647f5b"
"49b82f805a538c68915c1ae8035c900fd1d4b13902920fd05e1450822f36de9454"
"b7e9996de4900c8e723512883f93f4345f8a58bfe64ee38d3ad71ab027765d25cd"
"d0e448328a8e7a683b9a6af8b0af94fa09010d9186890b096a08471e4230a134",
"39e67b76b5a007d4921969779fe666da67b5213b096084ab674742f0d5ec62b9b9"
"142d0fab08e1b161efdbb28d18afc64d8f72160c958e53a950cdecf91c1a1bbab1"
"a9c0f01def762a77e2e8545d4dec241e98a89b6db2e9a5b070fc110caae2622690"
"bd7b76c02ab60750a3ea75426a6bb8803c370ffe465f07fb57def95df772c39f",
},
{
4,
"f30f5ab28fe047904037f77b6da4fea1e27241c5d132638d8bedce9d40494f328f"
"603ba4564453e06cdcee6cbe728a4519bbe6f0d41e8a14b5b225174a566dbfa61b"
"56afb1e452dc08c804f8c3143c9e2cc4a31bb738bf8c1917b55830c6e657972117"
"01dc0b98daa1faeaa6ee9e56ab606ce03a1a881e8f14e87a4acf4646272cfd12",
"7671dde590c95d5ac9616651ff5aa0a27bee5913a348e053b8aa9108917fe07011"
"6c0acff3f0d1fa97ab38d813fd46506089118147d83393019b068a55d646251ecf"
"81105f798d76a10ae413f3d925787d6216a7eb444e510fd56916f1d753a5544ecf"
"0072134a146b2615b42f50c179f56b8fae0788008e3e27c67482349e249cb86a",
},
{
5,
"b40b44dfd97e7a84a996a91af8b85188c66c126940ba7aad2e7ae6b385402aa2eb"
"cfdac6c5d32c31209e1f81a454751280db64942ce395104e1e4eaca62607de1c2c"
"a748251754ea5bbe8c20150e7f47efd57012c63b3c6a6632dc1c7cd15f3e1c9999"
"04037d60fac2eb9397f2adbe458d7f264e64f1e73aa927b30988e2aed2f03620",
"73ac69eecf286894d8102018a6fc729f4b1f4247d3703f69bdc6a5fe3e0c84616a"
"b199d1f2f3e53bffb17f0a2209fe8b4f7d4c7bae59c2bc7d01f1ff94c67588cc6b"
"38fa6024886f2c078bfe09b5d9e6584cd6c521c3bb52f4de7687b37117a2dbbec0"
"d59e92fa9a8cc3240d4432f91757aabcae03e87431dac003e7d73574bfdd8218",
},
{
6,
"06c4e8ffb6872fad96f9aaca5eee1553eb62aed0ad7198cef42e87f6a616c84461"
"1a30c4e4f37fe2fe23c0883cde5cf7059d88b657c7ed2087e3d210925ede716435"
"d6d5d82597a1e52b9553919e804f5656278bd739880692c94bff2824d8e0b48cac"
"1d24682699e4883389dc4f2faa2eb3b4db6e39debd5061ff3609916f3e07529a",
"82d3199d0013035682cc7f2a399d4c212544376a839aa863a0f4c91220ca7a6dc2"
"ffb3aa05f2631f0fa9ac19b6e97eb7e6669e5ec254799350c8b8d189e880780084"
"2a5383c4d907c932f34490aaf00064de8cdb157357bde37c1504d2960034930887"
"603abc5ccb9f5247f79224baff6120a3c622a46d7b1bcaee02c5025460941256",
},
{
7,
"3f8770f387faad08faa9d8414e9f449ac68e6ff0417f673f602a646a891419fe66"
"036ef6e6d1a8f54baa9fed1fc11c77cfb9cff65bae915045027046ebe0c01bf5a9"
"41f3bb0f73791d3fc0b84370f9f30af0cd5b0fc334dd61f70feb60dad785f070fe"
"f1f343ed933b49a5ca0d16a503f599a365a4296739248b28d1a20b0e2cc8975c",
"af0a7ec382aedc0cfd626e49e7628bc7a353a4cb108855541a5651bf64fbb28a7c"
"5035ba0f48a9c73dabb2be0533d02e8fd5d0d5639a18b2803ba6bf527e1d145d5f"
"d6406c437b79bcaad6c7bdf1cf4bd56a893c3eb9510335a7a798548c6753f74617"
"bede88bef924ba4b334f8852476d90b26c5dc4c3668a2519266a562c6c8034a6",
},
{
8,
"2351207d04fc16ade43ccab08600939c7c1fa70a5c0aaca76063d04c3228eaeb72"
"5d6d46ceed8f785ab9f2f9b06acfe398c6699c6129da084cb531177445a682894f"
"9685eaf836999221d17c9a64a3a057000524cd2823986db378b074290a1a9b93a2"
"2e135ed2c14c7e20c6d045cd00b903400374126676ea78874d79f2dd7883cf5c",
"be2f5495c61cba1bb348a34948c004045e3bd4dae8f0fe82bf44d0da245a060048"
"eb5e68ce6dea1eb0229e144f578b3aa7e9f4f85febd135df8525e6fe40c6f0340d"
"13dd09b255ccd5112a94238f2be3c0b5b7ecde06580426a93e0708555a265305ab"
"f86d874e34b4995b788e37a823491f25127a502fe0704baa6bfdf04e76c13276",
},
{
63,
"e9bc37a594daad83be9470df7f7b3798297c3d834ce80ba85d6e207627b7db7b11"
"97012b1e7d9af4d7cb7bdd1f3bb49a90a9b5dec3ea2bbc6eaebce77f4e470cbf46"
"87093b5352f04e4a4570fba233164e6acc36900e35d185886a827f7ea9bdc1e5c3"
"ce88b095a200e62c10c043b3e9bc6cb9b6ac4dfa51794b02ace9f98779040755",
"bb1eb5d4afa793c1ebdd9fb08def6c36d10096986ae0cfe148cd101170ce37aea0"
"5a63d74a840aecd514f654f080e51ac50fd617d22610d91780fe6b07a26b0847ab"
"b38291058c97474ef6ddd190d30fc318185c09ca1589d2024f0a6f16d45f116783"
"77483fa5c005b2a107cb9943e5da634e7046855eaa888663de55d6471371d55d",
},
{
64,
"4eed7141ea4a5cd4b788606bd23f46e212af9cacebacdc7d1f4c6dc7f2511b98fc"
"9cc56cb831ffe33ea8e7e1d1df09b26efd2767670066aa82d023b1dfe8ab1b2b7f"
"bb5b97592d46ffe3e05a6a9b592e2949c74160e4674301bc3f97e04903f8c6cf95"
"b863174c33228924cdef7ae47559b10b294acd660666c4538833582b43f82d74",
"ba8ced36f327700d213f120b1a207a3b8c04330528586f414d09f2f7d9ccb7e682"
"44c26010afc3f762615bbac552a1ca909e67c83e2fd5478cf46b9e811efccc93f7"
"7a21b17a152ebaca1695733fdb086e23cd0eb48c41c034d52523fc21236e5d8c92"
"55306e48d52ba40b4dac24256460d56573d1312319afcf3ed39d72d0bfc69acb",
},
{
65,
"de1e5fa0be70df6d2be8fffd0e99ceaa8eb6e8c93a63f2d8d1c30ecb6b263dee0e"
"16e0a4749d6811dd1d6d1265c29729b1b75a9ac346cf93f0e1d7296dfcfd4313b3"
"a227faaaaf7757cc95b4e87a49be3b8a270a12020233509b1c3632b3485eef309d"
"0abc4a4a696c9decc6e90454b53b000f456a3f10079072baaf7a981653221f2c",
"c0a4edefa2d2accb9277c371ac12fcdbb52988a86edc54f0716e1591b4326e72d5"
"e795f46a596b02d3d4bfb43abad1e5d19211152722ec1f20fef2cd413e3c22f2fc"
"5da3d73041275be6ede3517b3b9f0fc67ade5956a672b8b75d96cb43294b904149"
"7de92637ed3f2439225e683910cb3ae923374449ca788fb0f9bea92731bc26ad",
},
{
127,
"d81293fda863f008c09e92fc382a81f5a0b4a1251cba1634016a0f86a6bd640de3"
"137d477156d1fde56b0cf36f8ef18b44b2d79897bece12227539ac9ae0a5119da4"
"7644d934d26e74dc316145dcb8bb69ac3f2e05c242dd6ee06484fcb0e956dc4435"
"5b452c5e2bbb5e2b66e99f5dd443d0cbcaaafd4beebaed24ae2f8bb672bcef78",
"c64200ae7dfaf35577ac5a9521c47863fb71514a3bcad18819218b818de85818ee"
"7a317aaccc1458f78d6f65f3427ec97d9c0adb0d6dacd4471374b621b7b5f35cd5"
"4663c64dbe0b9e2d95632f84c611313ea5bd90b71ce97b3cf645776f3adc11e27d"
"135cbadb9875c2bf8d3ae6b02f8a0206aba0c35bfe42574011931c9a255ce6dc",
},
{
128,
"f17e570564b26578c33bb7f44643f539624b05df1a76c81f30acd548c44b45efa6"
"9faba091427f9c5c4caa873aa07828651f19c55bad85c47d1368b11c6fd99e47ec"
"ba5820a0325984d74fe3e4058494ca12e3f1d3293d0010a9722f7dee64f71246f7"
"5e9361f44cc8e214a100650db1313ff76a9f93ec6e84edb7add1cb4a95019b0c",
"b04fe15577457267ff3b6f3c947d93be581e7e3a4b018679125eaf86f6a628ecd8"
"6bbe0001f10bda47e6077b735016fca8119da11348d93ca302bbd125bde0db2b50"
"edbe728a620bb9d3e6f706286aedea973425c0b9eedf8a38873544cf91badf49ad"
"92a635a93f71ddfcee1eae536c25d1b270956be16588ef1cfef2f1d15f650bd5",
},
{
129,
"683aaae9f3c5ba37eaaf072aed0f9e30bac0865137bae68b1fde4ca2aebdcb12f9"
"6ffa7b36dd78ba321be7e842d364a62a42e3746681c8bace18a4a8a79649285c71"
"27bf8febf125be9de39586d251f0d41da20980b70d35e3dac0eee59e468a894fa7"
"e6a07129aaad09855f6ad4801512a116ba2b7841e6cfc99ad77594a8f2d181a7",
"d4a64dae6cdccbac1e5287f54f17c5f985105457c1a2ec1878ebd4b57e20d38f1c"
"9db018541eec241b748f87725665b7b1ace3e0065b29c3bcb232c90e37897fa5aa"
"ee7e1e8a2ecfcd9b51463e42238cfdd7fee1aecb3267fa7f2128079176132a412c"
"d8aaf0791276f6b98ff67359bd8652ef3a203976d5ff1cd41885573487bcd683",
},
{
1023,
"10108970eeda3eb932baac1428c7a2163b0e924c9a9e25b35bba72b28f70bd11a1"
"82d27a591b05592b15607500e1e8dd56bc6c7fc063715b7a1d737df5bad3339c56"
"778957d870eb9717b57ea3d9fb68d1b55127bba6a906a4a24bbd5acb2d123a37b2"
"8f9e9a81bbaae360d58f85e5fc9d75f7c370a0cc09b6522d9c8d822f2f28f485",
"c951ecdf03288d0fcc96ee3413563d8a6d3589547f2c2fb36d9786470f1b9d6e89"
"0316d2e6d8b8c25b0a5b2180f94fb1a158ef508c3cde45e2966bd796a696d3e13e"
"fd86259d756387d9becf5c8bf1ce2192b87025152907b6d8cc33d17826d8b7b9bc"
"97e38c3c85108ef09f013e01c229c20a83d9e8efac5b37470da28575fd755a10",
},
{
1024,
"42214739f095a406f3fc83deb889744ac00df831c10daa55189b5d121c855af71c"
"f8107265ecdaf8505b95d8fcec83a98a6a96ea5109d2c179c47a387ffbb404756f"
"6eeae7883b446b70ebb144527c2075ab8ab204c0086bb22b7c93d465efc57f8d91"
"7f0b385c6df265e77003b85102967486ed57db5c5ca170ba441427ed9afa684e",
"75c46f6f3d9eb4f55ecaaee480db732e6c2105546f1e675003687c31719c7ba4a7"
"8bc838c72852d4f49c864acb7adafe2478e824afe51c8919d06168414c265f298a"
"8094b1ad813a9b8614acabac321f24ce61c5a5346eb519520d38ecc43e89b50002"
"36df0597243e4d2493fd626730e2ba17ac4d8824d09d1a4a8f57b8227778e2de",
},
{
1025,
"d00278ae47eb27b34faecf67b4fe263f82d5412916c1ffd97c8cb7fb814b8444f4"
"c4a22b4b399155358a994e52bf255de60035742ec71bd08ac275a1b51cc6bfe332"
"b0ef84b409108cda080e6269ed4b3e2c3f7d722aa4cdc98d16deb554e5627be8f9"
"55c98e1d5f9565a9194cad0c4285f93700062d9595adb992ae68ff12800ab67a",
"357dc55de0c7e382c900fd6e320acc04146be01db6a8ce7210b7189bd664ea6936"
"2396b77fdc0d2634a552970843722066c3c15902ae5097e00ff53f1e116f1cd535"
"2720113a837ab2452cafbde4d54085d9cf5d21ca613071551b25d52e69d6c81123"
"872b6f19cd3bc1333edf0c52b94de23ba772cf82636cff4542540a7738d5b930",
},
{
2048,
"e776b6028c7cd22a4d0ba182a8bf62205d2ef576467e838ed6f2529b85fba24a9a"
"60bf80001410ec9eea6698cd537939fad4749edd484cb541aced55cd9bf54764d0"
"63f23f6f1e32e12958ba5cfeb1bf618ad094266d4fc3c968c2088f677454c288c6"
"7ba0dba337b9d91c7e1ba586dc9a5bc2d5e90c14f53a8863ac75655461cea8f9",
"879cf1fa2ea0e79126cb1063617a05b6ad9d0b696d0d757cf053439f60a99dd101"
"73b961cd574288194b23ece278c330fbb8585485e74967f31352a8183aa782b2b2"
"2f26cdcadb61eed1a5bc144b8198fbb0c13abbf8e3192c145d0a5c21633b0ef860"
"54f42809df823389ee40811a5910dcbd1018af31c3b43aa55201ed4edaac74fe",
},
{
2049,
"5f4d72f40d7a5f82b15ca2b2e44b1de3c2ef86c426c95c1af0b687952256303096"
"de31d71d74103403822a2e0bc1eb193e7aecc9643a76b7bbc0c9f9c52e8783aae9"
"8764ca468962b5c2ec92f0c74eb5448d519713e09413719431c802f948dd5d9042"
"5a4ecdadece9eb178d80f26efccae630734dff63340285adec2aed3b51073ad3",
"9f29700902f7c86e514ddc4df1e3049f258b2472b6dd5267f61bf13983b78dd5f9"
"a88abfefdfa1e00b418971f2b39c64ca621e8eb37fceac57fd0c8fc8e117d43b81"
"447be22d5d8186f8f5919ba6bcc6846bd7d50726c06d245672c2ad4f61702c6464"
"99ee1173daa061ffe15bf45a631e2946d616a4c345822f1151284712f76b2b0e",
},
{
3072,
"b98cb0ff3623be03326b373de6b9095218513e64f1ee2edd2525c7ad1e5cffd29a"
"3f6b0b978d6608335c09dc94ccf682f9951cdfc501bfe47b9c9189a6fc7b404d12"
"0258506341a6d802857322fbd20d3e5dae05b95c88793fa83db1cb08e7d8008d15"
"99b6209d78336e24839724c191b2a52a80448306e0daa84a3fdb566661a37e11",
"044a0e7b172a312dc02a4c9a818c036ffa2776368d7f528268d2e6b5df19177022"
"f302d0529e4174cc507c463671217975e81dab02b8fdeb0d7ccc7568dd22574c78"
"3a76be215441b32e91b9a904be8ea81f7a0afd14bad8ee7c8efc305ace5d3dd61b"
"996febe8da4f56ca0919359a7533216e2999fc87ff7d8f176fbecb3d6f34278b",
},
{
3073,
"7124b49501012f81cc7f11ca069ec9226cecb8a2c850cfe644e327d22d3e1cd39a"
"27ae3b79d68d89da9bf25bc27139ae65a324918a5f9b7828181e52cf373c84f35b"
"639b7fccbb985b6f2fa56aea0c18f531203497b8bbd3a07ceb5926f1cab74d14bd"
"66486d9a91eba99059a98bd1cd25876b2af5a76c3e9eed554ed72ea952b603bf",
"68dede9bef00ba89e43f31a6825f4cf433389fedae75c04ee9f0cf16a427c95a96"
"d6da3fe985054d3478865be9a092250839a697bbda74e279e8a9e69f0025e4cfdd"
"d6cfb434b1cd9543aaf97c635d1b451a4386041e4bb100f5e45407cbbc24fa53ea"
"2de3536ccb329e4eb9466ec37093a42cf62b82903c696a93a50b702c80f3c3c5",
},
{
4096,
"015094013f57a5277b59d8475c0501042c0b642e531b0a1c8f58d2163229e96902"
"89e9409ddb1b99768eafe1623da896faf7e1114bebeadc1be30829b6f8af707d85"
"c298f4f0ff4d9438aef948335612ae921e76d411c3a9111df62d27eaf871959ae0"
"062b5492a0feb98ef3ed4af277f5395172dbe5c311918ea0074ce0036454f620",
"befc660aea2f1718884cd8deb9902811d332f4fc4a38cf7c7300d597a081bfc0bb"
"b64a36edb564e01e4b4aaf3b060092a6b838bea44afebd2deb8298fa562b7b597c"
"757b9df4c911c3ca462e2ac89e9a787357aaf74c3b56d5c07bc93ce899568a3eb1"
"7d9250c20f6c5f6c1e792ec9a2dcb715398d5a6ec6d5c54f586a00403a1af1de",
},
{
4097,
"9b4052b38f1c5fc8b1f9ff7ac7b27cd242487b3d890d15c96a1c25b8aa0fb99505"
"f91b0b5600a11251652eacfa9497b31cd3c409ce2e45cfe6c0a016967316c426bd"
"26f619eab5d70af9a418b845c608840390f361630bd497b1ab44019316357c61db"
"e091ce72fc16dc340ac3d6e009e050b3adac4b5b2c92e722cffdc46501531956",
"00df940cd36bb9fa7cbbc3556744e0dbc8191401afe70520ba292ee3ca80abbc60"
"6db4976cfdd266ae0abf667d9481831ff12e0caa268e7d3e57260c0824115a54ce"
"595ccc897786d9dcbf495599cfd90157186a46ec800a6763f1c59e36197e9939e9"
"00809f7077c102f888caaf864b253bc41eea812656d46742e4ea42769f89b83f",
},
{
5120,
"9cadc15fed8b5d854562b26a9536d9707cadeda9b143978f319ab34230535833ac"
"c61c8fdc114a2010ce8038c853e121e1544985133fccdd0a2d507e8e615e611e9a"
"0ba4f47915f49e53d721816a9198e8b30f12d20ec3689989175f1bf7a300eee0d9"
"321fad8da232ece6efb8e9fd81b42ad161f6b9550a069e66b11b40487a5f5059",
"2c493e48e9b9bf31e0553a22b23503c0a3388f035cece68eb438d22fa1943e209b"
"4dc9209cd80ce7c1f7c9a744658e7e288465717ae6e56d5463d4f80cdb2ef56495"
"f6a4f5487f69749af0c34c2cdfa857f3056bf8d807336a14d7b89bf62bef2fb54f"
"9af6a546f818dc1e98b9e07f8a5834da50fa28fb5874af91bf06020d1bf0120e",
},
{
5121,
"628bd2cb2004694adaab7bbd778a25df25c47b9d4155a55f8fbd79f2fe154cff96"
"adaab0613a6146cdaabe498c3a94e529d3fc1da2bd08edf54ed64d40dcd6777647"
"eac51d8277d70219a9694334a68bc8f0f23e20b0ff70ada6f844542dfa32cd4204"
"ca1846ef76d811cdb296f65e260227f477aa7aa008bac878f72257484f2b6c95",
"6ccf1c34753e7a044db80798ecd0782a8f76f33563accaddbfbb2e0ea4b2d0240d"
"07e63f13667a8d1490e5e04f13eb617aea16a8c8a5aaed1ef6fbde1b0515e3c810"
"50b361af6ead126032998290b563e3caddeaebfab592e155f2e161fb7cba939092"
"133f23f9e65245e58ec23457b78a2e8a125588aad6e07d7f11a85b88d375b72d",
},
{
6144,
"3e2e5b74e048f3add6d21faab3f83aa44d3b2278afb83b80b3c35164ebeca2054d"
"742022da6fdda444ebc384b04a54c3ac5839b49da7d39f6d8a9db03deab32aade1"
"56c1c0311e9b3435cde0ddba0dce7b26a376cad121294b689193508dd63151603c"
"6ddb866ad16c2ee41585d1633a2cea093bea714f4c5d6b903522045b20395c83",
"3d6b6d21281d0ade5b2b016ae4034c5dec10ca7e475f90f76eac7138e9bc8f1dc3"
"5754060091dc5caf3efabe0603c60f45e415bb3407db67e6beb3d11cf8e4f79075"
"61f05dace0c15807f4b5f389c841eb114d81a82c02a00b57206b1d11fa6e803486"
"b048a5ce87105a686dee041207e095323dfe172df73deb8c9532066d88f9da7e",
},
{
6145,
"f1323a8631446cc50536a9f705ee5cb619424d46887f3c376c695b70e0f0507f18"
"a2cfdd73c6e39dd75ce7c1c6e3ef238fd54465f053b25d21044ccb2093beb01501"
"5532b108313b5829c3621ce324b8e14229091b7c93f32db2e4e63126a377d2a63a"
"3597997d4f1cba59309cb4af240ba70cebff9a23d5e3ff0cdae2cfd54e070022",
"9ac301e9e39e45e3250a7e3b3df701aa0fb6889fbd80eeecf28dbc6300fbc539f3"
"c184ca2f59780e27a576c1d1fb9772e99fd17881d02ac7dfd39675aca918453283"
"ed8c3169085ef4a466b91c1649cc341dfdee60e32231fc34c9c4e0b9a2ba87ca8f"
"372589c744c15fd6f985eec15e98136f25beeb4b13c4e43dc84abcc79cd4646c",
},
{
7168,
"61da957ec2499a95d6b8023e2b0e604ec7f6b50e80a9678b89d2628e99ada77a57"
"07c321c83361793b9af62a40f43b523df1c8633cecb4cd14d00bdc79c78fca5165"
"b863893f6d38b02ff7236c5a9a8ad2dba87d24c547cab046c29fc5bc1ed142e1de"
"4763613bb162a5a538e6ef05ed05199d751f9eb58d332791b8d73fb74e4fce95",
"b42835e40e9d4a7f42ad8cc04f85a963a76e18198377ed84adddeaecacc6f3fca2"
"f01d5277d69bb681c70fa8d36094f73ec06e452c80d2ff2257ed82e7ba34840098"
"9a65ee8daa7094ae0933e3d2210ac6395c4af24f91c2b590ef87d7788d7066ea3e"
"aebca4c08a4f14b9a27644f99084c3543711b64a070b94f2c9d1d8a90d035d52",
},
{
7169,
"a003fc7a51754a9b3c7fae0367ab3d782dccf28855a03d435f8cfe74605e781798"
"a8b20534be1ca9eb2ae2df3fae2ea60e48c6fb0b850b1385b5de0fe460dbe9d9f9"
"b0d8db4435da75c601156df9d047f4ede008732eb17adc05d96180f8a735485228"
"40779e6062d643b79478a6e8dbce68927f36ebf676ffa7d72d5f68f050b119c8",
"ed9b1a922c046fdb3d423ae34e143b05ca1bf28b710432857bf738bcedbfa5113c"
"9e28d72fcbfc020814ce3f5d4fc867f01c8f5b6caf305b3ea8a8ba2da3ab69fabc"
"b438f19ff11f5378ad4484d75c478de425fb8e6ee809b54eec9bdb184315dc8566"
"17c09f5340451bf42fd3270a7b0b6566169f242e533777604c118a6358250f54",
},
{
8192,
"aae792484c8efe4f19e2ca7d371d8c467ffb10748d8a5a1ae579948f718a2a635f"
"e51a27db045a567c1ad51be5aa34c01c6651c4d9b5b5ac5d0fd58cf18dd61a4777"
"8566b797a8c67df7b1d60b97b19288d2d877bb2df417ace009dcb0241ca1257d62"
"712b6a4043b4ff33f690d849da91ea3bf711ed583cb7b7a7da2839ba71309bbf",
"dc9637c8845a770b4cbf76b8daec0eebf7dc2eac11498517f08d44c8fc00d58a48"
"34464159dcbc12a0ba0c6d6eb41bac0ed6585cabfe0aca36a375e6c5480c22afdc"
"40785c170f5a6b8a1107dbee282318d00d915ac9ed1143ad40765ec120042ee121"
"cd2baa36250c618adaf9e27260fda2f94dea8fb6f08c04f8f10c78292aa46102",
},
{
8193,
"bab6c09cb8ce8cf459261398d2e7aef35700bf488116ceb94a36d0f5f1b7bc3bb2"
"282aa69be089359ea1154b9a9286c4a56af4de975a9aa4a5c497654914d279bea6"
"0bb6d2cf7225a2fa0ff5ef56bbe4b149f3ed15860f78b4e2ad04e158e375c1e0c0"
"b551cd7dfc82f1b155c11b6b3ed51ec9edb30d133653bb5709d1dbd55f4e1ff6",
"954a2a75420c8d6547e3ba5b98d963e6fa6491addc8c023189cc519821b4a1f5f0"
"3228648fd983aef045c2fa8290934b0866b615f585149587dda229903996532883"
"5a2b18f1d63b7e300fc76ff260b571839fe44876a4eae66cbac8c67694411ed7e0"
"9df51068a22c6e67d6d3dd2cca8ff12e3275384006c80f4db68023f24eebba57",
},
{
16384,
"f875d6646de28985646f34ee13be9a576fd515f76b5b0a26bb324735041ddde49d"
"764c270176e53e97bdffa58d549073f2c660be0e81293767ed4e4929f9ad34bbb3"
"9a529334c57c4a381ffd2a6d4bfdbf1482651b172aa883cc13408fa67758a3e475"
"03f93f87720a3177325f7823251b85275f64636a8f1d599c2e49722f42e93893",
"9e9fc4eb7cf081ea7c47d1807790ed211bfec56aa25bb7037784c13c4b707b0df9"
"e601b101e4cf63a404dfe50f2e1865bb12edc8fca166579ce0c70dba5a5c0fc960"
"ad6f3772183416a00bd29d4c6e651ea7620bb100c9449858bf14e1ddc9ecd35725"
"581ca5b9160de04060045993d972571c3e8f71e9d0496bfa744656861b169d65",
},
{
31744,
"62b6960e1a44bcc1eb1a611a8d6235b6b4b78f32e7abc4fb4c6cdcce94895c4786"
"0cc51f2b0c28a7b77304bd55fe73af663c02d3f52ea053ba43431ca5bab7bfea2f"
"5e9d7121770d88f70ae9649ea713087d1914f7f312147e247f87eb2d4ffef0ac97"
"8bf7b6579d57d533355aa20b8b77b13fd09748728a5cc327a8ec470f4013226f",
"efa53b389ab67c593dba624d898d0f7353ab99e4ac9d42302ee64cbf9939a4193a"
"7258db2d9cd32a7a3ecfce46144114b15c2fcb68a618a976bd74515d47be08b628"
"be420b5e830fade7c080e351a076fbc38641ad80c736c8a18fe3c66ce12f95c61c"
"2462a9770d60d0f77115bbcd3782b593016a4e728d4c06cee4505cb0c08a42ec",
},
{
102400,
"bc3e3d41a1146b069abffad3c0d44860cf664390afce4d9661f7902e7943e085e0"
"1c59dab908c04c3342b816941a26d69c2605ebee5ec5291cc55e15b76146e6745f"
"0601156c3596cb75065a9c57f35585a52e1ac70f69131c23d611ce11ee4ab1ec2c"
"009012d236648e77be9295dd0426f29b764d65de58eb7d01dd42248204f45f8e",
"1c35d1a5811083fd7119f5d5d1ba027b4d01c0c6c49fb6ff2cf75393ea5db4a7f9"
"dbdd3e1d81dcbca3ba241bb18760f207710b751846faaeb9dff8262710999a59b2"
"aa1aca298a032d94eacfadf1aa192418eb54808db23b56e34213266aa08499a16b"
"354f018fc4967d05f8b9d2ad87a7278337be9693fc638a3bfdbe314574ee6fc4",
},
{
0, 0, 0
}
};
#ifdef BLAKE3_DEBUG
#define dprintf printf
#else
#define dprintf(...)
#endif
static char fmt_tohex(char c);
static size_t fmt_hexdump(char *dest, const char *src, size_t len);
static char fmt_tohex(char c) {
return ((char)(c >= 10 ? c-10+'a' : c+'0'));
}
static size_t fmt_hexdump(char *dest, const char *src, size_t len) {
register const unsigned char *s = (const unsigned char *) src;
size_t written = 0, i;
if (!dest)
return ((len > ((size_t)-1)/2) ? (size_t)-1 : len*2);
for (i = 0; i < len; ++i) {
dest[written] = fmt_tohex(s[i]>>4);
dest[written+1] = fmt_tohex(s[i]&15);
written += 2;
}
return (written);
}
int
main(int argc, char *argv[])
{
boolean_t failed = B_FALSE;
uint8_t buffer[102400];
uint64_t cpu_mhz = 0;
int id, i, j;
if (argc == 2)
cpu_mhz = atoi(argv[1]);
/* fill test message */
for (i = 0, j = 0; i < sizeof (buffer); i++, j++) {
if (j == 251)
j = 0;
buffer[i] = (uint8_t)j;
}
(void) printf("Running algorithm correctness tests:\n");
for (id = 0; id < blake3_get_impl_count(); id++) {
blake3_set_impl_id(id);
const char *name = blake3_get_impl_name();
dprintf("Result for BLAKE3-%s:\n", name);
for (i = 0; TestArray[i].hash; i++) {
blake3_test_t *cur = &TestArray[i];
BLAKE3_CTX ctx;
uint8_t digest[TEST_DIGEST_LEN];
char result[TEST_DIGEST_LEN];
/* default hashing */
Blake3_Init(&ctx);
Blake3_Update(&ctx, buffer, cur->input_len);
Blake3_FinalSeek(&ctx, 0, digest, TEST_DIGEST_LEN);
fmt_hexdump(result, (char *)digest, 131);
if (memcmp(result, cur->hash, 131) != 0)
failed = B_TRUE;
dprintf("HASH-res: %s\n", result);
dprintf("HASH-ref: %s\n", cur->hash);
/* salted hashing */
Blake3_InitKeyed(&ctx, (const uint8_t *)salt);
Blake3_Update(&ctx, buffer, cur->input_len);
Blake3_FinalSeek(&ctx, 0, digest, TEST_DIGEST_LEN);
fmt_hexdump(result, (char *)digest, 131);
if (memcmp(result, cur->shash, 131) != 0)
failed = B_TRUE;
dprintf("SHASH-res: %s\n", result);
dprintf("SHASH-ref: %s\n", cur->shash);
printf("BLAKE3-%s Message (inlen=%d)\tResult: %s\n",
name, cur->input_len, failed?"FAILED!":"OK");
}
}
if (failed)
return (1);
#define BLAKE3_PERF_TEST(impl, diglen) \
do { \
BLAKE3_CTX ctx; \
uint8_t digest[diglen / 8]; \
uint8_t block[131072]; \
uint64_t delta; \
double cpb = 0; \
int i; \
struct timeval start, end; \
memset(block, 0, sizeof (block)); \
(void) gettimeofday(&start, NULL); \
Blake3_Init(&ctx); \
for (i = 0; i < 8192; i++) \
Blake3_Update(&ctx, block, sizeof (block)); \
Blake3_Final(&ctx, digest); \
(void) gettimeofday(&end, NULL); \
delta = (end.tv_sec * 1000000llu + end.tv_usec) - \
(start.tv_sec * 1000000llu + start.tv_usec); \
if (cpu_mhz != 0) { \
cpb = (cpu_mhz * 1e6 * ((double)delta / \
1000000)) / (8192 * 128 * 1024); \
} \
(void) printf("BLAKE3-%s %llu us (%.02f CPB)\n", impl, \
(u_longlong_t)delta, cpb); \
} while (0)
printf("Running performance tests (hashing 1024 MiB of data):\n");
for (id = 0; id < blake3_get_impl_count(); id++) {
blake3_set_impl_id(id);
const char *name = blake3_get_impl_name();
BLAKE3_PERF_TEST(name, 256);
}
return (0);
}

3
tests/zfs-tests/cmd/checksum/edonr_test.c
View File

@ -28,9 +28,6 @@
* gettimeofday due to -D_KERNEL (we can do this since we're actually
* running in userspace, but we need -D_KERNEL for the remaining Edon-R code).
*/
#ifdef _KERNEL
#undef _KERNEL
#endif
#include <sys/edonr.h>
#include <stdlib.h>

3
tests/zfs-tests/cmd/checksum/sha2_test.c
View File

@ -28,9 +28,6 @@
* gettimeofday due to -D_KERNEL (we can do this since we're actually
* running in userspace, but we need -D_KERNEL for the remaining SHA2 code).
*/
#ifdef _KERNEL
#undef _KERNEL
#endif
#include <stdarg.h>
#include <stdlib.h>

3
tests/zfs-tests/cmd/checksum/skein_test.c
View File

@ -28,9 +28,6 @@
* gettimeofday due to -D_KERNEL (we can do this since we're actually
* running in userspace, but we need -D_KERNEL for the remaining Skein code).
*/
#ifdef _KERNEL
#undef _KERNEL
#endif
#include <sys/skein.h>
#include <stdlib.h>

1
tests/zfs-tests/include/commands.cfg
View File

@ -212,6 +212,7 @@ export ZFSTEST_FILES='badsend
zed_fd_spill-zedlet
suid_write_to_file
cp_files
blake3_test
edonr_test
skein_test
sha2_test

2
tests/zfs-tests/include/properties.shlib
View File

@ -17,7 +17,7 @@
typeset -a compress_prop_vals=('off' 'lzjb' 'lz4' 'gzip' 'zle' 'zstd')
typeset -a checksum_prop_vals=('on' 'off' 'fletcher2' 'fletcher4' 'sha256'
'noparity' 'sha512' 'skein')
'noparity' 'sha512' 'skein' 'blake3')
if ! is_freebsd; then
checksum_prop_vals+=('edonr')
fi

1
tests/zfs-tests/tests/Makefile.am
View File

@ -545,6 +545,7 @@ nobase_dist_datadir_zfs_tests_tests_SCRIPTS += \
functional/checksum/cleanup.ksh \
functional/checksum/filetest_001_pos.ksh \
functional/checksum/filetest_002_pos.ksh \
functional/checksum/run_blake3_test.ksh \
functional/checksum/run_edonr_test.ksh \
functional/checksum/run_sha2_test.ksh \
functional/checksum/run_skein_test.ksh \

2
tests/zfs-tests/tests/functional/checksum/default.cfg
View File

@ -30,4 +30,4 @@
. $STF_SUITE/include/libtest.shlib
set -A CHECKSUM_TYPES "fletcher2" "fletcher4" "sha256" "sha512" "skein" "edonr"
set -A CHECKSUM_TYPES "fletcher2" "fletcher4" "blake3" "sha256" "sha512" "skein" "edonr"

30
tests/zfs-tests/tests/functional/checksum/run_blake3_test.ksh Executable file
View File

@ -0,0 +1,30 @@
#!/bin/ksh -p
#
# This file and its contents are supplied under the terms of the
# Common Development and Distribution License ("CDDL"), version 1.0.
# You may only use this file in accordance with the terms of version
# 1.0 of the CDDL.
#
# A full copy of the text of the CDDL should have accompanied this
# source. A copy of the CDDL is also available via the Internet at
# http://www.illumos.org/license/CDDL.
#
#
# Copyright (c) 2015, 2016 by Delphix. All rights reserved.
#
. $STF_SUITE/include/libtest.shlib
#
# Description:
# Run the tests for the BLAKE3 hash algorithm.
#
log_assert "Run the tests for the BLAKE3 hash algorithm."
freq=$(get_cpu_freq)
log_must blake3_test $freq
log_pass "BLAKE3 tests passed."

2
tests/zfs-tests/tests/functional/cli_root/zfs_set/checksum_001_pos.ksh
View File

@ -46,7 +46,7 @@
verify_runnable "both"
set -A dataset "$TESTPOOL" "$TESTPOOL/$TESTFS" "$TESTPOOL/$TESTVOL"
set -A values "on" "off" "fletcher2" "fletcher4" "sha256" "sha512" "skein" "edonr" "noparity"
set -A values "on" "off" "fletcher2" "fletcher4" "sha256" "sha512" "skein" "edonr" "blake3" "noparity"
log_assert "Setting a valid checksum on a file system, volume," \
"it should be successful."

1
tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg
View File

@ -99,5 +99,6 @@ if is_linux || is_freebsd; then
"feature@zstd_compress"
"feature@zilsaxattr"
"feature@head_errlog"
"feature@blake3"
)
fi