History

Warner Losh 6ba2e72b78 aarch64: Use proper guards for NEON instructions The zstd code assumes that if you are on aarch64, you have NEON instructions. This is not necessarily true. In a boot loader, where you might not have the VFP properly initialized, these instructions may not be available. It's also an error to include arm_neon.h when the NEON insturctions aren't enabled. Change the guards for using the NEON instructions from __aarch64__ to __ARM_NEON which is the standard symbol for knowing if they are available. __ARM_NEON is the proper symbol, defined in ARM C Language Extensions Release 2.1 (https://developer.arm.com/documentation/ihi0053/d/). Some sources suggest __ARM_NEON__, but that's the obsolete spelling from prior versions of the standard. Updated based on zstd pull request https://github.com/facebook/zstd/pull/2356 Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Warner Losh <imp@bsdimp.com> Closes #11055		2020-10-13 21:01:40 -07:00
..
include	Avoid symbol collision with in-kernel zstdlib	2020-08-24 12:20:41 -07:00
lib	aarch64: Use proper guards for NEON instructions	2020-10-13 21:01:40 -07:00
Makefile.in	module/zstd: pass -U__BMI__	2020-08-27 15:50:13 -07:00
README.md	Avoid symbol collision with in-kernel zstdlib	2020-08-24 12:20:41 -07:00
zfs_zstd.c	do a cyclic seek for unused memory objects in pool	2020-09-30 13:22:34 -07:00
zstd-in.c	Import ZStandard v1.4.5	2020-08-20 10:30:06 -07:00

README.md

ZSTD-On-ZFS Library Manual

Introduction

This subtree contains the ZSTD library used in ZFS. It is heavily cut-down by dropping any unneeded files, and combined into a single file, but otherwise is intentionally unmodified. Please do not alter the file containing the zstd library, besides upgrading to a newer ZSTD release.

Tree structure:

zfs_zstd.c is the actual zzstd kernel module.
lib/ contains the the unmodified, "amalgamated" version of the Zstandard library, generated from our template file
zstd-in.c is our template file for generating the library
include/: This directory contains supplemental includes for platform compatibility, which are not expected to be used by ZFS elsewhere in the future. Thus we keep them private to ZSTD.

Updating ZSTD

To update ZSTD the following steps need to be taken:

Grab the latest release of ZSTD.
Update module/zstd/zstd-in.c if required. (see zstd/contrib/single_file_libs/zstd-in.c in the zstd repository)
Generate the "single-file-library" and put it to module/zstd/lib/.
Copy the following files to module/zstd/lib/:
- zstd/lib/zstd.h
- zstd/lib/common/zstd_errors.h

This can be done using a few shell commands from inside the zfs repo:

cd PATH/TO/ZFS

url="https://github.com/facebook/zstd"
release="$(curl -s "${url}"/releases/latest | grep -oP '(?<=v)[\d\.]+')"
zstd="/tmp/zstd-${release}/"

wget -O /tmp/zstd.tar.gz \
    "${url}/releases/download/v${release}/zstd-${release}.tar.gz"
tar -C /tmp -xzf /tmp/zstd.tar.gz

cp ${zstd}/lib/zstd.h module/zstd/lib/
cp ${zstd}/lib/zstd_errors.h module/zstd/lib/
${zstd}/contrib/single_file_libs/combine.sh \
    -r ${zstd}/lib -o module/zstd/lib/zstd.c module/zstd/zstd-in.c

Note: if the zstd library for zfs is updated to a newer version, the macro list in include/zstd_compat_wrapper.h usually needs to be updated. this can be done with some hand crafting of the output of the following script: nm zstd.o | awk '{print "#define "$3 " zfs_" $3}' > macrotable

Altering ZSTD and breaking changes

If ZSTD made changes that break compatibility or you need to make breaking changes to the way we handle ZSTD, it is required to maintain backwards compatibility.

We already save the ZSTD version number within the block header to be used to add future compatibility checks and/or fixes. However, currently it is not actually used in such a way.