63652e1546
`configure` now accepts `--enable-asan` and `--enable-ubsan` switches which results in passing `-fsanitize=address` and `-fsanitize=undefined`, respectively, to the compiler. Those flags are enabled in GitHub workflows for ZTS and zloop. Errors reported by both instrumentations are corrected, except for: - Memory leak reporting is (temporarily) suppressed. The cost of fixing them is relatively high compared to the gains. - Checksum computing functions in `module/zcommon/zfs_fletcher*` have UBSan errors suppressed. It is completely impractical to enforce 64-byte payload alignment there due to performance impact. - There's no ASan heap poisoning in `module/zstd/lib/zstd.c`. A custom memory allocator is used there rendering that measure unfeasible. - Memory leaks detection has to be suppressed for `cmd/zvol_id`. `zvol_id` is run by udev with the help of `ptrace(2)`. Tracing is incompatible with memory leaks detection. Reviewed-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz> Reviewed-by: George Melikov <mail@gmelikov.ru> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: szubersk <szuberskidamian@gmail.com> Closes #12928 |
||
|---|---|---|
| .. | ||
| include | ||
| lib | ||
| Makefile.in | ||
| README.md | ||
| zfs_zstd.c | ||
| zstd_sparc.c | ||
| zstd-in.c | ||
ZSTD-On-ZFS Library Manual
Introduction
This subtree contains the ZSTD library used in ZFS. It is heavily cut-down by dropping any unneeded files, and combined into a single file, but otherwise is intentionally unmodified. Please do not alter the file containing the zstd library, besides upgrading to a newer ZSTD release.
Tree structure:
zfs_zstd.cis the actualzzstdkernel module.lib/contains the unmodified, "amalgamated" version of theZstandardlibrary, generated from our template filezstd-in.cis our template file for generating the libraryinclude/: This directory contains supplemental includes for platform compatibility, which are not expected to be used by ZFS elsewhere in the future. Thus we keep them private to ZSTD.
Updating ZSTD
To update ZSTD the following steps need to be taken:
- Grab the latest release of ZSTD.
- Update
module/zstd/zstd-in.cif required. (seezstd/contrib/single_file_libs/zstd-in.cin the zstd repository) - Generate the "single-file-library" and put it to
module/zstd/lib/. - Copy the following files to
module/zstd/lib/:zstd/lib/zstd.hzstd/lib/common/zstd_errors.h
This can be done using a few shell commands from inside the zfs repo:
cd PATH/TO/ZFS
url="https://github.com/facebook/zstd"
release="$(curl -s "${url}"/releases/latest | grep -oP '(?<=v)[\d\.]+')"
zstd="/tmp/zstd-${release}/"
wget -O /tmp/zstd.tar.gz \
"${url}/releases/download/v${release}/zstd-${release}.tar.gz"
tar -C /tmp -xzf /tmp/zstd.tar.gz
cp ${zstd}/lib/zstd.h module/zstd/lib/
cp ${zstd}/lib/zstd_errors.h module/zstd/lib/
${zstd}/contrib/single_file_libs/combine.sh \
-r ${zstd}/lib -o module/zstd/lib/zstd.c module/zstd/zstd-in.c
Note: if the zstd library for zfs is updated to a newer version, the macro list in include/zstd_compat_wrapper.h usually needs to be updated. this can be done with some hand crafting of the output of the following script: nm zstd.o | awk '{print "#define "$3 " zfs_" $3}' > macrotable
Altering ZSTD and breaking changes
If ZSTD made changes that break compatibility or you need to make breaking changes to the way we handle ZSTD, it is required to maintain backwards compatibility.
We already save the ZSTD version number within the block header to be used to add future compatibility checks and/or fixes. However, currently it is not actually used in such a way.