Add `zstream redup` command to convert deduplicated send streams
Deduplicated send and receive is deprecated. To ease migration to the
new dedup-send-less world, the commit adds a `zstream redup` utility to
convert deduplicated send streams to normal streams, so that they can
continue to be received indefinitely.
The new `zstream` command also replaces the functionality of
`zstreamdump`, by way of the `zstream dump` subcommand. The
`zstreamdump` command is replaced by a shell script which invokes
`zstream dump`.
The way that `zstream redup` works under the hood is that as we read the
send stream, we build up a hash table which maps from `<GUID, object,
offset> -> <file_offset>`.
Whenever we see a WRITE record, we add a new entry to the hash table,
which indicates where in the stream file to find the WRITE record for
this block. (The key is `drr_toguid, drr_object, drr_offset`.)
For entries other than WRITE_BYREF, we pass them through unchanged
(except for the running checksum, which is recalculated).
For WRITE_BYREF records, we change them to WRITE records. We find the
referenced WRITE record by looking in the hash table (for the record
with key `drr_refguid, drr_refobject, drr_refoffset`), and then reading
the record header and payload from the specified offset in the stream
file. This is why the stream can not be a pipe. The found WRITE record
replaces the WRITE_BYREF record, with its `drr_toguid`, `drr_object`,
and `drr_offset` fields changed to be the same as the WRITE_BYREF's
(i.e. we are writing the same logical block, but with the data supplied
by the previous WRITE record).
This algorithm requires memory proportional to the number of WRITE
records (same as `zfs send -D`), but the size per WRITE record is
relatively low (40 bytes, vs. 72 for `zfs send -D`). A 1TB send stream
with 8KB blocks (`recordsize=8k`) would use around 5GB of RAM to
"redup".
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #10124
Closes #10156
2020-04-10 17:39:55 +00:00
|
|
|
SUBDIRS = zfs zpool zdb zhack zinject zstream zstreamdump ztest
|
2020-05-20 17:05:33 +00:00
|
|
|
SUBDIRS += fsck_zfs vdev_id raidz_test zfs_ids_to_path
|
2020-10-09 16:29:21 +00:00
|
|
|
SUBDIRS += zpool_influxdb
|
2019-06-05 01:05:46 +00:00
|
|
|
|
2021-01-22 20:54:34 +00:00
|
|
|
CPPCHECKDIRS = zfs zpool zdb zhack zinject zstream ztest
|
|
|
|
|
CPPCHECKDIRS += raidz_test zfs_ids_to_path zpool_influxdb
|
|
|
|
|
|
2019-06-05 01:05:46 +00:00
|
|
|
if USING_PYTHON
|
|
|
|
|
SUBDIRS += arcstat arc_summary dbufstat
|
|
|
|
|
endif
|
|
|
|
|
|
2019-11-11 17:24:04 +00:00
|
|
|
if BUILD_LINUX
|
2020-04-14 18:36:28 +00:00
|
|
|
SUBDIRS += mount_zfs zed zgenhostid zvol_id zvol_wait
|
2021-01-22 20:54:34 +00:00
|
|
|
CPPCHECKDIRS += mount_zfs zed zgenhostid zvol_id
|
2019-11-11 17:24:04 +00:00
|
|
|
endif
|
2021-01-22 20:54:34 +00:00
|
|
|
|
|
|
|
|
PHONY = cppcheck
|
|
|
|
|
cppcheck: $(CPPCHECKDIRS)
|
|
|
|
|
set -e ; for dir in $(CPPCHECKDIRS) ; do \
|
|
|
|
|
$(MAKE) -C $$dir cppcheck ; \
|
|
|
|
|
done
|
