8f8cb840b0
and geom_uncompress(4): 1. mkuzip(8): - Proper support for eliminating all-zero blocks when compressing an image. This feature is already supported by the geom_uzip(4) module and CLOOP format in general, so it's just a matter of making mkuzip(8) match. It should be noted, however that this feature while it sounds great, results in very slight improvement in the overall compression ratio, since compressing default 16k all-zero block produces only 39 bytes compressed output block, which is 99.8% compression ratio. With typical average compression ratio of amd64 binaries and data being around 60-70% the difference between 99.8% and 100.0% is not that great further diluted by the ratio of number of zero blocks in the uncompressed image to the overall number of blocks being less than 0.5 (typically). However, this may be important from performance standpoint, so that kernel are not spinning its wheels decompressing those empty blocks every time this zero region is read. It could also be important when you create huge image mostly filled with zero blocks for testing purposes. - New feature allowing to de-duplicate output image. It turns out that if you twist CLOOP format a bit you can do that as well. And unlike zero-blocks elimination, this gives a noticeable improvement in the overall compression ratio, reducing output image by something like 3-4% on my test UFS2 3GB image consisting of full FreeBSD base system plus some of the packages (openjdk, apache etc), about 2.3GB worth of file data (800+MB compressed). The only caveat is that images created with this feature "on" would not work on older versions of FeeBSDxi kernel, hence it's turned off by default. - provide options to control both features and document them in manual page. - merge in all relevant LZMA compression support from the mkulzma(8), add new option to select between both. - switch license from ad-hoc beerware into standard 2-clause BSD. 2. geom_uzip(4): - implement support for de-duplicated images; - optimize some code paths to handle "all-zero" blocks without reading any compressed data; - beef up manual page to explain that geom_uzip(4) is not limited only to md(4) images. The compressed data can be written to the block device and accessed directly via magic of GEOM(4) and devfs(4), including to mount root fs from a compressed drive. - convert debug log code from being compiled in conditionally into being present all the time and provide two sysctls to turn it on or off. Due to intended use of the module, it can be used in environments where there may not be a luxury to put new kernel with debug code enabled. Having those options handy allows debug issues without as much problem by just having access to serial console or network shell access to a box/appliance. The resulting additional CPU cycles are just few int comparisons and branches, and those are minuscule when compared to data decompression which is the main feature of the module. - hopefully improve robustness and resiliency of the geom_uzip(4) by performing some of the data validation / range checking on the TOC entries and rejecting to attach to an image if those checks fail. - merge in all relevant LZMA decompression support from the geom_uncompress(4), enable automatically when appropriate format is indicated in the header. - move compilation work into its own worker thread so that it does not clog g_up. This allows multiple instances work in parallel utilizing smp cores. - document new knobs in the manual page. Reviewed by: adrian MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D5333
173 lines
5.1 KiB
Groff
173 lines
5.1 KiB
Groff
.\"-
|
|
.\" Copyright (c) 2004-2016 Maxim Sobolev <sobomax@FreeBSD.org>
|
|
.\" All rights reserved.
|
|
.\"
|
|
.\" Redistribution and use in source and binary forms, with or without
|
|
.\" modification, are permitted provided that the following conditions
|
|
.\" are met:
|
|
.\" 1. Redistributions of source code must retain the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer.
|
|
.\" 2. Redistributions in binary form must reproduce the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer in the
|
|
.\" documentation and/or other materials provided with the distribution.
|
|
.\"
|
|
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
|
|
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
|
|
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
.\" SUCH DAMAGE.
|
|
.\"
|
|
.\" $FreeBSD$
|
|
.\"
|
|
.Dd March 17, 2006
|
|
.Dt MKUZIP 8
|
|
.Os
|
|
.Sh NAME
|
|
.Nm mkuzip
|
|
.Nd compress disk image for use with
|
|
.Xr geom_uzip 4
|
|
class
|
|
.Sh SYNOPSIS
|
|
.Nm
|
|
.Op Fl v
|
|
.Op Fl o Ar outfile
|
|
.Op Fl s Ar cluster_size
|
|
.Ar infile
|
|
.Sh DESCRIPTION
|
|
The
|
|
.Nm
|
|
utility compresses a disk image file so that the
|
|
.Xr geom_uzip 4
|
|
class will be able to decompress the resulting image at run-time.
|
|
This allows for a significant reduction of size of disk image at
|
|
the expense of some CPU time required to decompress the data each
|
|
time it is read.
|
|
The
|
|
.Nm
|
|
utility
|
|
works in two phases:
|
|
.Bl -enum
|
|
.It
|
|
An
|
|
.Ar infile
|
|
image is split into clusters; each cluster is compressed using
|
|
.Xr zlib 3
|
|
or
|
|
.Xr lzma 3 .
|
|
.It
|
|
The resulting set of compressed clusters along with headers that allow
|
|
locating each individual cluster is written to the output file.
|
|
.El
|
|
.Pp
|
|
The options are:
|
|
.Bl -tag -width indent
|
|
.It Fl o Ar outfile
|
|
Name of the output file
|
|
.Ar outfile .
|
|
The default is to use the input name with the suffix
|
|
.Pa .uzip
|
|
for the
|
|
.Xr zlib 3
|
|
compression or
|
|
.Pa .ulzma
|
|
for the
|
|
.Xr lzma 3 .
|
|
.It Fl L
|
|
Use
|
|
.Xr lzma 3
|
|
compression algorithm instead of the default
|
|
.Xr zlib 3 .
|
|
The
|
|
.Xr lzma 3
|
|
provides noticeable better compression levels on the same data set
|
|
at the expense of much slower compression speed (10-20x) and somewhat slower
|
|
decompression (2-3x).
|
|
.It Fl s Ar cluster_size
|
|
Split the image into clusters of
|
|
.Ar cluster_size
|
|
bytes, 16384 bytes by default.
|
|
The
|
|
.Ar cluster_size
|
|
should be a multiple of 512 bytes.
|
|
.It Fl v
|
|
Display verbose messages.
|
|
.It Fl Z
|
|
Disable zero-blocks detection and elimination.
|
|
When this option is set, the
|
|
.Nm
|
|
would compress empty blocks (i.e. clusters that consist of only zero bytes)
|
|
just as it would any other block.
|
|
When the option is not set, the
|
|
.Nm
|
|
detects such blocks and skips them from the output.
|
|
Setting
|
|
.Fl Z
|
|
results is slight increase of compressed image size, typically less than 0.1%
|
|
of a final size of the compressed image.
|
|
.It Fl d
|
|
Enable de-duplication.
|
|
When the option is enabled the
|
|
.Nm
|
|
detects identical blocks in the input and replaces each subsequent occurence
|
|
of such block with pointer to the very first one in the output.
|
|
Setting this option results is moderate decrease of compressed image size,
|
|
typically around 3-5% of a final size of the compressed image.
|
|
.El
|
|
.Sh NOTES
|
|
The compression ratio largely depends on the cluster size used.
|
|
.\" The following two sentences are unclear: how can gzip(1) be
|
|
.\" used in a comparable fashion, and wouldn't a gzip-compressed
|
|
.\" image suffer from larger cluster sizes as well?
|
|
For large cluster sizes (16K and higher), typical compression ratios
|
|
are only 1-2% less than those achieved with
|
|
.Xr gzip 1 .
|
|
However, it should be kept in mind that larger cluster
|
|
sizes lead to higher overhead in the
|
|
.Xr geom_uzip 4
|
|
class, as the class has to decompress the whole cluster even if
|
|
only a few bytes from that cluster have to be read.
|
|
.Pp
|
|
The
|
|
.Nm
|
|
utility
|
|
inserts a short shell script at the beginning of the generated image,
|
|
which makes it possible to
|
|
.Dq run
|
|
the image just like any other shell script.
|
|
The script tries to load the
|
|
.Xr geom_uzip 4
|
|
class if it is not loaded, configure the image as an
|
|
.Xr md 4
|
|
disk device using
|
|
.Xr mdconfig 8 ,
|
|
and automatically mount it using
|
|
.Xr mount_cd9660 8
|
|
on the mount point provided as the first argument to the script.
|
|
.Pp
|
|
The de-duplication is a
|
|
.Fx
|
|
specific feature and while it does not require any changes to on-disk
|
|
compressed image format, however it did require some matching changes to the
|
|
.Xr geom_uzip 4
|
|
to handle resulting images correctly.
|
|
.Sh EXIT STATUS
|
|
.Ex -std
|
|
.Sh SEE ALSO
|
|
.Xr gzip 1 ,
|
|
.Xr xz 1 ,
|
|
.Xr zlib 3 ,
|
|
.Xr lzma 3 ,
|
|
.Xr geom 4 ,
|
|
.Xr geom_uzip 4 ,
|
|
.Xr md 4 ,
|
|
.Xr mdconfig 8 ,
|
|
.Xr mount_cd9660 8
|
|
.Sh AUTHORS
|
|
.An Maxim Sobolev Aq Mt sobomax@FreeBSD.org
|