import zstd 1.3.2
This commit is contained in:
parent
299df5d48c
commit
653667f9dc
Notes:
svn2git
2020-12-20 02:59:44 +00:00
svn path=/vendor/zstd/dist/; revision=325597 svn path=/vendor/zstd/1.3.2/; revision=325598; tag=vendor/zstd/1.3.2
339
COPYING
Normal file
339
COPYING
Normal file
@ -0,0 +1,339 @@
|
||||
GNU GENERAL PUBLIC LICENSE
|
||||
Version 2, June 1991
|
||||
|
||||
Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
|
||||
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The licenses for most software are designed to take away your
|
||||
freedom to share and change it. By contrast, the GNU General Public
|
||||
License is intended to guarantee your freedom to share and change free
|
||||
software--to make sure the software is free for all its users. This
|
||||
General Public License applies to most of the Free Software
|
||||
Foundation's software and to any other program whose authors commit to
|
||||
using it. (Some other Free Software Foundation software is covered by
|
||||
the GNU Lesser General Public License instead.) You can apply it to
|
||||
your programs, too.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not
|
||||
price. Our General Public Licenses are designed to make sure that you
|
||||
have the freedom to distribute copies of free software (and charge for
|
||||
this service if you wish), that you receive source code or can get it
|
||||
if you want it, that you can change the software or use pieces of it
|
||||
in new free programs; and that you know you can do these things.
|
||||
|
||||
To protect your rights, we need to make restrictions that forbid
|
||||
anyone to deny you these rights or to ask you to surrender the rights.
|
||||
These restrictions translate to certain responsibilities for you if you
|
||||
distribute copies of the software, or if you modify it.
|
||||
|
||||
For example, if you distribute copies of such a program, whether
|
||||
gratis or for a fee, you must give the recipients all the rights that
|
||||
you have. You must make sure that they, too, receive or can get the
|
||||
source code. And you must show them these terms so they know their
|
||||
rights.
|
||||
|
||||
We protect your rights with two steps: (1) copyright the software, and
|
||||
(2) offer you this license which gives you legal permission to copy,
|
||||
distribute and/or modify the software.
|
||||
|
||||
Also, for each author's protection and ours, we want to make certain
|
||||
that everyone understands that there is no warranty for this free
|
||||
software. If the software is modified by someone else and passed on, we
|
||||
want its recipients to know that what they have is not the original, so
|
||||
that any problems introduced by others will not reflect on the original
|
||||
authors' reputations.
|
||||
|
||||
Finally, any free program is threatened constantly by software
|
||||
patents. We wish to avoid the danger that redistributors of a free
|
||||
program will individually obtain patent licenses, in effect making the
|
||||
program proprietary. To prevent this, we have made it clear that any
|
||||
patent must be licensed for everyone's free use or not licensed at all.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow.
|
||||
|
||||
GNU GENERAL PUBLIC LICENSE
|
||||
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
|
||||
|
||||
0. This License applies to any program or other work which contains
|
||||
a notice placed by the copyright holder saying it may be distributed
|
||||
under the terms of this General Public License. The "Program", below,
|
||||
refers to any such program or work, and a "work based on the Program"
|
||||
means either the Program or any derivative work under copyright law:
|
||||
that is to say, a work containing the Program or a portion of it,
|
||||
either verbatim or with modifications and/or translated into another
|
||||
language. (Hereinafter, translation is included without limitation in
|
||||
the term "modification".) Each licensee is addressed as "you".
|
||||
|
||||
Activities other than copying, distribution and modification are not
|
||||
covered by this License; they are outside its scope. The act of
|
||||
running the Program is not restricted, and the output from the Program
|
||||
is covered only if its contents constitute a work based on the
|
||||
Program (independent of having been made by running the Program).
|
||||
Whether that is true depends on what the Program does.
|
||||
|
||||
1. You may copy and distribute verbatim copies of the Program's
|
||||
source code as you receive it, in any medium, provided that you
|
||||
conspicuously and appropriately publish on each copy an appropriate
|
||||
copyright notice and disclaimer of warranty; keep intact all the
|
||||
notices that refer to this License and to the absence of any warranty;
|
||||
and give any other recipients of the Program a copy of this License
|
||||
along with the Program.
|
||||
|
||||
You may charge a fee for the physical act of transferring a copy, and
|
||||
you may at your option offer warranty protection in exchange for a fee.
|
||||
|
||||
2. You may modify your copy or copies of the Program or any portion
|
||||
of it, thus forming a work based on the Program, and copy and
|
||||
distribute such modifications or work under the terms of Section 1
|
||||
above, provided that you also meet all of these conditions:
|
||||
|
||||
a) You must cause the modified files to carry prominent notices
|
||||
stating that you changed the files and the date of any change.
|
||||
|
||||
b) You must cause any work that you distribute or publish, that in
|
||||
whole or in part contains or is derived from the Program or any
|
||||
part thereof, to be licensed as a whole at no charge to all third
|
||||
parties under the terms of this License.
|
||||
|
||||
c) If the modified program normally reads commands interactively
|
||||
when run, you must cause it, when started running for such
|
||||
interactive use in the most ordinary way, to print or display an
|
||||
announcement including an appropriate copyright notice and a
|
||||
notice that there is no warranty (or else, saying that you provide
|
||||
a warranty) and that users may redistribute the program under
|
||||
these conditions, and telling the user how to view a copy of this
|
||||
License. (Exception: if the Program itself is interactive but
|
||||
does not normally print such an announcement, your work based on
|
||||
the Program is not required to print an announcement.)
|
||||
|
||||
These requirements apply to the modified work as a whole. If
|
||||
identifiable sections of that work are not derived from the Program,
|
||||
and can be reasonably considered independent and separate works in
|
||||
themselves, then this License, and its terms, do not apply to those
|
||||
sections when you distribute them as separate works. But when you
|
||||
distribute the same sections as part of a whole which is a work based
|
||||
on the Program, the distribution of the whole must be on the terms of
|
||||
this License, whose permissions for other licensees extend to the
|
||||
entire whole, and thus to each and every part regardless of who wrote it.
|
||||
|
||||
Thus, it is not the intent of this section to claim rights or contest
|
||||
your rights to work written entirely by you; rather, the intent is to
|
||||
exercise the right to control the distribution of derivative or
|
||||
collective works based on the Program.
|
||||
|
||||
In addition, mere aggregation of another work not based on the Program
|
||||
with the Program (or with a work based on the Program) on a volume of
|
||||
a storage or distribution medium does not bring the other work under
|
||||
the scope of this License.
|
||||
|
||||
3. You may copy and distribute the Program (or a work based on it,
|
||||
under Section 2) in object code or executable form under the terms of
|
||||
Sections 1 and 2 above provided that you also do one of the following:
|
||||
|
||||
a) Accompany it with the complete corresponding machine-readable
|
||||
source code, which must be distributed under the terms of Sections
|
||||
1 and 2 above on a medium customarily used for software interchange; or,
|
||||
|
||||
b) Accompany it with a written offer, valid for at least three
|
||||
years, to give any third party, for a charge no more than your
|
||||
cost of physically performing source distribution, a complete
|
||||
machine-readable copy of the corresponding source code, to be
|
||||
distributed under the terms of Sections 1 and 2 above on a medium
|
||||
customarily used for software interchange; or,
|
||||
|
||||
c) Accompany it with the information you received as to the offer
|
||||
to distribute corresponding source code. (This alternative is
|
||||
allowed only for noncommercial distribution and only if you
|
||||
received the program in object code or executable form with such
|
||||
an offer, in accord with Subsection b above.)
|
||||
|
||||
The source code for a work means the preferred form of the work for
|
||||
making modifications to it. For an executable work, complete source
|
||||
code means all the source code for all modules it contains, plus any
|
||||
associated interface definition files, plus the scripts used to
|
||||
control compilation and installation of the executable. However, as a
|
||||
special exception, the source code distributed need not include
|
||||
anything that is normally distributed (in either source or binary
|
||||
form) with the major components (compiler, kernel, and so on) of the
|
||||
operating system on which the executable runs, unless that component
|
||||
itself accompanies the executable.
|
||||
|
||||
If distribution of executable or object code is made by offering
|
||||
access to copy from a designated place, then offering equivalent
|
||||
access to copy the source code from the same place counts as
|
||||
distribution of the source code, even though third parties are not
|
||||
compelled to copy the source along with the object code.
|
||||
|
||||
4. You may not copy, modify, sublicense, or distribute the Program
|
||||
except as expressly provided under this License. Any attempt
|
||||
otherwise to copy, modify, sublicense or distribute the Program is
|
||||
void, and will automatically terminate your rights under this License.
|
||||
However, parties who have received copies, or rights, from you under
|
||||
this License will not have their licenses terminated so long as such
|
||||
parties remain in full compliance.
|
||||
|
||||
5. You are not required to accept this License, since you have not
|
||||
signed it. However, nothing else grants you permission to modify or
|
||||
distribute the Program or its derivative works. These actions are
|
||||
prohibited by law if you do not accept this License. Therefore, by
|
||||
modifying or distributing the Program (or any work based on the
|
||||
Program), you indicate your acceptance of this License to do so, and
|
||||
all its terms and conditions for copying, distributing or modifying
|
||||
the Program or works based on it.
|
||||
|
||||
6. Each time you redistribute the Program (or any work based on the
|
||||
Program), the recipient automatically receives a license from the
|
||||
original licensor to copy, distribute or modify the Program subject to
|
||||
these terms and conditions. You may not impose any further
|
||||
restrictions on the recipients' exercise of the rights granted herein.
|
||||
You are not responsible for enforcing compliance by third parties to
|
||||
this License.
|
||||
|
||||
7. If, as a consequence of a court judgment or allegation of patent
|
||||
infringement or for any other reason (not limited to patent issues),
|
||||
conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot
|
||||
distribute so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you
|
||||
may not distribute the Program at all. For example, if a patent
|
||||
license would not permit royalty-free redistribution of the Program by
|
||||
all those who receive copies directly or indirectly through you, then
|
||||
the only way you could satisfy both it and this License would be to
|
||||
refrain entirely from distribution of the Program.
|
||||
|
||||
If any portion of this section is held invalid or unenforceable under
|
||||
any particular circumstance, the balance of the section is intended to
|
||||
apply and the section as a whole is intended to apply in other
|
||||
circumstances.
|
||||
|
||||
It is not the purpose of this section to induce you to infringe any
|
||||
patents or other property right claims or to contest validity of any
|
||||
such claims; this section has the sole purpose of protecting the
|
||||
integrity of the free software distribution system, which is
|
||||
implemented by public license practices. Many people have made
|
||||
generous contributions to the wide range of software distributed
|
||||
through that system in reliance on consistent application of that
|
||||
system; it is up to the author/donor to decide if he or she is willing
|
||||
to distribute software through any other system and a licensee cannot
|
||||
impose that choice.
|
||||
|
||||
This section is intended to make thoroughly clear what is believed to
|
||||
be a consequence of the rest of this License.
|
||||
|
||||
8. If the distribution and/or use of the Program is restricted in
|
||||
certain countries either by patents or by copyrighted interfaces, the
|
||||
original copyright holder who places the Program under this License
|
||||
may add an explicit geographical distribution limitation excluding
|
||||
those countries, so that distribution is permitted only in or among
|
||||
countries not thus excluded. In such case, this License incorporates
|
||||
the limitation as if written in the body of this License.
|
||||
|
||||
9. The Free Software Foundation may publish revised and/or new versions
|
||||
of the General Public License from time to time. Such new versions will
|
||||
be similar in spirit to the present version, but may differ in detail to
|
||||
address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the Program
|
||||
specifies a version number of this License which applies to it and "any
|
||||
later version", you have the option of following the terms and conditions
|
||||
either of that version or of any later version published by the Free
|
||||
Software Foundation. If the Program does not specify a version number of
|
||||
this License, you may choose any version ever published by the Free Software
|
||||
Foundation.
|
||||
|
||||
10. If you wish to incorporate parts of the Program into other free
|
||||
programs whose distribution conditions are different, write to the author
|
||||
to ask for permission. For software which is copyrighted by the Free
|
||||
Software Foundation, write to the Free Software Foundation; we sometimes
|
||||
make exceptions for this. Our decision will be guided by the two goals
|
||||
of preserving the free status of all derivatives of our free software and
|
||||
of promoting the sharing and reuse of software generally.
|
||||
|
||||
NO WARRANTY
|
||||
|
||||
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
|
||||
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
|
||||
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
|
||||
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
|
||||
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
|
||||
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
|
||||
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
|
||||
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
|
||||
REPAIR OR CORRECTION.
|
||||
|
||||
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
|
||||
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
|
||||
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
|
||||
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
|
||||
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
|
||||
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
|
||||
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
|
||||
POSSIBILITY OF SUCH DAMAGES.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Programs
|
||||
|
||||
If you develop a new program, and you want it to be of the greatest
|
||||
possible use to the public, the best way to achieve this is to make it
|
||||
free software which everyone can redistribute and change under these terms.
|
||||
|
||||
To do so, attach the following notices to the program. It is safest
|
||||
to attach them to the start of each source file to most effectively
|
||||
convey the exclusion of warranty; and each file should have at least
|
||||
the "copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the program's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This program is free software; you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
the Free Software Foundation; either version 2 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU General Public License along
|
||||
with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
If the program is interactive, make it output a short notice like this
|
||||
when it starts in an interactive mode:
|
||||
|
||||
Gnomovision version 69, Copyright (C) year name of author
|
||||
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
|
||||
This is free software, and you are welcome to redistribute it
|
||||
under certain conditions; type `show c' for details.
|
||||
|
||||
The hypothetical commands `show w' and `show c' should show the appropriate
|
||||
parts of the General Public License. Of course, the commands you use may
|
||||
be called something other than `show w' and `show c'; they could even be
|
||||
mouse-clicks or menu items--whatever suits your program.
|
||||
|
||||
You should also get your employer (if you work as a programmer) or your
|
||||
school, if any, to sign a "copyright disclaimer" for the program, if
|
||||
necessary. Here is a sample; alter the names:
|
||||
|
||||
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
|
||||
`Gnomovision' (which makes passes at compilers) written by James Hacker.
|
||||
|
||||
<signature of Ty Coon>, 1 April 1989
|
||||
Ty Coon, President of Vice
|
||||
|
||||
This General Public License does not permit incorporating your program into
|
||||
proprietary programs. If your program is a subroutine library, you may
|
||||
consider it more useful to permit linking proprietary applications with the
|
||||
library. If this is what you want to do, use the GNU Lesser General
|
||||
Public License instead of this License.
|
@ -1,11 +0,0 @@
|
||||
Copyright (c) 2016-present, Facebook, Inc. All rights reserved.
|
||||
|
||||
The examples provided by Facebook are for non-commercial testing and evaluation
|
||||
purposes only. Facebook reserves all rights not expressly granted.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
|
||||
FACEBOOK BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
|
||||
ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
68
Makefile
68
Makefile
@ -1,10 +1,10 @@
|
||||
# ################################################################
|
||||
# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
|
||||
# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
|
||||
# All rights reserved.
|
||||
#
|
||||
# This source code is licensed under the BSD-style license found in the
|
||||
# LICENSE file in the root directory of this source tree. An additional grant
|
||||
# of patent rights can be found in the PATENTS file in the same directory.
|
||||
# This source code is licensed under both the BSD-style license (found in the
|
||||
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
# in the COPYING file in the root directory of this source tree).
|
||||
# ################################################################
|
||||
|
||||
PRGDIR = programs
|
||||
@ -12,6 +12,7 @@ ZSTDDIR = lib
|
||||
BUILDIR = build
|
||||
ZWRAPDIR = zlibWrapper
|
||||
TESTDIR = tests
|
||||
FUZZDIR = $(TESTDIR)/fuzz
|
||||
|
||||
# Define nul output
|
||||
VOID = /dev/null
|
||||
@ -29,15 +30,12 @@ default: lib-release zstd-release
|
||||
all: | allmost examples manual
|
||||
|
||||
.PHONY: allmost
|
||||
allmost:
|
||||
$(MAKE) -C $(ZSTDDIR) all
|
||||
$(MAKE) -C $(PRGDIR) all
|
||||
$(MAKE) -C $(TESTDIR) all
|
||||
allmost: allzstd
|
||||
$(MAKE) -C $(ZWRAPDIR) all
|
||||
|
||||
#skip zwrapper, can't build that on alternate architectures without the proper zlib installed
|
||||
.PHONY: allarch
|
||||
allarch:
|
||||
.PHONY: allzstd
|
||||
allzstd:
|
||||
$(MAKE) -C $(ZSTDDIR) all
|
||||
$(MAKE) -C $(PRGDIR) all
|
||||
$(MAKE) -C $(TESTDIR) all
|
||||
@ -100,6 +98,7 @@ clean:
|
||||
@$(MAKE) -C examples/ $@ > $(VOID)
|
||||
@$(MAKE) -C contrib/gen_html $@ > $(VOID)
|
||||
@$(RM) zstd$(EXT) zstdmt$(EXT) tmp*
|
||||
@$(RM) -r lz4
|
||||
@echo Cleaning completed
|
||||
|
||||
#------------------------------------------------------------------------------
|
||||
@ -114,7 +113,7 @@ CMAKE_PARAMS = -DZSTD_BUILD_CONTRIB:BOOL=ON -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_B
|
||||
list:
|
||||
@$(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' | sort | egrep -v -e '^[^[:alnum:]]' -e '^$@$$' | xargs
|
||||
|
||||
.PHONY: install clangtest gpptest armtest usan asan uasan
|
||||
.PHONY: install clangtest armtest usan asan uasan
|
||||
install:
|
||||
@$(MAKE) -C $(ZSTDDIR) $@
|
||||
@$(MAKE) -C $(PRGDIR) $@
|
||||
@ -158,16 +157,16 @@ m32build: clean
|
||||
$(MAKE) all32
|
||||
|
||||
armbuild: clean
|
||||
CC=arm-linux-gnueabi-gcc CFLAGS="-Werror" $(MAKE) allarch
|
||||
CC=arm-linux-gnueabi-gcc CFLAGS="-Werror" $(MAKE) allzstd
|
||||
|
||||
aarch64build: clean
|
||||
CC=aarch64-linux-gnu-gcc CFLAGS="-Werror" $(MAKE) allarch
|
||||
CC=aarch64-linux-gnu-gcc CFLAGS="-Werror" $(MAKE) allzstd
|
||||
|
||||
ppcbuild: clean
|
||||
CC=powerpc-linux-gnu-gcc CLAGS="-m32 -Wno-attributes -Werror" $(MAKE) allarch
|
||||
CC=powerpc-linux-gnu-gcc CLAGS="-m32 -Wno-attributes -Werror" $(MAKE) allzstd
|
||||
|
||||
ppc64build: clean
|
||||
CC=powerpc-linux-gnu-gcc CFLAGS="-m64 -Werror" $(MAKE) allarch
|
||||
CC=powerpc-linux-gnu-gcc CFLAGS="-m64 -Werror" $(MAKE) allzstd
|
||||
|
||||
armfuzz: clean
|
||||
CC=arm-linux-gnueabi-gcc QEMU_SYS=qemu-arm-static MOREFLAGS="-static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest
|
||||
@ -181,8 +180,10 @@ ppcfuzz: clean
|
||||
ppc64fuzz: clean
|
||||
CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc64-static MOREFLAGS="-m64 -static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest
|
||||
|
||||
gpptest: clean
|
||||
CC=$(CXX) $(MAKE) -C $(PRGDIR) all CFLAGS="-O3 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Werror"
|
||||
.PHONY: cxxtest
|
||||
cxxtest: CXXFLAGS += -Wall -Wextra -Wundef -Wshadow -Wcast-align -Werror
|
||||
cxxtest: clean
|
||||
$(MAKE) -C $(PRGDIR) all CC="$(CXX) -Wno-deprecated" CFLAGS="$(CXXFLAGS)" # adding -Wno-deprecated to avoid clang++ warning on dealing with C files directly
|
||||
|
||||
gcc5test: clean
|
||||
gcc-5 -v
|
||||
@ -218,6 +219,15 @@ arm-ppc-compilation:
|
||||
$(MAKE) -C $(PRGDIR) clean zstd CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc-static ZSTDRTTEST= MOREFLAGS="-Werror -Wno-attributes -static"
|
||||
$(MAKE) -C $(PRGDIR) clean zstd CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc64-static ZSTDRTTEST= MOREFLAGS="-m64 -static"
|
||||
|
||||
regressiontest:
|
||||
$(MAKE) -C $(FUZZDIR) regressiontest
|
||||
|
||||
uasanregressiontest:
|
||||
$(MAKE) -C $(FUZZDIR) regressiontest CC=clang CXX=clang++ CFLAGS="-O3 -fsanitize=address,undefined" CXXFLAGS="-O3 -fsanitize=address,undefined"
|
||||
|
||||
msanregressiontest:
|
||||
$(MAKE) -C $(FUZZDIR) regressiontest CC=clang CXX=clang++ CFLAGS="-O3 -fsanitize=memory" CXXFLAGS="-O3 -fsanitize=memory"
|
||||
|
||||
# run UBsan with -fsanitize-recover=signed-integer-overflow
|
||||
# due to a bug in UBsan when doing pointer subtraction
|
||||
# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63303
|
||||
@ -277,6 +287,10 @@ gpp6install: apt-add-repo
|
||||
clang38install:
|
||||
APT_PACKAGES="clang-3.8" $(MAKE) apt-install
|
||||
|
||||
# Ubuntu 14.04 ships a too-old lz4
|
||||
lz4install:
|
||||
[ -e lz4 ] || git clone https://github.com/lz4/lz4 && sudo $(MAKE) -C lz4 install
|
||||
|
||||
endif
|
||||
|
||||
|
||||
@ -287,7 +301,7 @@ endif
|
||||
|
||||
|
||||
#------------------------------------------------------------------------
|
||||
#make tests validated only for MSYS, Linux, OSX, kFreeBSD and Hurd targets
|
||||
# target specific tests
|
||||
#------------------------------------------------------------------------
|
||||
ifneq (,$(filter $(HOST_OS),MSYS POSIX))
|
||||
cmakebuild:
|
||||
@ -297,38 +311,38 @@ cmakebuild:
|
||||
cd $(BUILDIR)/cmake/build ; cmake -DCMAKE_INSTALL_PREFIX:PATH=~/install_test_dir $(CMAKE_PARAMS) .. ; $(MAKE) install ; $(MAKE) uninstall
|
||||
|
||||
c90build: clean
|
||||
gcc -v
|
||||
$(CC) -v
|
||||
CFLAGS="-std=c90" $(MAKE) allmost # will fail, due to missing support for `long long`
|
||||
|
||||
gnu90build: clean
|
||||
gcc -v
|
||||
$(CC) -v
|
||||
CFLAGS="-std=gnu90" $(MAKE) allmost
|
||||
|
||||
c99build: clean
|
||||
gcc -v
|
||||
$(CC) -v
|
||||
CFLAGS="-std=c99" $(MAKE) allmost
|
||||
|
||||
gnu99build: clean
|
||||
gcc -v
|
||||
$(CC) -v
|
||||
CFLAGS="-std=gnu99" $(MAKE) allmost
|
||||
|
||||
c11build: clean
|
||||
gcc -v
|
||||
$(CC) -v
|
||||
CFLAGS="-std=c11" $(MAKE) allmost
|
||||
|
||||
bmix64build: clean
|
||||
gcc -v
|
||||
$(CC) -v
|
||||
CFLAGS="-O3 -mbmi -Werror" $(MAKE) -C $(TESTDIR) test
|
||||
|
||||
bmix32build: clean
|
||||
gcc -v
|
||||
$(CC) -v
|
||||
CFLAGS="-O3 -mbmi -mx32 -Werror" $(MAKE) -C $(TESTDIR) test
|
||||
|
||||
bmi32build: clean
|
||||
gcc -v
|
||||
$(CC) -v
|
||||
CFLAGS="-O3 -mbmi -m32 -Werror" $(MAKE) -C $(TESTDIR) test
|
||||
|
||||
staticAnalyze: clean
|
||||
gcc -v
|
||||
$(CC) -v
|
||||
CPPFLAGS=-g scan-build --status-bugs -v $(MAKE) all
|
||||
endif
|
||||
|
23
NEWS
23
NEWS
@ -1,3 +1,26 @@
|
||||
v1.3.2
|
||||
new : long range mode, using --long command, by Stella Lau (@stellamplau)
|
||||
new : ability to generate and decode magicless frames (#591)
|
||||
changed : maximum nb of threads reduced to 200, to avoid address space exhaustion in 32-bits mode
|
||||
fix : multi-threading compression works with custom allocators
|
||||
fix : ZSTD_sizeof_CStream() was over-evaluating memory usage
|
||||
fix : a rare compression bug when compression generates very large distances and bunch of other conditions (only possible at --ultra -22)
|
||||
fix : 32-bits build can now decode large offsets (levels 21+)
|
||||
cli : added LZ4 frame support by default, by Felix Handte (@felixhandte)
|
||||
cli : improved --list output
|
||||
cli : new : can split input file for dictionary training, using command -B#
|
||||
cli : new : clean operation artefact on Ctrl-C interruption
|
||||
cli : fix : do not change /dev/null permissions when using command -t with root access, reported by @mike155 (#851)
|
||||
cli : fix : write file size in header in multiple-files mode
|
||||
api : added macro ZSTD_COMPRESSBOUND() for static allocation
|
||||
api : experimental : new advanced decompression API
|
||||
api : fix : sizeof_CCtx() used to over-estimate
|
||||
build: fix : no-multithread variant compiles without pool.c dependency, reported by Mitchell Blank Jr (@mitchblank) (#819)
|
||||
build: better compatibility with reproducible builds, by Bernhard M. Wiedemann (@bmwiedemann) (#818)
|
||||
example : added streaming_memory_usage
|
||||
license : changed /examples license to BSD + GPLv2
|
||||
license : fix a few header files to reflect new license (#825)
|
||||
|
||||
v1.3.1
|
||||
New license : BSD + GPLv2
|
||||
perf: substantially decreased memory usage in Multi-threading mode, thanks to reports by Tino Reichardt (@mcmilk)
|
||||
|
12
appveyor.yml
12
appveyor.yml
@ -9,19 +9,19 @@
|
||||
- COMPILER: "gcc"
|
||||
HOST: "mingw"
|
||||
PLATFORM: "x64"
|
||||
SCRIPT: "make allarch && make -C tests test-symbols fullbench-dll fullbench-lib"
|
||||
SCRIPT: "make allzstd MOREFLAGS=-static && make -C tests test-symbols fullbench-dll fullbench-lib"
|
||||
ARTIFACT: "true"
|
||||
BUILD: "true"
|
||||
- COMPILER: "gcc"
|
||||
HOST: "mingw"
|
||||
PLATFORM: "x86"
|
||||
SCRIPT: "make allarch"
|
||||
SCRIPT: "make allzstd MOREFLAGS=-static"
|
||||
ARTIFACT: "true"
|
||||
BUILD: "true"
|
||||
- COMPILER: "clang"
|
||||
HOST: "mingw"
|
||||
PLATFORM: "x64"
|
||||
SCRIPT: "MOREFLAGS='--target=x86_64-w64-mingw32 -Werror -Wconversion -Wno-sign-conversion' make allarch"
|
||||
SCRIPT: "MOREFLAGS='--target=x86_64-w64-mingw32 -Werror -Wconversion -Wno-sign-conversion' make allzstd"
|
||||
BUILD: "true"
|
||||
|
||||
- COMPILER: "gcc"
|
||||
@ -172,15 +172,15 @@
|
||||
- COMPILER: "gcc"
|
||||
HOST: "mingw"
|
||||
PLATFORM: "x64"
|
||||
SCRIPT: "make allarch"
|
||||
SCRIPT: "make allzstd"
|
||||
- COMPILER: "gcc"
|
||||
HOST: "mingw"
|
||||
PLATFORM: "x86"
|
||||
SCRIPT: "make allarch"
|
||||
SCRIPT: "make allzstd"
|
||||
- COMPILER: "clang"
|
||||
HOST: "mingw"
|
||||
PLATFORM: "x64"
|
||||
SCRIPT: "MOREFLAGS='--target=x86_64-w64-mingw32 -Werror -Wconversion -Wno-sign-conversion' make allarch"
|
||||
SCRIPT: "MOREFLAGS='--target=x86_64-w64-mingw32 -Werror -Wconversion -Wno-sign-conversion' make allzstd"
|
||||
|
||||
- COMPILER: "visual"
|
||||
HOST: "visual"
|
||||
|
@ -9,7 +9,7 @@ dependencies:
|
||||
test:
|
||||
override:
|
||||
- ? |
|
||||
if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then cc -v; make all && make clean; fi &&
|
||||
if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then cc -v; make all && make clean && make -C lib libzstd-nomt && make clean; fi &&
|
||||
if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make gnu90build && make clean; fi
|
||||
:
|
||||
parallel: true
|
||||
@ -45,7 +45,7 @@ test:
|
||||
parallel: true
|
||||
- ? |
|
||||
if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then make ppc64build && make clean; fi &&
|
||||
if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make gcc7build && make clean; fi #could add another test here
|
||||
if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make gcc7build && make clean; fi
|
||||
:
|
||||
parallel: true
|
||||
- ? |
|
||||
@ -53,6 +53,11 @@ test:
|
||||
if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make -C tests test-legacy test-longmatch test-symbols && make clean; fi
|
||||
:
|
||||
parallel: true
|
||||
- ? |
|
||||
if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then make -j regressiontest && make clean; fi &&
|
||||
if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then true; fi # Could add another test here
|
||||
:
|
||||
parallel: true
|
||||
|
||||
post:
|
||||
- echo Circle CI tests finished
|
||||
|
@ -1,11 +1,11 @@
|
||||
# ##########################################################################
|
||||
# ################################################################
|
||||
# Copyright (c) 2016-present, Facebook, Inc.
|
||||
# All rights reserved.
|
||||
#
|
||||
# This source code is licensed under the BSD-style license found in the
|
||||
# LICENSE file in the root directory of this source tree. An additional grant
|
||||
# of patent rights can be found in the PATENTS file in the same directory.
|
||||
# ##########################################################################
|
||||
# This source code is licensed under both the BSD-style license (found in the
|
||||
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
# in the COPYING file in the root directory of this source tree).
|
||||
# ################################################################
|
||||
|
||||
CFLAGS ?= -O3
|
||||
CFLAGS += -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wno-comment
|
||||
|
@ -2,9 +2,9 @@
|
||||
* Copyright (c) 2016-present, Przemyslaw Skibinski, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
#include <iostream>
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
|
@ -1,11 +1,11 @@
|
||||
# ##########################################################################
|
||||
# ################################################################
|
||||
# Copyright (c) 2016-present, Facebook, Inc.
|
||||
# All rights reserved.
|
||||
#
|
||||
# This source code is licensed under the BSD-style license found in the
|
||||
# LICENSE file in the root directory of this source tree. An additional grant
|
||||
# of patent rights can be found in the PATENTS file in the same directory.
|
||||
# ##########################################################################
|
||||
# This source code is licensed under both the BSD-style license (found in the
|
||||
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
# in the COPYING file in the root directory of this source tree).
|
||||
# ################################################################
|
||||
|
||||
# Standard variables for installation
|
||||
DESTDIR ?=
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#include "Options.h"
|
||||
#include "util.h"
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#include "Pzstd.h"
|
||||
#include "SkippableFrame.h"
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#include "SkippableFrame.h"
|
||||
#include "mem.h"
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#include "ErrorHolder.h"
|
||||
#include "Options.h"
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#include "Options.h"
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#include "Pzstd.h"
|
||||
extern "C" {
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
extern "C" {
|
||||
#include "datagen.h"
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
/**
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
/**
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#pragma once
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#include "utils/Buffer.h"
|
||||
#include "utils/Range.h"
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#include "utils/Range.h"
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#include "utils/ResourcePool.h"
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#include "utils/ScopeGuard.h"
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#include "utils/ThreadPool.h"
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/**
|
||||
/*
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
#include "utils/Buffer.h"
|
||||
#include "utils/WorkQueue.h"
|
||||
|
4
contrib/seekable_format/examples/.gitignore
vendored
Normal file
4
contrib/seekable_format/examples/.gitignore
vendored
Normal file
@ -0,0 +1,4 @@
|
||||
seekable_compression
|
||||
seekable_decompression
|
||||
parallel_processing
|
||||
parallel_compression
|
42
contrib/seekable_format/examples/Makefile
Normal file
42
contrib/seekable_format/examples/Makefile
Normal file
@ -0,0 +1,42 @@
|
||||
# ################################################################
|
||||
# Copyright (c) 2017-present, Facebook, Inc.
|
||||
# All rights reserved.
|
||||
#
|
||||
# This source code is licensed under both the BSD-style license (found in the
|
||||
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
# in the COPYING file in the root directory of this source tree).
|
||||
# ################################################################
|
||||
|
||||
# This Makefile presumes libzstd is built, using `make` in / or /lib/
|
||||
|
||||
LDFLAGS += ../../../lib/libzstd.a
|
||||
CPPFLAGS += -I../ -I../../../lib -I../../../lib/common
|
||||
|
||||
CFLAGS ?= -O3
|
||||
CFLAGS += -g
|
||||
|
||||
SEEKABLE_OBJS = ../zstdseek_compress.c ../zstdseek_decompress.c
|
||||
|
||||
.PHONY: default all clean test
|
||||
|
||||
default: all
|
||||
|
||||
all: seekable_compression seekable_decompression parallel_processing
|
||||
|
||||
seekable_compression : seekable_compression.c $(SEEKABLE_OBJS)
|
||||
$(CC) $(CPPFLAGS) $(CFLAGS) $^ $(LDFLAGS) -o $@
|
||||
|
||||
seekable_decompression : seekable_decompression.c $(SEEKABLE_OBJS)
|
||||
$(CC) $(CPPFLAGS) $(CFLAGS) $^ $(LDFLAGS) -o $@
|
||||
|
||||
parallel_processing : parallel_processing.c $(SEEKABLE_OBJS)
|
||||
$(CC) $(CPPFLAGS) $(CFLAGS) $^ $(LDFLAGS) -o $@ -pthread
|
||||
|
||||
parallel_compression : parallel_compression.c $(SEEKABLE_OBJS)
|
||||
$(CC) $(CPPFLAGS) $(CFLAGS) $^ $(LDFLAGS) -o $@ -pthread
|
||||
|
||||
clean:
|
||||
@rm -f core *.o tmp* result* *.zst \
|
||||
seekable_compression seekable_decompression \
|
||||
parallel_processing parallel_compression
|
||||
@echo Cleaning completed
|
215
contrib/seekable_format/examples/parallel_compression.c
Normal file
215
contrib/seekable_format/examples/parallel_compression.c
Normal file
@ -0,0 +1,215 @@
|
||||
/*
|
||||
* Copyright (c) 2017-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
#include <stdlib.h> // malloc, free, exit, atoi
|
||||
#include <stdio.h> // fprintf, perror, feof, fopen, etc.
|
||||
#include <string.h> // strlen, memset, strcat
|
||||
#define ZSTD_STATIC_LINKING_ONLY
|
||||
#include <zstd.h> // presumes zstd library is installed
|
||||
#include <zstd_errors.h>
|
||||
#if defined(WIN32) || defined(_WIN32)
|
||||
# include <windows.h>
|
||||
# define SLEEP(x) Sleep(x)
|
||||
#else
|
||||
# include <unistd.h>
|
||||
# define SLEEP(x) usleep(x * 1000)
|
||||
#endif
|
||||
|
||||
#define XXH_NAMESPACE ZSTD_
|
||||
#include "xxhash.h"
|
||||
|
||||
#include "pool.h" // use zstd thread pool for demo
|
||||
|
||||
#include "zstd_seekable.h"
|
||||
|
||||
static void* malloc_orDie(size_t size)
|
||||
{
|
||||
void* const buff = malloc(size);
|
||||
if (buff) return buff;
|
||||
/* error */
|
||||
perror("malloc:");
|
||||
exit(1);
|
||||
}
|
||||
|
||||
static FILE* fopen_orDie(const char *filename, const char *instruction)
|
||||
{
|
||||
FILE* const inFile = fopen(filename, instruction);
|
||||
if (inFile) return inFile;
|
||||
/* error */
|
||||
perror(filename);
|
||||
exit(3);
|
||||
}
|
||||
|
||||
static size_t fread_orDie(void* buffer, size_t sizeToRead, FILE* file)
|
||||
{
|
||||
size_t const readSize = fread(buffer, 1, sizeToRead, file);
|
||||
if (readSize == sizeToRead) return readSize; /* good */
|
||||
if (feof(file)) return readSize; /* good, reached end of file */
|
||||
/* error */
|
||||
perror("fread");
|
||||
exit(4);
|
||||
}
|
||||
|
||||
static size_t fwrite_orDie(const void* buffer, size_t sizeToWrite, FILE* file)
|
||||
{
|
||||
size_t const writtenSize = fwrite(buffer, 1, sizeToWrite, file);
|
||||
if (writtenSize == sizeToWrite) return sizeToWrite; /* good */
|
||||
/* error */
|
||||
perror("fwrite");
|
||||
exit(5);
|
||||
}
|
||||
|
||||
static size_t fclose_orDie(FILE* file)
|
||||
{
|
||||
if (!fclose(file)) return 0;
|
||||
/* error */
|
||||
perror("fclose");
|
||||
exit(6);
|
||||
}
|
||||
|
||||
static void fseek_orDie(FILE* file, long int offset, int origin)
|
||||
{
|
||||
if (!fseek(file, offset, origin)) {
|
||||
if (!fflush(file)) return;
|
||||
}
|
||||
/* error */
|
||||
perror("fseek");
|
||||
exit(7);
|
||||
}
|
||||
|
||||
static long int ftell_orDie(FILE* file)
|
||||
{
|
||||
long int off = ftell(file);
|
||||
if (off != -1) return off;
|
||||
/* error */
|
||||
perror("ftell");
|
||||
exit(8);
|
||||
}
|
||||
|
||||
struct job {
|
||||
const void* src;
|
||||
size_t srcSize;
|
||||
void* dst;
|
||||
size_t dstSize;
|
||||
|
||||
unsigned checksum;
|
||||
|
||||
int compressionLevel;
|
||||
int done;
|
||||
};
|
||||
|
||||
static void compressFrame(void* opaque)
|
||||
{
|
||||
struct job* job = opaque;
|
||||
|
||||
job->checksum = XXH64(job->src, job->srcSize, 0);
|
||||
|
||||
size_t ret = ZSTD_compress(job->dst, job->dstSize, job->src, job->srcSize, job->compressionLevel);
|
||||
if (ZSTD_isError(ret)) {
|
||||
fprintf(stderr, "ZSTD_compress() error : %s \n", ZSTD_getErrorName(ret));
|
||||
exit(20);
|
||||
}
|
||||
|
||||
job->dstSize = ret;
|
||||
job->done = 1;
|
||||
}
|
||||
|
||||
static void compressFile_orDie(const char* fname, const char* outName, int cLevel, unsigned frameSize, int nbThreads)
|
||||
{
|
||||
POOL_ctx* pool = POOL_create(nbThreads, nbThreads);
|
||||
if (pool == NULL) { fprintf(stderr, "POOL_create() error \n"); exit(9); }
|
||||
|
||||
FILE* const fin = fopen_orDie(fname, "rb");
|
||||
FILE* const fout = fopen_orDie(outName, "wb");
|
||||
|
||||
if (ZSTD_compressBound(frameSize) > 0xFFFFFFFFU) { fprintf(stderr, "Frame size too large \n"); exit(10); }
|
||||
unsigned dstSize = ZSTD_compressBound(frameSize);
|
||||
|
||||
|
||||
fseek_orDie(fin, 0, SEEK_END);
|
||||
long int length = ftell_orDie(fin);
|
||||
fseek_orDie(fin, 0, SEEK_SET);
|
||||
|
||||
size_t numFrames = (length + frameSize - 1) / frameSize;
|
||||
|
||||
struct job* jobs = malloc_orDie(sizeof(struct job) * numFrames);
|
||||
|
||||
size_t i;
|
||||
for(i = 0; i < numFrames; i++) {
|
||||
void* in = malloc_orDie(frameSize);
|
||||
void* out = malloc_orDie(dstSize);
|
||||
|
||||
size_t inSize = fread_orDie(in, frameSize, fin);
|
||||
|
||||
jobs[i].src = in;
|
||||
jobs[i].srcSize = inSize;
|
||||
jobs[i].dst = out;
|
||||
jobs[i].dstSize = dstSize;
|
||||
jobs[i].compressionLevel = cLevel;
|
||||
jobs[i].done = 0;
|
||||
POOL_add(pool, compressFrame, &jobs[i]);
|
||||
}
|
||||
|
||||
ZSTD_frameLog* fl = ZSTD_seekable_createFrameLog(1);
|
||||
if (fl == NULL) { fprintf(stderr, "ZSTD_seekable_createFrameLog() failed \n"); exit(11); }
|
||||
for (i = 0; i < numFrames; i++) {
|
||||
while (!jobs[i].done) SLEEP(5); /* wake up every 5 milliseconds to check */
|
||||
fwrite_orDie(jobs[i].dst, jobs[i].dstSize, fout);
|
||||
free((void*)jobs[i].src);
|
||||
free(jobs[i].dst);
|
||||
|
||||
size_t ret = ZSTD_seekable_logFrame(fl, jobs[i].dstSize, jobs[i].srcSize, jobs[i].checksum);
|
||||
if (ZSTD_isError(ret)) { fprintf(stderr, "ZSTD_seekable_logFrame() error : %s \n", ZSTD_getErrorName(ret)); }
|
||||
}
|
||||
|
||||
{ unsigned char seekTableBuff[1024];
|
||||
ZSTD_outBuffer out = {seekTableBuff, 1024, 0};
|
||||
while (ZSTD_seekable_writeSeekTable(fl, &out) != 0) {
|
||||
fwrite_orDie(seekTableBuff, out.pos, fout);
|
||||
out.pos = 0;
|
||||
}
|
||||
fwrite_orDie(seekTableBuff, out.pos, fout);
|
||||
}
|
||||
|
||||
ZSTD_seekable_freeFrameLog(fl);
|
||||
free(jobs);
|
||||
fclose_orDie(fout);
|
||||
fclose_orDie(fin);
|
||||
}
|
||||
|
||||
static const char* createOutFilename_orDie(const char* filename)
|
||||
{
|
||||
size_t const inL = strlen(filename);
|
||||
size_t const outL = inL + 5;
|
||||
void* outSpace = malloc_orDie(outL);
|
||||
memset(outSpace, 0, outL);
|
||||
strcat(outSpace, filename);
|
||||
strcat(outSpace, ".zst");
|
||||
return (const char*)outSpace;
|
||||
}
|
||||
|
||||
int main(int argc, const char** argv) {
|
||||
const char* const exeName = argv[0];
|
||||
if (argc!=4) {
|
||||
printf("wrong arguments\n");
|
||||
printf("usage:\n");
|
||||
printf("%s FILE FRAME_SIZE NB_THREADS\n", exeName);
|
||||
return 1;
|
||||
}
|
||||
|
||||
{ const char* const inFileName = argv[1];
|
||||
unsigned const frameSize = (unsigned)atoi(argv[2]);
|
||||
int const nbThreads = atoi(argv[3]);
|
||||
|
||||
const char* const outFileName = createOutFilename_orDie(inFileName);
|
||||
compressFile_orDie(inFileName, outFileName, 5, frameSize, nbThreads);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
194
contrib/seekable_format/examples/parallel_processing.c
Normal file
194
contrib/seekable_format/examples/parallel_processing.c
Normal file
@ -0,0 +1,194 @@
|
||||
/*
|
||||
* Copyright (c) 2017-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
/*
|
||||
* A simple demo that sums up all the bytes in the file in parallel using
|
||||
* seekable decompression and the zstd thread pool
|
||||
*/
|
||||
|
||||
#include <stdlib.h> // malloc, exit
|
||||
#include <stdio.h> // fprintf, perror, feof
|
||||
#include <string.h> // strerror
|
||||
#include <errno.h> // errno
|
||||
#define ZSTD_STATIC_LINKING_ONLY
|
||||
#include <zstd.h> // presumes zstd library is installed
|
||||
#include <zstd_errors.h>
|
||||
#if defined(WIN32) || defined(_WIN32)
|
||||
# include <windows.h>
|
||||
# define SLEEP(x) Sleep(x)
|
||||
#else
|
||||
# include <unistd.h>
|
||||
# define SLEEP(x) usleep(x * 1000)
|
||||
#endif
|
||||
|
||||
#include "pool.h" // use zstd thread pool for demo
|
||||
|
||||
#include "zstd_seekable.h"
|
||||
|
||||
#define MIN(a, b) ((a) < (b) ? (a) : (b))
|
||||
|
||||
static void* malloc_orDie(size_t size)
|
||||
{
|
||||
void* const buff = malloc(size);
|
||||
if (buff) return buff;
|
||||
/* error */
|
||||
perror("malloc");
|
||||
exit(1);
|
||||
}
|
||||
|
||||
static void* realloc_orDie(void* ptr, size_t size)
|
||||
{
|
||||
ptr = realloc(ptr, size);
|
||||
if (ptr) return ptr;
|
||||
/* error */
|
||||
perror("realloc");
|
||||
exit(1);
|
||||
}
|
||||
|
||||
static FILE* fopen_orDie(const char *filename, const char *instruction)
|
||||
{
|
||||
FILE* const inFile = fopen(filename, instruction);
|
||||
if (inFile) return inFile;
|
||||
/* error */
|
||||
perror(filename);
|
||||
exit(3);
|
||||
}
|
||||
|
||||
static size_t fread_orDie(void* buffer, size_t sizeToRead, FILE* file)
|
||||
{
|
||||
size_t const readSize = fread(buffer, 1, sizeToRead, file);
|
||||
if (readSize == sizeToRead) return readSize; /* good */
|
||||
if (feof(file)) return readSize; /* good, reached end of file */
|
||||
/* error */
|
||||
perror("fread");
|
||||
exit(4);
|
||||
}
|
||||
|
||||
static size_t fwrite_orDie(const void* buffer, size_t sizeToWrite, FILE* file)
|
||||
{
|
||||
size_t const writtenSize = fwrite(buffer, 1, sizeToWrite, file);
|
||||
if (writtenSize == sizeToWrite) return sizeToWrite; /* good */
|
||||
/* error */
|
||||
perror("fwrite");
|
||||
exit(5);
|
||||
}
|
||||
|
||||
static size_t fclose_orDie(FILE* file)
|
||||
{
|
||||
if (!fclose(file)) return 0;
|
||||
/* error */
|
||||
perror("fclose");
|
||||
exit(6);
|
||||
}
|
||||
|
||||
static void fseek_orDie(FILE* file, long int offset, int origin) {
|
||||
if (!fseek(file, offset, origin)) {
|
||||
if (!fflush(file)) return;
|
||||
}
|
||||
/* error */
|
||||
perror("fseek");
|
||||
exit(7);
|
||||
}
|
||||
|
||||
struct sum_job {
|
||||
const char* fname;
|
||||
unsigned long long sum;
|
||||
unsigned frameNb;
|
||||
int done;
|
||||
};
|
||||
|
||||
static void sumFrame(void* opaque)
|
||||
{
|
||||
struct sum_job* job = (struct sum_job*)opaque;
|
||||
job->done = 0;
|
||||
|
||||
FILE* const fin = fopen_orDie(job->fname, "rb");
|
||||
|
||||
ZSTD_seekable* const seekable = ZSTD_seekable_create();
|
||||
if (seekable==NULL) { fprintf(stderr, "ZSTD_seekable_create() error \n"); exit(10); }
|
||||
|
||||
size_t const initResult = ZSTD_seekable_initFile(seekable, fin);
|
||||
if (ZSTD_isError(initResult)) { fprintf(stderr, "ZSTD_seekable_init() error : %s \n", ZSTD_getErrorName(initResult)); exit(11); }
|
||||
|
||||
size_t const frameSize = ZSTD_seekable_getFrameDecompressedSize(seekable, job->frameNb);
|
||||
unsigned char* data = malloc_orDie(frameSize);
|
||||
|
||||
size_t result = ZSTD_seekable_decompressFrame(seekable, data, frameSize, job->frameNb);
|
||||
if (ZSTD_isError(result)) { fprintf(stderr, "ZSTD_seekable_decompressFrame() error : %s \n", ZSTD_getErrorName(result)); exit(12); }
|
||||
|
||||
unsigned long long sum = 0;
|
||||
size_t i;
|
||||
for (i = 0; i < frameSize; i++) {
|
||||
sum += data[i];
|
||||
}
|
||||
job->sum = sum;
|
||||
job->done = 1;
|
||||
|
||||
fclose(fin);
|
||||
ZSTD_seekable_free(seekable);
|
||||
free(data);
|
||||
}
|
||||
|
||||
static void sumFile_orDie(const char* fname, int nbThreads)
|
||||
{
|
||||
POOL_ctx* pool = POOL_create(nbThreads, nbThreads);
|
||||
if (pool == NULL) { fprintf(stderr, "POOL_create() error \n"); exit(9); }
|
||||
|
||||
FILE* const fin = fopen_orDie(fname, "rb");
|
||||
|
||||
ZSTD_seekable* const seekable = ZSTD_seekable_create();
|
||||
if (seekable==NULL) { fprintf(stderr, "ZSTD_seekable_create() error \n"); exit(10); }
|
||||
|
||||
size_t const initResult = ZSTD_seekable_initFile(seekable, fin);
|
||||
if (ZSTD_isError(initResult)) { fprintf(stderr, "ZSTD_seekable_init() error : %s \n", ZSTD_getErrorName(initResult)); exit(11); }
|
||||
|
||||
size_t const numFrames = ZSTD_seekable_getNumFrames(seekable);
|
||||
struct sum_job* jobs = (struct sum_job*)malloc(numFrames * sizeof(struct sum_job));
|
||||
|
||||
size_t i;
|
||||
for (i = 0; i < numFrames; i++) {
|
||||
jobs[i] = (struct sum_job){ fname, 0, i, 0 };
|
||||
POOL_add(pool, sumFrame, &jobs[i]);
|
||||
}
|
||||
|
||||
unsigned long long total = 0;
|
||||
|
||||
for (i = 0; i < numFrames; i++) {
|
||||
while (!jobs[i].done) SLEEP(5); /* wake up every 5 milliseconds to check */
|
||||
total += jobs[i].sum;
|
||||
}
|
||||
|
||||
printf("Sum: %llu\n", total);
|
||||
|
||||
POOL_free(pool);
|
||||
ZSTD_seekable_free(seekable);
|
||||
fclose(fin);
|
||||
free(jobs);
|
||||
}
|
||||
|
||||
|
||||
int main(int argc, const char** argv)
|
||||
{
|
||||
const char* const exeName = argv[0];
|
||||
|
||||
if (argc!=3) {
|
||||
fprintf(stderr, "wrong arguments\n");
|
||||
fprintf(stderr, "usage:\n");
|
||||
fprintf(stderr, "%s FILE NB_THREADS\n", exeName);
|
||||
return 1;
|
||||
}
|
||||
|
||||
{
|
||||
const char* const inFilename = argv[1];
|
||||
int const nbThreads = atoi(argv[2]);
|
||||
sumFile_orDie(inFilename, nbThreads);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
132
contrib/seekable_format/examples/seekable_compression.c
Normal file
132
contrib/seekable_format/examples/seekable_compression.c
Normal file
@ -0,0 +1,132 @@
|
||||
/*
|
||||
* Copyright (c) 2017-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
#include <stdlib.h> // malloc, free, exit, atoi
|
||||
#include <stdio.h> // fprintf, perror, feof, fopen, etc.
|
||||
#include <string.h> // strlen, memset, strcat
|
||||
#define ZSTD_STATIC_LINKING_ONLY
|
||||
#include <zstd.h> // presumes zstd library is installed
|
||||
|
||||
#include "zstd_seekable.h"
|
||||
|
||||
static void* malloc_orDie(size_t size)
|
||||
{
|
||||
void* const buff = malloc(size);
|
||||
if (buff) return buff;
|
||||
/* error */
|
||||
perror("malloc:");
|
||||
exit(1);
|
||||
}
|
||||
|
||||
static FILE* fopen_orDie(const char *filename, const char *instruction)
|
||||
{
|
||||
FILE* const inFile = fopen(filename, instruction);
|
||||
if (inFile) return inFile;
|
||||
/* error */
|
||||
perror(filename);
|
||||
exit(3);
|
||||
}
|
||||
|
||||
static size_t fread_orDie(void* buffer, size_t sizeToRead, FILE* file)
|
||||
{
|
||||
size_t const readSize = fread(buffer, 1, sizeToRead, file);
|
||||
if (readSize == sizeToRead) return readSize; /* good */
|
||||
if (feof(file)) return readSize; /* good, reached end of file */
|
||||
/* error */
|
||||
perror("fread");
|
||||
exit(4);
|
||||
}
|
||||
|
||||
static size_t fwrite_orDie(const void* buffer, size_t sizeToWrite, FILE* file)
|
||||
{
|
||||
size_t const writtenSize = fwrite(buffer, 1, sizeToWrite, file);
|
||||
if (writtenSize == sizeToWrite) return sizeToWrite; /* good */
|
||||
/* error */
|
||||
perror("fwrite");
|
||||
exit(5);
|
||||
}
|
||||
|
||||
static size_t fclose_orDie(FILE* file)
|
||||
{
|
||||
if (!fclose(file)) return 0;
|
||||
/* error */
|
||||
perror("fclose");
|
||||
exit(6);
|
||||
}
|
||||
|
||||
static void compressFile_orDie(const char* fname, const char* outName, int cLevel, unsigned frameSize)
|
||||
{
|
||||
FILE* const fin = fopen_orDie(fname, "rb");
|
||||
FILE* const fout = fopen_orDie(outName, "wb");
|
||||
size_t const buffInSize = ZSTD_CStreamInSize(); /* can always read one full block */
|
||||
void* const buffIn = malloc_orDie(buffInSize);
|
||||
size_t const buffOutSize = ZSTD_CStreamOutSize(); /* can always flush a full block */
|
||||
void* const buffOut = malloc_orDie(buffOutSize);
|
||||
|
||||
ZSTD_seekable_CStream* const cstream = ZSTD_seekable_createCStream();
|
||||
if (cstream==NULL) { fprintf(stderr, "ZSTD_seekable_createCStream() error \n"); exit(10); }
|
||||
size_t const initResult = ZSTD_seekable_initCStream(cstream, cLevel, 1, frameSize);
|
||||
if (ZSTD_isError(initResult)) { fprintf(stderr, "ZSTD_seekable_initCStream() error : %s \n", ZSTD_getErrorName(initResult)); exit(11); }
|
||||
|
||||
size_t read, toRead = buffInSize;
|
||||
while( (read = fread_orDie(buffIn, toRead, fin)) ) {
|
||||
ZSTD_inBuffer input = { buffIn, read, 0 };
|
||||
while (input.pos < input.size) {
|
||||
ZSTD_outBuffer output = { buffOut, buffOutSize, 0 };
|
||||
toRead = ZSTD_seekable_compressStream(cstream, &output , &input); /* toRead is guaranteed to be <= ZSTD_CStreamInSize() */
|
||||
if (ZSTD_isError(toRead)) { fprintf(stderr, "ZSTD_seekable_compressStream() error : %s \n", ZSTD_getErrorName(toRead)); exit(12); }
|
||||
if (toRead > buffInSize) toRead = buffInSize; /* Safely handle case when `buffInSize` is manually changed to a value < ZSTD_CStreamInSize()*/
|
||||
fwrite_orDie(buffOut, output.pos, fout);
|
||||
}
|
||||
}
|
||||
|
||||
while (1) {
|
||||
ZSTD_outBuffer output = { buffOut, buffOutSize, 0 };
|
||||
size_t const remainingToFlush = ZSTD_seekable_endStream(cstream, &output); /* close stream */
|
||||
if (ZSTD_isError(remainingToFlush)) { fprintf(stderr, "ZSTD_seekable_endStream() error : %s \n", ZSTD_getErrorName(remainingToFlush)); exit(13); }
|
||||
fwrite_orDie(buffOut, output.pos, fout);
|
||||
if (!remainingToFlush) break;
|
||||
}
|
||||
|
||||
ZSTD_seekable_freeCStream(cstream);
|
||||
fclose_orDie(fout);
|
||||
fclose_orDie(fin);
|
||||
free(buffIn);
|
||||
free(buffOut);
|
||||
}
|
||||
|
||||
static const char* createOutFilename_orDie(const char* filename)
|
||||
{
|
||||
size_t const inL = strlen(filename);
|
||||
size_t const outL = inL + 5;
|
||||
void* outSpace = malloc_orDie(outL);
|
||||
memset(outSpace, 0, outL);
|
||||
strcat(outSpace, filename);
|
||||
strcat(outSpace, ".zst");
|
||||
return (const char*)outSpace;
|
||||
}
|
||||
|
||||
int main(int argc, const char** argv) {
|
||||
const char* const exeName = argv[0];
|
||||
if (argc!=3) {
|
||||
printf("wrong arguments\n");
|
||||
printf("usage:\n");
|
||||
printf("%s FILE FRAME_SIZE\n", exeName);
|
||||
return 1;
|
||||
}
|
||||
|
||||
{ const char* const inFileName = argv[1];
|
||||
unsigned const frameSize = (unsigned)atoi(argv[2]);
|
||||
|
||||
const char* const outFileName = createOutFilename_orDie(inFileName);
|
||||
compressFile_orDie(inFileName, outFileName, 5, frameSize);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
138
contrib/seekable_format/examples/seekable_decompression.c
Normal file
138
contrib/seekable_format/examples/seekable_decompression.c
Normal file
@ -0,0 +1,138 @@
|
||||
/*
|
||||
* Copyright (c) 2017-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
|
||||
#include <stdlib.h> // malloc, exit
|
||||
#include <stdio.h> // fprintf, perror, feof
|
||||
#include <string.h> // strerror
|
||||
#include <errno.h> // errno
|
||||
#define ZSTD_STATIC_LINKING_ONLY
|
||||
#include <zstd.h> // presumes zstd library is installed
|
||||
#include <zstd_errors.h>
|
||||
|
||||
#include "zstd_seekable.h"
|
||||
|
||||
#define MIN(a, b) ((a) < (b) ? (a) : (b))
|
||||
|
||||
static void* malloc_orDie(size_t size)
|
||||
{
|
||||
void* const buff = malloc(size);
|
||||
if (buff) return buff;
|
||||
/* error */
|
||||
perror("malloc");
|
||||
exit(1);
|
||||
}
|
||||
|
||||
static void* realloc_orDie(void* ptr, size_t size)
|
||||
{
|
||||
ptr = realloc(ptr, size);
|
||||
if (ptr) return ptr;
|
||||
/* error */
|
||||
perror("realloc");
|
||||
exit(1);
|
||||
}
|
||||
|
||||
static FILE* fopen_orDie(const char *filename, const char *instruction)
|
||||
{
|
||||
FILE* const inFile = fopen(filename, instruction);
|
||||
if (inFile) return inFile;
|
||||
/* error */
|
||||
perror(filename);
|
||||
exit(3);
|
||||
}
|
||||
|
||||
static size_t fread_orDie(void* buffer, size_t sizeToRead, FILE* file)
|
||||
{
|
||||
size_t const readSize = fread(buffer, 1, sizeToRead, file);
|
||||
if (readSize == sizeToRead) return readSize; /* good */
|
||||
if (feof(file)) return readSize; /* good, reached end of file */
|
||||
/* error */
|
||||
perror("fread");
|
||||
exit(4);
|
||||
}
|
||||
|
||||
static size_t fwrite_orDie(const void* buffer, size_t sizeToWrite, FILE* file)
|
||||
{
|
||||
size_t const writtenSize = fwrite(buffer, 1, sizeToWrite, file);
|
||||
if (writtenSize == sizeToWrite) return sizeToWrite; /* good */
|
||||
/* error */
|
||||
perror("fwrite");
|
||||
exit(5);
|
||||
}
|
||||
|
||||
static size_t fclose_orDie(FILE* file)
|
||||
{
|
||||
if (!fclose(file)) return 0;
|
||||
/* error */
|
||||
perror("fclose");
|
||||
exit(6);
|
||||
}
|
||||
|
||||
static void fseek_orDie(FILE* file, long int offset, int origin) {
|
||||
if (!fseek(file, offset, origin)) {
|
||||
if (!fflush(file)) return;
|
||||
}
|
||||
/* error */
|
||||
perror("fseek");
|
||||
exit(7);
|
||||
}
|
||||
|
||||
|
||||
static void decompressFile_orDie(const char* fname, unsigned startOffset, unsigned endOffset)
|
||||
{
|
||||
FILE* const fin = fopen_orDie(fname, "rb");
|
||||
FILE* const fout = stdout;
|
||||
size_t const buffOutSize = ZSTD_DStreamOutSize(); /* Guarantee to successfully flush at least one complete compressed block in all circumstances. */
|
||||
void* const buffOut = malloc_orDie(buffOutSize);
|
||||
|
||||
ZSTD_seekable* const seekable = ZSTD_seekable_create();
|
||||
if (seekable==NULL) { fprintf(stderr, "ZSTD_seekable_create() error \n"); exit(10); }
|
||||
|
||||
size_t const initResult = ZSTD_seekable_initFile(seekable, fin);
|
||||
if (ZSTD_isError(initResult)) { fprintf(stderr, "ZSTD_seekable_init() error : %s \n", ZSTD_getErrorName(initResult)); exit(11); }
|
||||
|
||||
while (startOffset < endOffset) {
|
||||
size_t const result = ZSTD_seekable_decompress(seekable, buffOut, MIN(endOffset - startOffset, buffOutSize), startOffset);
|
||||
|
||||
if (ZSTD_isError(result)) {
|
||||
fprintf(stderr, "ZSTD_seekable_decompress() error : %s \n",
|
||||
ZSTD_getErrorName(result));
|
||||
exit(12);
|
||||
}
|
||||
fwrite_orDie(buffOut, result, fout);
|
||||
startOffset += result;
|
||||
}
|
||||
|
||||
ZSTD_seekable_free(seekable);
|
||||
fclose_orDie(fin);
|
||||
fclose_orDie(fout);
|
||||
free(buffOut);
|
||||
}
|
||||
|
||||
|
||||
int main(int argc, const char** argv)
|
||||
{
|
||||
const char* const exeName = argv[0];
|
||||
|
||||
if (argc!=4) {
|
||||
fprintf(stderr, "wrong arguments\n");
|
||||
fprintf(stderr, "usage:\n");
|
||||
fprintf(stderr, "%s FILE START END\n", exeName);
|
||||
return 1;
|
||||
}
|
||||
|
||||
{
|
||||
const char* const inFilename = argv[1];
|
||||
unsigned const startOffset = (unsigned) atoi(argv[2]);
|
||||
unsigned const endOffset = (unsigned) atoi(argv[3]);
|
||||
decompressFile_orDie(inFilename, startOffset, endOffset);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
184
contrib/seekable_format/zstd_seekable.h
Normal file
184
contrib/seekable_format/zstd_seekable.h
Normal file
@ -0,0 +1,184 @@
|
||||
#ifndef SEEKABLE_H
|
||||
#define SEEKABLE_H
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdio.h>
|
||||
|
||||
static const unsigned ZSTD_seekTableFooterSize = 9;
|
||||
|
||||
#define ZSTD_SEEKABLE_MAGICNUMBER 0x8F92EAB1
|
||||
|
||||
#define ZSTD_SEEKABLE_MAXFRAMES 0x8000000U
|
||||
|
||||
/* Limit the maximum size to avoid any potential issues storing the compressed size */
|
||||
#define ZSTD_SEEKABLE_MAX_FRAME_DECOMPRESSED_SIZE 0x80000000U
|
||||
|
||||
/*-****************************************************************************
|
||||
* Seekable Format
|
||||
*
|
||||
* The seekable format splits the compressed data into a series of "frames",
|
||||
* each compressed individually so that decompression of a section in the
|
||||
* middle of an archive only requires zstd to decompress at most a frame's
|
||||
* worth of extra data, instead of the entire archive.
|
||||
******************************************************************************/
|
||||
|
||||
typedef struct ZSTD_seekable_CStream_s ZSTD_seekable_CStream;
|
||||
typedef struct ZSTD_seekable_s ZSTD_seekable;
|
||||
|
||||
/*-****************************************************************************
|
||||
* Seekable compression - HowTo
|
||||
* A ZSTD_seekable_CStream object is required to tracking streaming operation.
|
||||
* Use ZSTD_seekable_createCStream() and ZSTD_seekable_freeCStream() to create/
|
||||
* release resources.
|
||||
*
|
||||
* Streaming objects are reusable to avoid allocation and deallocation,
|
||||
* to start a new compression operation call ZSTD_seekable_initCStream() on the
|
||||
* compressor.
|
||||
*
|
||||
* Data streamed to the seekable compressor will automatically be split into
|
||||
* frames of size `maxFrameSize` (provided in ZSTD_seekable_initCStream()),
|
||||
* or if none is provided, will be cut off whenever ZSTD_seekable_endFrame() is
|
||||
* called or when the default maximum frame size (2GB) is reached.
|
||||
*
|
||||
* Use ZSTD_seekable_initCStream() to initialize a ZSTD_seekable_CStream object
|
||||
* for a new compression operation.
|
||||
* `maxFrameSize` indicates the size at which to automatically start a new
|
||||
* seekable frame. `maxFrameSize == 0` implies the default maximum size.
|
||||
* `checksumFlag` indicates whether or not the seek table should include frame
|
||||
* checksums on the uncompressed data for verification.
|
||||
* @return : a size hint for input to provide for compression, or an error code
|
||||
* checkable with ZSTD_isError()
|
||||
*
|
||||
* Use ZSTD_seekable_compressStream() repetitively to consume input stream.
|
||||
* The function will automatically update both `pos` fields.
|
||||
* Note that it may not consume the entire input, in which case `pos < size`,
|
||||
* and it's up to the caller to present again remaining data.
|
||||
* @return : a size hint, preferred nb of bytes to use as input for next
|
||||
* function call or an error code, which can be tested using
|
||||
* ZSTD_isError().
|
||||
* Note 1 : it's just a hint, to help latency a little, any other
|
||||
* value will work fine.
|
||||
*
|
||||
* At any time, call ZSTD_seekable_endFrame() to end the current frame and
|
||||
* start a new one.
|
||||
*
|
||||
* ZSTD_seekable_endStream() will end the current frame, and then write the seek
|
||||
* table so that decompressors can efficiently find compressed frames.
|
||||
* ZSTD_seekable_endStream() may return a number > 0 if it was unable to flush
|
||||
* all the necessary data to `output`. In this case, it should be called again
|
||||
* until all remaining data is flushed out and 0 is returned.
|
||||
******************************************************************************/
|
||||
|
||||
/*===== Seekable compressor management =====*/
|
||||
ZSTDLIB_API ZSTD_seekable_CStream* ZSTD_seekable_createCStream(void);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_freeCStream(ZSTD_seekable_CStream* zcs);
|
||||
|
||||
/*===== Seekable compression functions =====*/
|
||||
ZSTDLIB_API size_t ZSTD_seekable_initCStream(ZSTD_seekable_CStream* zcs, int compressionLevel, int checksumFlag, unsigned maxFrameSize);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_compressStream(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_endFrame(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_endStream(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output);
|
||||
|
||||
/*= Raw seek table API
|
||||
* These functions allow for the seek table to be constructed directly.
|
||||
* This table can then be appended to a file of concatenated frames.
|
||||
* This allows the frames to be compressed independently, even in parallel,
|
||||
* and compiled together afterward into a seekable archive.
|
||||
*
|
||||
* Use ZSTD_seekable_createFrameLog() to allocate and initialize a tracking
|
||||
* structure.
|
||||
*
|
||||
* Call ZSTD_seekable_logFrame() once for each frame in the archive.
|
||||
* checksum is optional, and will not be used if checksumFlag was 0 when the
|
||||
* frame log was created. If present, it should be the least significant 32
|
||||
* bits of the XXH64 hash of the uncompressed data.
|
||||
*
|
||||
* Call ZSTD_seekable_writeSeekTable to serialize the data into a seek table.
|
||||
* If the entire table was written, the return value will be 0. Otherwise,
|
||||
* it will be equal to the number of bytes left to write. */
|
||||
typedef struct ZSTD_frameLog_s ZSTD_frameLog;
|
||||
ZSTDLIB_API ZSTD_frameLog* ZSTD_seekable_createFrameLog(int checksumFlag);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_freeFrameLog(ZSTD_frameLog* fl);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_logFrame(ZSTD_frameLog* fl, unsigned compressedSize, unsigned decompressedSize, unsigned checksum);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_writeSeekTable(ZSTD_frameLog* fl, ZSTD_outBuffer* output);
|
||||
|
||||
/*-****************************************************************************
|
||||
* Seekable decompression - HowTo
|
||||
* A ZSTD_seekable object is required to tracking the seekTable.
|
||||
*
|
||||
* Call ZSTD_seekable_init* to initialize a ZSTD_seekable object with the
|
||||
* the seek table provided in the input.
|
||||
* There are three modes for ZSTD_seekable_init:
|
||||
* - ZSTD_seekable_initBuff() : An in-memory API. The data contained in
|
||||
* `src` should be the entire seekable file, including the seek table.
|
||||
* `src` should be kept alive and unmodified until the ZSTD_seekable object
|
||||
* is freed or reset.
|
||||
* - ZSTD_seekable_initFile() : A simplified file API using stdio. fread and
|
||||
* fseek will be used to access the required data for building the seek
|
||||
* table and doing decompression operations. `src` should not be closed
|
||||
* or modified until the ZSTD_seekable object is freed or reset.
|
||||
* - ZSTD_seekable_initAdvanced() : A general API allowing the client to
|
||||
* provide its own read and seek callbacks.
|
||||
* + ZSTD_seekable_read() : read exactly `n` bytes into `buffer`.
|
||||
* Premature EOF should be treated as an error.
|
||||
* + ZSTD_seekable_seek() : seek the read head to `offset` from `origin`,
|
||||
* where origin is either SEEK_SET (beginning of
|
||||
* file), or SEEK_END (end of file).
|
||||
* Both functions should return a non-negative value in case of success, and a
|
||||
* negative value in case of failure. If implementing using this API and
|
||||
* stdio, be careful with files larger than 4GB and fseek. All of these
|
||||
* functions return an error code checkable with ZSTD_isError().
|
||||
*
|
||||
* Call ZSTD_seekable_decompress to decompress `dstSize` bytes at decompressed
|
||||
* offset `offset`. ZSTD_seekable_decompress may have to decompress the entire
|
||||
* prefix of the frame before the desired data if it has not already processed
|
||||
* this section. If ZSTD_seekable_decompress is called multiple times for a
|
||||
* consecutive range of data, it will efficiently retain the decompressor object
|
||||
* and avoid redecompressing frame prefixes. The return value is the number of
|
||||
* bytes decompressed, or an error code checkable with ZSTD_isError().
|
||||
*
|
||||
* The seek table access functions can be used to obtain the data contained
|
||||
* in the seek table. If frameIndex is larger than the value returned by
|
||||
* ZSTD_seekable_getNumFrames(), they will return error codes checkable with
|
||||
* ZSTD_isError(). Note that since the offset access functions return
|
||||
* unsigned long long instead of size_t, in this case they will instead return
|
||||
* the value ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE.
|
||||
******************************************************************************/
|
||||
|
||||
/*===== Seekable decompressor management =====*/
|
||||
ZSTDLIB_API ZSTD_seekable* ZSTD_seekable_create(void);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_free(ZSTD_seekable* zs);
|
||||
|
||||
/*===== Seekable decompression functions =====*/
|
||||
ZSTDLIB_API size_t ZSTD_seekable_initBuff(ZSTD_seekable* zs, const void* src, size_t srcSize);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_initFile(ZSTD_seekable* zs, FILE* src);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_decompress(ZSTD_seekable* zs, void* dst, size_t dstSize, unsigned long long offset);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_decompressFrame(ZSTD_seekable* zs, void* dst, size_t dstSize, unsigned frameIndex);
|
||||
|
||||
#define ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE (0ULL-2)
|
||||
/*===== Seek Table access functions =====*/
|
||||
ZSTDLIB_API unsigned ZSTD_seekable_getNumFrames(ZSTD_seekable* const zs);
|
||||
ZSTDLIB_API unsigned long long ZSTD_seekable_getFrameCompressedOffset(ZSTD_seekable* const zs, unsigned frameIndex);
|
||||
ZSTDLIB_API unsigned long long ZSTD_seekable_getFrameDecompressedOffset(ZSTD_seekable* const zs, unsigned frameIndex);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_getFrameCompressedSize(ZSTD_seekable* const zs, unsigned frameIndex);
|
||||
ZSTDLIB_API size_t ZSTD_seekable_getFrameDecompressedSize(ZSTD_seekable* const zs, unsigned frameIndex);
|
||||
ZSTDLIB_API unsigned ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* const zs, unsigned long long offset);
|
||||
|
||||
/*===== Seekable advanced I/O API =====*/
|
||||
typedef int(ZSTD_seekable_read)(void* opaque, void* buffer, size_t n);
|
||||
typedef int(ZSTD_seekable_seek)(void* opaque, long long offset, int origin);
|
||||
typedef struct {
|
||||
void* opaque;
|
||||
ZSTD_seekable_read* read;
|
||||
ZSTD_seekable_seek* seek;
|
||||
} ZSTD_seekable_customFile;
|
||||
ZSTDLIB_API size_t ZSTD_seekable_initAdvanced(ZSTD_seekable* zs, ZSTD_seekable_customFile src);
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif
|
116
contrib/seekable_format/zstd_seekable_compression_format.md
Normal file
116
contrib/seekable_format/zstd_seekable_compression_format.md
Normal file
@ -0,0 +1,116 @@
|
||||
# Zstandard Seekable Format
|
||||
|
||||
### Notices
|
||||
|
||||
Copyright (c) 2017-present Facebook, Inc.
|
||||
|
||||
Permission is granted to copy and distribute this document
|
||||
for any purpose and without charge,
|
||||
including translations into other languages
|
||||
and incorporation into compilations,
|
||||
provided that the copyright notice and this notice are preserved,
|
||||
and that any substantive changes or deletions from the original
|
||||
are clearly marked.
|
||||
Distribution of this document is unlimited.
|
||||
|
||||
### Version
|
||||
0.1.0 (11/04/17)
|
||||
|
||||
## Introduction
|
||||
This document defines a format for compressed data to be stored so that subranges of the data can be efficiently decompressed without requiring the entire document to be decompressed.
|
||||
This is done by splitting up the input data into frames,
|
||||
each of which are compressed independently,
|
||||
and so can be decompressed independently.
|
||||
Decompression then takes advantage of a provided 'seek table', which allows the decompressor to immediately jump to the desired data. This is done in a way that is compatible with the original Zstandard format by placing the seek table in a Zstandard skippable frame.
|
||||
|
||||
### Overall conventions
|
||||
In this document:
|
||||
- square brackets i.e. `[` and `]` are used to indicate optional fields or parameters.
|
||||
- the naming convention for identifiers is `Mixed_Case_With_Underscores`
|
||||
- All numeric fields are little-endian unless specified otherwise
|
||||
|
||||
## Format
|
||||
|
||||
The format consists of a number of frames (Zstandard compressed frames and skippable frames), followed by a final skippable frame at the end containing the seek table.
|
||||
|
||||
### Seek Table Format
|
||||
The structure of the seek table frame is as follows:
|
||||
|
||||
|`Skippable_Magic_Number`|`Frame_Size`|`[Seek_Table_Entries]`|`Seek_Table_Footer`|
|
||||
|------------------------|------------|----------------------|-------------------|
|
||||
| 4 bytes | 4 bytes | 8-12 bytes each | 9 bytes |
|
||||
|
||||
__`Skippable_Magic_Number`__
|
||||
|
||||
Value : 0x184D2A5E.
|
||||
This is for compatibility with [Zstandard skippable frames].
|
||||
Since it is legal for other Zstandard skippable frames to use the same
|
||||
magic number, it is not recommended for a decoder to recognize frames
|
||||
solely on this.
|
||||
|
||||
__`Frame_Size`__
|
||||
|
||||
The total size of the skippable frame, not including the `Skippable_Magic_Number` or `Frame_Size`.
|
||||
This is for compatibility with [Zstandard skippable frames].
|
||||
|
||||
[Zstandard skippable frames]: https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#skippable-frames
|
||||
|
||||
#### `Seek_Table_Footer`
|
||||
The seek table footer format is as follows:
|
||||
|
||||
|`Number_Of_Frames`|`Seek_Table_Descriptor`|`Seekable_Magic_Number`|
|
||||
|------------------|-----------------------|-----------------------|
|
||||
| 4 bytes | 1 byte | 4 bytes |
|
||||
|
||||
__`Seekable_Magic_Number`__
|
||||
|
||||
Value : 0x8F92EAB1.
|
||||
This value must be the last bytes present in the compressed file so that decoders
|
||||
can efficiently find it and determine if there is an actual seek table present.
|
||||
|
||||
__`Number_Of_Frames`__
|
||||
|
||||
The number of stored frames in the data.
|
||||
|
||||
__`Seek_Table_Descriptor`__
|
||||
|
||||
A bitfield describing the format of the seek table.
|
||||
|
||||
| Bit number | Field name |
|
||||
| ---------- | ---------- |
|
||||
| 7 | `Checksum_Flag` |
|
||||
| 6-2 | `Reserved_Bits` |
|
||||
| 1-0 | `Unused_Bits` |
|
||||
|
||||
While only `Checksum_Flag` currently exists, there are 7 other bits in this field that can be used for future changes to the format,
|
||||
for example the addition of inline dictionaries.
|
||||
|
||||
__`Checksum_Flag`__
|
||||
|
||||
If the checksum flag is set, each of the seek table entries contains a 4 byte checksum of the uncompressed data contained in its frame.
|
||||
|
||||
`Reserved_Bits` are not currently used but may be used in the future for breaking changes, so a compliant decoder should ensure they are set to 0. `Unused_Bits` may be used in the future for non-breaking changes, so a compliant decoder should not interpret these bits.
|
||||
|
||||
#### __`Seek_Table_Entries`__
|
||||
|
||||
`Seek_Table_Entries` consists of `Number_Of_Frames` (one for each frame in the data, not including the seek table frame) entries of the following form, in sequence:
|
||||
|
||||
|`Compressed_Size`|`Decompressed_Size`|`[Checksum]`|
|
||||
|-----------------|-------------------|------------|
|
||||
| 4 bytes | 4 bytes | 4 bytes |
|
||||
|
||||
__`Compressed_Size`__
|
||||
|
||||
The compressed size of the frame.
|
||||
The cumulative sum of the `Compressed_Size` fields of frames `0` to `i` gives the offset in the compressed file of frame `i+1`.
|
||||
|
||||
__`Decompressed_Size`__
|
||||
|
||||
The size of the decompressed data contained in the frame. For skippable or otherwise empty frames, this value is 0.
|
||||
|
||||
__`Checksum`__
|
||||
|
||||
Only present if `Checksum_Flag` is set in the `Seek_Table_Descriptor`. Value : the least significant 32 bits of the XXH64 digest of the uncompressed data, stored in little-endian format.
|
||||
|
||||
## Version Changes
|
||||
- 0.1.0: initial version
|
366
contrib/seekable_format/zstdseek_compress.c
Normal file
366
contrib/seekable_format/zstdseek_compress.c
Normal file
@ -0,0 +1,366 @@
|
||||
/*
|
||||
* Copyright (c) 2017-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
#include <stdlib.h> /* malloc, free */
|
||||
|
||||
#define XXH_STATIC_LINKING_ONLY
|
||||
#define XXH_NAMESPACE ZSTD_
|
||||
#include "xxhash.h"
|
||||
|
||||
#define ZSTD_STATIC_LINKING_ONLY
|
||||
#include "zstd.h"
|
||||
#include "zstd_errors.h"
|
||||
#include "mem.h"
|
||||
#include "zstd_seekable.h"
|
||||
|
||||
#define CHECK_Z(f) { size_t const ret = (f); if (ret != 0) return ret; }
|
||||
|
||||
#undef ERROR
|
||||
#define ERROR(name) ((size_t)-ZSTD_error_##name)
|
||||
|
||||
#undef MIN
|
||||
#undef MAX
|
||||
#define MIN(a, b) ((a) < (b) ? (a) : (b))
|
||||
#define MAX(a, b) ((a) > (b) ? (a) : (b))
|
||||
|
||||
typedef struct {
|
||||
U32 cSize;
|
||||
U32 dSize;
|
||||
U32 checksum;
|
||||
} framelogEntry_t;
|
||||
|
||||
struct ZSTD_frameLog_s {
|
||||
framelogEntry_t* entries;
|
||||
U32 size;
|
||||
U32 capacity;
|
||||
|
||||
int checksumFlag;
|
||||
|
||||
/* for use when streaming out the seek table */
|
||||
U32 seekTablePos;
|
||||
U32 seekTableIndex;
|
||||
} framelog_t;
|
||||
|
||||
struct ZSTD_seekable_CStream_s {
|
||||
ZSTD_CStream* cstream;
|
||||
ZSTD_frameLog framelog;
|
||||
|
||||
U32 frameCSize;
|
||||
U32 frameDSize;
|
||||
|
||||
XXH64_state_t xxhState;
|
||||
|
||||
U32 maxFrameSize;
|
||||
|
||||
int writingSeekTable;
|
||||
};
|
||||
|
||||
size_t ZSTD_seekable_frameLog_allocVec(ZSTD_frameLog* fl)
|
||||
{
|
||||
/* allocate some initial space */
|
||||
size_t const FRAMELOG_STARTING_CAPACITY = 16;
|
||||
fl->entries = (framelogEntry_t*)malloc(
|
||||
sizeof(framelogEntry_t) * FRAMELOG_STARTING_CAPACITY);
|
||||
if (fl->entries == NULL) return ERROR(memory_allocation);
|
||||
fl->capacity = FRAMELOG_STARTING_CAPACITY;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_frameLog_freeVec(ZSTD_frameLog* fl)
|
||||
{
|
||||
if (fl != NULL) free(fl->entries);
|
||||
return 0;
|
||||
}
|
||||
|
||||
ZSTD_frameLog* ZSTD_seekable_createFrameLog(int checksumFlag)
|
||||
{
|
||||
ZSTD_frameLog* fl = malloc(sizeof(ZSTD_frameLog));
|
||||
if (fl == NULL) return NULL;
|
||||
|
||||
if (ZSTD_isError(ZSTD_seekable_frameLog_allocVec(fl))) {
|
||||
free(fl);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
fl->checksumFlag = checksumFlag;
|
||||
fl->seekTablePos = 0;
|
||||
fl->seekTableIndex = 0;
|
||||
fl->size = 0;
|
||||
|
||||
return fl;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_freeFrameLog(ZSTD_frameLog* fl)
|
||||
{
|
||||
ZSTD_seekable_frameLog_freeVec(fl);
|
||||
free(fl);
|
||||
return 0;
|
||||
}
|
||||
|
||||
ZSTD_seekable_CStream* ZSTD_seekable_createCStream()
|
||||
{
|
||||
ZSTD_seekable_CStream* zcs = malloc(sizeof(ZSTD_seekable_CStream));
|
||||
|
||||
if (zcs == NULL) return NULL;
|
||||
|
||||
memset(zcs, 0, sizeof(*zcs));
|
||||
|
||||
zcs->cstream = ZSTD_createCStream();
|
||||
if (zcs->cstream == NULL) goto failed1;
|
||||
|
||||
if (ZSTD_isError(ZSTD_seekable_frameLog_allocVec(&zcs->framelog))) goto failed2;
|
||||
|
||||
return zcs;
|
||||
|
||||
failed2:
|
||||
ZSTD_freeCStream(zcs->cstream);
|
||||
failed1:
|
||||
free(zcs);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_freeCStream(ZSTD_seekable_CStream* zcs)
|
||||
{
|
||||
if (zcs == NULL) return 0; /* support free on null */
|
||||
ZSTD_freeCStream(zcs->cstream);
|
||||
ZSTD_seekable_frameLog_freeVec(&zcs->framelog);
|
||||
free(zcs);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_initCStream(ZSTD_seekable_CStream* zcs,
|
||||
int compressionLevel,
|
||||
int checksumFlag,
|
||||
U32 maxFrameSize)
|
||||
{
|
||||
zcs->framelog.size = 0;
|
||||
zcs->frameCSize = 0;
|
||||
zcs->frameDSize = 0;
|
||||
|
||||
/* make sure maxFrameSize has a reasonable value */
|
||||
if (maxFrameSize > ZSTD_SEEKABLE_MAX_FRAME_DECOMPRESSED_SIZE) {
|
||||
return ERROR(compressionParameter_unsupported);
|
||||
}
|
||||
|
||||
zcs->maxFrameSize = maxFrameSize
|
||||
? maxFrameSize
|
||||
: ZSTD_SEEKABLE_MAX_FRAME_DECOMPRESSED_SIZE;
|
||||
|
||||
zcs->framelog.checksumFlag = checksumFlag;
|
||||
if (zcs->framelog.checksumFlag) {
|
||||
XXH64_reset(&zcs->xxhState, 0);
|
||||
}
|
||||
|
||||
zcs->framelog.seekTablePos = 0;
|
||||
zcs->framelog.seekTableIndex = 0;
|
||||
zcs->writingSeekTable = 0;
|
||||
|
||||
return ZSTD_initCStream(zcs->cstream, compressionLevel);
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_logFrame(ZSTD_frameLog* fl,
|
||||
unsigned compressedSize,
|
||||
unsigned decompressedSize,
|
||||
unsigned checksum)
|
||||
{
|
||||
if (fl->size == ZSTD_SEEKABLE_MAXFRAMES)
|
||||
return ERROR(frameIndex_tooLarge);
|
||||
|
||||
/* grow the buffer if required */
|
||||
if (fl->size == fl->capacity) {
|
||||
/* exponential size increase for constant amortized runtime */
|
||||
size_t const newCapacity = fl->capacity * 2;
|
||||
framelogEntry_t* const newEntries = realloc(fl->entries,
|
||||
sizeof(framelogEntry_t) * newCapacity);
|
||||
|
||||
if (newEntries == NULL) return ERROR(memory_allocation);
|
||||
|
||||
fl->entries = newEntries;
|
||||
fl->capacity = newCapacity;
|
||||
}
|
||||
|
||||
fl->entries[fl->size] = (framelogEntry_t){
|
||||
compressedSize, decompressedSize, checksum
|
||||
};
|
||||
fl->size++;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_endFrame(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output)
|
||||
{
|
||||
size_t const prevOutPos = output->pos;
|
||||
/* end the frame */
|
||||
size_t ret = ZSTD_endStream(zcs->cstream, output);
|
||||
|
||||
zcs->frameCSize += output->pos - prevOutPos;
|
||||
|
||||
/* need to flush before doing the rest */
|
||||
if (ret) return ret;
|
||||
|
||||
/* frame done */
|
||||
|
||||
/* store the frame data for later */
|
||||
ret = ZSTD_seekable_logFrame(
|
||||
&zcs->framelog, zcs->frameCSize, zcs->frameDSize,
|
||||
zcs->framelog.checksumFlag
|
||||
? XXH64_digest(&zcs->xxhState) & 0xFFFFFFFFU
|
||||
: 0);
|
||||
if (ret) return ret;
|
||||
|
||||
/* reset for the next frame */
|
||||
zcs->frameCSize = 0;
|
||||
zcs->frameDSize = 0;
|
||||
|
||||
ZSTD_resetCStream(zcs->cstream, 0);
|
||||
if (zcs->framelog.checksumFlag)
|
||||
XXH64_reset(&zcs->xxhState, 0);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_compressStream(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input)
|
||||
{
|
||||
const BYTE* const inBase = (const BYTE*) input->src + input->pos;
|
||||
size_t inLen = input->size - input->pos;
|
||||
|
||||
inLen = MIN(inLen, (size_t)(zcs->maxFrameSize - zcs->frameDSize));
|
||||
|
||||
/* if we haven't finished flushing the last frame, don't start writing a new one */
|
||||
if (inLen > 0) {
|
||||
ZSTD_inBuffer inTmp = { inBase, inLen, 0 };
|
||||
size_t const prevOutPos = output->pos;
|
||||
|
||||
size_t const ret = ZSTD_compressStream(zcs->cstream, output, &inTmp);
|
||||
|
||||
if (zcs->framelog.checksumFlag) {
|
||||
XXH64_update(&zcs->xxhState, inBase, inTmp.pos);
|
||||
}
|
||||
|
||||
zcs->frameCSize += output->pos - prevOutPos;
|
||||
zcs->frameDSize += inTmp.pos;
|
||||
|
||||
input->pos += inTmp.pos;
|
||||
|
||||
if (ZSTD_isError(ret)) return ret;
|
||||
}
|
||||
|
||||
if (zcs->maxFrameSize == zcs->frameDSize) {
|
||||
/* log the frame and start over */
|
||||
size_t const ret = ZSTD_seekable_endFrame(zcs, output);
|
||||
if (ZSTD_isError(ret)) return ret;
|
||||
|
||||
/* get the client ready for the next frame */
|
||||
return (size_t)zcs->maxFrameSize;
|
||||
}
|
||||
|
||||
return (size_t)(zcs->maxFrameSize - zcs->frameDSize);
|
||||
}
|
||||
|
||||
static inline size_t ZSTD_seekable_seekTableSize(const ZSTD_frameLog* fl)
|
||||
{
|
||||
size_t const sizePerFrame = 8 + (fl->checksumFlag?4:0);
|
||||
size_t const seekTableLen = ZSTD_skippableHeaderSize +
|
||||
sizePerFrame * fl->size +
|
||||
ZSTD_seekTableFooterSize;
|
||||
|
||||
return seekTableLen;
|
||||
}
|
||||
|
||||
static inline size_t ZSTD_stwrite32(ZSTD_frameLog* fl,
|
||||
ZSTD_outBuffer* output, U32 const value,
|
||||
U32 const offset)
|
||||
{
|
||||
if (fl->seekTablePos < offset + 4) {
|
||||
BYTE tmp[4]; /* so that we can work with buffers too small to write a whole word to */
|
||||
size_t const lenWrite =
|
||||
MIN(output->size - output->pos, offset + 4 - fl->seekTablePos);
|
||||
MEM_writeLE32(tmp, value);
|
||||
memcpy((BYTE*)output->dst + output->pos,
|
||||
tmp + (fl->seekTablePos - offset), lenWrite);
|
||||
output->pos += lenWrite;
|
||||
fl->seekTablePos += lenWrite;
|
||||
|
||||
if (lenWrite < 4) return ZSTD_seekable_seekTableSize(fl) - fl->seekTablePos;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_writeSeekTable(ZSTD_frameLog* fl, ZSTD_outBuffer* output)
|
||||
{
|
||||
/* seekTableIndex: the current index in the table and
|
||||
* seekTableSize: the amount of the table written so far
|
||||
*
|
||||
* This function is written this way so that if it has to return early
|
||||
* because of a small buffer, it can keep going where it left off.
|
||||
*/
|
||||
|
||||
size_t const sizePerFrame = 8 + (fl->checksumFlag?4:0);
|
||||
size_t const seekTableLen = ZSTD_seekable_seekTableSize(fl);
|
||||
|
||||
CHECK_Z(ZSTD_stwrite32(fl, output, ZSTD_MAGIC_SKIPPABLE_START | 0xE, 0));
|
||||
CHECK_Z(ZSTD_stwrite32(fl, output, seekTableLen - ZSTD_skippableHeaderSize,
|
||||
4));
|
||||
|
||||
while (fl->seekTableIndex < fl->size) {
|
||||
CHECK_Z(ZSTD_stwrite32(fl, output,
|
||||
fl->entries[fl->seekTableIndex].cSize,
|
||||
ZSTD_skippableHeaderSize +
|
||||
sizePerFrame * fl->seekTableIndex + 0));
|
||||
|
||||
CHECK_Z(ZSTD_stwrite32(fl, output,
|
||||
fl->entries[fl->seekTableIndex].dSize,
|
||||
ZSTD_skippableHeaderSize +
|
||||
sizePerFrame * fl->seekTableIndex + 4));
|
||||
|
||||
if (fl->checksumFlag) {
|
||||
CHECK_Z(ZSTD_stwrite32(
|
||||
fl, output, fl->entries[fl->seekTableIndex].checksum,
|
||||
ZSTD_skippableHeaderSize +
|
||||
sizePerFrame * fl->seekTableIndex + 8));
|
||||
}
|
||||
|
||||
fl->seekTableIndex++;
|
||||
}
|
||||
|
||||
CHECK_Z(ZSTD_stwrite32(fl, output, fl->size,
|
||||
seekTableLen - ZSTD_seekTableFooterSize));
|
||||
|
||||
if (output->size - output->pos < 1) return seekTableLen - fl->seekTablePos;
|
||||
if (fl->seekTablePos < seekTableLen - 4) {
|
||||
BYTE sfd = 0;
|
||||
sfd |= (fl->checksumFlag) << 7;
|
||||
|
||||
((BYTE*)output->dst)[output->pos] = sfd;
|
||||
output->pos++;
|
||||
fl->seekTablePos++;
|
||||
}
|
||||
|
||||
CHECK_Z(ZSTD_stwrite32(fl, output, ZSTD_SEEKABLE_MAGICNUMBER,
|
||||
seekTableLen - 4));
|
||||
|
||||
if (fl->seekTablePos != seekTableLen) return ERROR(GENERIC);
|
||||
return 0;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_endStream(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output)
|
||||
{
|
||||
if (!zcs->writingSeekTable && zcs->frameDSize) {
|
||||
const size_t endFrame = ZSTD_seekable_endFrame(zcs, output);
|
||||
if (ZSTD_isError(endFrame)) return endFrame;
|
||||
/* return an accurate size hint */
|
||||
if (endFrame) return endFrame + ZSTD_seekable_seekTableSize(&zcs->framelog);
|
||||
}
|
||||
|
||||
zcs->writingSeekTable = 1;
|
||||
|
||||
return ZSTD_seekable_writeSeekTable(&zcs->framelog, output);
|
||||
}
|
462
contrib/seekable_format/zstdseek_decompress.c
Normal file
462
contrib/seekable_format/zstdseek_decompress.c
Normal file
@ -0,0 +1,462 @@
|
||||
/*
|
||||
* Copyright (c) 2017-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
/* *********************************************************
|
||||
* Turn on Large Files support (>4GB) for 32-bit Linux/Unix
|
||||
***********************************************************/
|
||||
#if !defined(__64BIT__) || defined(__MINGW32__) /* No point defining Large file for 64 bit but MinGW-w64 requires it */
|
||||
# if !defined(_FILE_OFFSET_BITS)
|
||||
# define _FILE_OFFSET_BITS 64 /* turn off_t into a 64-bit type for ftello, fseeko */
|
||||
# endif
|
||||
# if !defined(_LARGEFILE_SOURCE) /* obsolete macro, replaced with _FILE_OFFSET_BITS */
|
||||
# define _LARGEFILE_SOURCE 1 /* Large File Support extension (LFS) - fseeko, ftello */
|
||||
# endif
|
||||
# if defined(_AIX) || defined(__hpux)
|
||||
# define _LARGE_FILES /* Large file support on 32-bits AIX and HP-UX */
|
||||
# endif
|
||||
#endif
|
||||
|
||||
/* ************************************************************
|
||||
* Avoid fseek()'s 2GiB barrier with MSVC, MacOS, *BSD, MinGW
|
||||
***************************************************************/
|
||||
#if defined(_MSC_VER) && _MSC_VER >= 1400
|
||||
# define LONG_SEEK _fseeki64
|
||||
#elif !defined(__64BIT__) && (PLATFORM_POSIX_VERSION >= 200112L) /* No point defining Large file for 64 bit */
|
||||
# define LONG_SEEK fseeko
|
||||
#elif defined(__MINGW32__) && !defined(__STRICT_ANSI__) && !defined(__NO_MINGW_LFS) && defined(__MSVCRT__)
|
||||
# define LONG_SEEK fseeko64
|
||||
#elif defined(_WIN32) && !defined(__DJGPP__)
|
||||
# include <windows.h>
|
||||
static int LONG_SEEK(FILE* file, __int64 offset, int origin) {
|
||||
LARGE_INTEGER off;
|
||||
DWORD method;
|
||||
off.QuadPart = offset;
|
||||
if (origin == SEEK_END)
|
||||
method = FILE_END;
|
||||
else if (origin == SEEK_CUR)
|
||||
method = FILE_CURRENT;
|
||||
else
|
||||
method = FILE_BEGIN;
|
||||
|
||||
if (SetFilePointerEx((HANDLE) _get_osfhandle(_fileno(file)), off, NULL, method))
|
||||
return 0;
|
||||
else
|
||||
return -1;
|
||||
}
|
||||
#else
|
||||
# define LONG_SEEK fseek
|
||||
#endif
|
||||
|
||||
#include <stdlib.h> /* malloc, free */
|
||||
#include <stdio.h> /* FILE* */
|
||||
|
||||
#define XXH_STATIC_LINKING_ONLY
|
||||
#define XXH_NAMESPACE ZSTD_
|
||||
#include "xxhash.h"
|
||||
|
||||
#define ZSTD_STATIC_LINKING_ONLY
|
||||
#include "zstd.h"
|
||||
#include "zstd_errors.h"
|
||||
#include "mem.h"
|
||||
#include "zstd_seekable.h"
|
||||
|
||||
#undef ERROR
|
||||
#define ERROR(name) ((size_t)-ZSTD_error_##name)
|
||||
|
||||
#define CHECK_IO(f) { int const errcod = (f); if (errcod < 0) return ERROR(seekableIO); }
|
||||
|
||||
#undef MIN
|
||||
#undef MAX
|
||||
#define MIN(a, b) ((a) < (b) ? (a) : (b))
|
||||
#define MAX(a, b) ((a) > (b) ? (a) : (b))
|
||||
|
||||
/* Special-case callbacks for FILE* and in-memory modes, so that we can treat
|
||||
* them the same way as the advanced API */
|
||||
static int ZSTD_seekable_read_FILE(void* opaque, void* buffer, size_t n)
|
||||
{
|
||||
size_t const result = fread(buffer, 1, n, (FILE*)opaque);
|
||||
if (result != n) {
|
||||
return -1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int ZSTD_seekable_seek_FILE(void* opaque, S64 offset, int origin)
|
||||
{
|
||||
int const ret = LONG_SEEK((FILE*)opaque, offset, origin);
|
||||
if (ret) return ret;
|
||||
return fflush((FILE*)opaque);
|
||||
}
|
||||
|
||||
typedef struct {
|
||||
const void *ptr;
|
||||
size_t size;
|
||||
size_t pos;
|
||||
} buffWrapper_t;
|
||||
|
||||
static int ZSTD_seekable_read_buff(void* opaque, void* buffer, size_t n)
|
||||
{
|
||||
buffWrapper_t* buff = (buffWrapper_t*) opaque;
|
||||
if (buff->size + n > buff->pos) return -1;
|
||||
memcpy(buffer, (const BYTE*)buff->ptr + buff->pos, n);
|
||||
buff->pos += n;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int ZSTD_seekable_seek_buff(void* opaque, S64 offset, int origin)
|
||||
{
|
||||
buffWrapper_t* buff = (buffWrapper_t*) opaque;
|
||||
unsigned long long newOffset;
|
||||
switch (origin) {
|
||||
case SEEK_SET:
|
||||
newOffset = offset;
|
||||
break;
|
||||
case SEEK_CUR:
|
||||
newOffset = (unsigned long long)buff->pos + offset;
|
||||
break;
|
||||
case SEEK_END:
|
||||
newOffset = (unsigned long long)buff->size - offset;
|
||||
break;
|
||||
}
|
||||
if (newOffset < 0 || newOffset > buff->size) {
|
||||
return -1;
|
||||
}
|
||||
buff->pos = newOffset;
|
||||
return 0;
|
||||
}
|
||||
|
||||
typedef struct {
|
||||
U64 cOffset;
|
||||
U64 dOffset;
|
||||
U32 checksum;
|
||||
} seekEntry_t;
|
||||
|
||||
typedef struct {
|
||||
seekEntry_t* entries;
|
||||
size_t tableLen;
|
||||
|
||||
int checksumFlag;
|
||||
} seekTable_t;
|
||||
|
||||
#define SEEKABLE_BUFF_SIZE ZSTD_BLOCKSIZE_ABSOLUTEMAX
|
||||
|
||||
struct ZSTD_seekable_s {
|
||||
ZSTD_DStream* dstream;
|
||||
seekTable_t seekTable;
|
||||
ZSTD_seekable_customFile src;
|
||||
|
||||
U64 decompressedOffset;
|
||||
U32 curFrame;
|
||||
|
||||
BYTE inBuff[SEEKABLE_BUFF_SIZE]; /* need to do our own input buffering */
|
||||
BYTE outBuff[SEEKABLE_BUFF_SIZE]; /* so we can efficiently decompress the
|
||||
starts of chunks before we get to the
|
||||
desired section */
|
||||
ZSTD_inBuffer in; /* maintain continuity across ZSTD_seekable_decompress operations */
|
||||
buffWrapper_t buffWrapper; /* for `src.opaque` in in-memory mode */
|
||||
|
||||
XXH64_state_t xxhState;
|
||||
};
|
||||
|
||||
ZSTD_seekable* ZSTD_seekable_create(void)
|
||||
{
|
||||
ZSTD_seekable* zs = malloc(sizeof(ZSTD_seekable));
|
||||
|
||||
if (zs == NULL) return NULL;
|
||||
|
||||
/* also initializes stage to zsds_init */
|
||||
memset(zs, 0, sizeof(*zs));
|
||||
|
||||
zs->dstream = ZSTD_createDStream();
|
||||
if (zs->dstream == NULL) {
|
||||
free(zs);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
return zs;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_free(ZSTD_seekable* zs)
|
||||
{
|
||||
if (zs == NULL) return 0; /* support free on null */
|
||||
ZSTD_freeDStream(zs->dstream);
|
||||
free(zs->seekTable.entries);
|
||||
free(zs);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/** ZSTD_seekable_offsetToFrameIndex() :
|
||||
* Performs a binary search to find the last frame with a decompressed offset
|
||||
* <= pos
|
||||
* @return : the frame's index */
|
||||
U32 ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* const zs, U64 pos)
|
||||
{
|
||||
U32 lo = 0;
|
||||
U32 hi = zs->seekTable.tableLen;
|
||||
|
||||
if (pos >= zs->seekTable.entries[zs->seekTable.tableLen].dOffset) {
|
||||
return zs->seekTable.tableLen;
|
||||
}
|
||||
|
||||
while (lo + 1 < hi) {
|
||||
U32 const mid = lo + ((hi - lo) >> 1);
|
||||
if (zs->seekTable.entries[mid].dOffset <= pos) {
|
||||
lo = mid;
|
||||
} else {
|
||||
hi = mid;
|
||||
}
|
||||
}
|
||||
return lo;
|
||||
}
|
||||
|
||||
U32 ZSTD_seekable_getNumFrames(ZSTD_seekable* const zs)
|
||||
{
|
||||
return zs->seekTable.tableLen;
|
||||
}
|
||||
|
||||
U64 ZSTD_seekable_getFrameCompressedOffset(ZSTD_seekable* const zs, U32 frameIndex)
|
||||
{
|
||||
if (frameIndex >= zs->seekTable.tableLen) return ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE;
|
||||
return zs->seekTable.entries[frameIndex].cOffset;
|
||||
}
|
||||
|
||||
U64 ZSTD_seekable_getFrameDecompressedOffset(ZSTD_seekable* const zs, U32 frameIndex)
|
||||
{
|
||||
if (frameIndex >= zs->seekTable.tableLen) return ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE;
|
||||
return zs->seekTable.entries[frameIndex].dOffset;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_getFrameCompressedSize(ZSTD_seekable* const zs, U32 frameIndex)
|
||||
{
|
||||
if (frameIndex >= zs->seekTable.tableLen) return ERROR(frameIndex_tooLarge);
|
||||
return zs->seekTable.entries[frameIndex + 1].cOffset -
|
||||
zs->seekTable.entries[frameIndex].cOffset;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_getFrameDecompressedSize(ZSTD_seekable* const zs, U32 frameIndex)
|
||||
{
|
||||
if (frameIndex > zs->seekTable.tableLen) return ERROR(frameIndex_tooLarge);
|
||||
return zs->seekTable.entries[frameIndex + 1].dOffset -
|
||||
zs->seekTable.entries[frameIndex].dOffset;
|
||||
}
|
||||
|
||||
static size_t ZSTD_seekable_loadSeekTable(ZSTD_seekable* zs)
|
||||
{
|
||||
int checksumFlag;
|
||||
ZSTD_seekable_customFile src = zs->src;
|
||||
/* read the footer, fixed size */
|
||||
CHECK_IO(src.seek(src.opaque, -(int)ZSTD_seekTableFooterSize, SEEK_END));
|
||||
CHECK_IO(src.read(src.opaque, zs->inBuff, ZSTD_seekTableFooterSize));
|
||||
|
||||
if (MEM_readLE32(zs->inBuff + 5) != ZSTD_SEEKABLE_MAGICNUMBER) {
|
||||
return ERROR(prefix_unknown);
|
||||
}
|
||||
|
||||
{ BYTE const sfd = zs->inBuff[4];
|
||||
checksumFlag = sfd >> 7;
|
||||
|
||||
/* check reserved bits */
|
||||
if ((checksumFlag >> 2) & 0x1f) {
|
||||
return ERROR(corruption_detected);
|
||||
}
|
||||
}
|
||||
|
||||
{ U32 const numFrames = MEM_readLE32(zs->inBuff);
|
||||
U32 const sizePerEntry = 8 + (checksumFlag?4:0);
|
||||
U32 const tableSize = sizePerEntry * numFrames;
|
||||
U32 const frameSize = tableSize + ZSTD_seekTableFooterSize + ZSTD_skippableHeaderSize;
|
||||
|
||||
U32 remaining = frameSize - ZSTD_seekTableFooterSize; /* don't need to re-read footer */
|
||||
{
|
||||
U32 const toRead = MIN(remaining, SEEKABLE_BUFF_SIZE);
|
||||
|
||||
CHECK_IO(src.seek(src.opaque, -(S64)frameSize, SEEK_END));
|
||||
CHECK_IO(src.read(src.opaque, zs->inBuff, toRead));
|
||||
|
||||
remaining -= toRead;
|
||||
}
|
||||
|
||||
if (MEM_readLE32(zs->inBuff) != (ZSTD_MAGIC_SKIPPABLE_START | 0xE)) {
|
||||
return ERROR(prefix_unknown);
|
||||
}
|
||||
if (MEM_readLE32(zs->inBuff+4) + ZSTD_skippableHeaderSize != frameSize) {
|
||||
return ERROR(prefix_unknown);
|
||||
}
|
||||
|
||||
{ /* Allocate an extra entry at the end so that we can do size
|
||||
* computations on the last element without special case */
|
||||
seekEntry_t* entries = (seekEntry_t*)malloc(sizeof(seekEntry_t) * (numFrames + 1));
|
||||
const BYTE* tableBase = zs->inBuff + ZSTD_skippableHeaderSize;
|
||||
|
||||
U32 idx = 0;
|
||||
U32 pos = 8;
|
||||
|
||||
|
||||
U64 cOffset = 0;
|
||||
U64 dOffset = 0;
|
||||
|
||||
if (!entries) {
|
||||
free(entries);
|
||||
return ERROR(memory_allocation);
|
||||
}
|
||||
|
||||
/* compute cumulative positions */
|
||||
for (; idx < numFrames; idx++) {
|
||||
if (pos + sizePerEntry > SEEKABLE_BUFF_SIZE) {
|
||||
U32 const toRead = MIN(remaining, SEEKABLE_BUFF_SIZE);
|
||||
U32 const offset = SEEKABLE_BUFF_SIZE - pos;
|
||||
memmove(zs->inBuff, zs->inBuff + pos, offset); /* move any data we haven't read yet */
|
||||
CHECK_IO(src.read(src.opaque, zs->inBuff+offset, toRead));
|
||||
remaining -= toRead;
|
||||
pos = 0;
|
||||
}
|
||||
entries[idx].cOffset = cOffset;
|
||||
entries[idx].dOffset = dOffset;
|
||||
|
||||
cOffset += MEM_readLE32(zs->inBuff + pos);
|
||||
pos += 4;
|
||||
dOffset += MEM_readLE32(zs->inBuff + pos);
|
||||
pos += 4;
|
||||
if (checksumFlag) {
|
||||
entries[idx].checksum = MEM_readLE32(zs->inBuff + pos);
|
||||
pos += 4;
|
||||
}
|
||||
}
|
||||
entries[numFrames].cOffset = cOffset;
|
||||
entries[numFrames].dOffset = dOffset;
|
||||
|
||||
zs->seekTable.entries = entries;
|
||||
zs->seekTable.tableLen = numFrames;
|
||||
zs->seekTable.checksumFlag = checksumFlag;
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_initBuff(ZSTD_seekable* zs, const void* src, size_t srcSize)
|
||||
{
|
||||
zs->buffWrapper = (buffWrapper_t){src, srcSize, 0};
|
||||
{ ZSTD_seekable_customFile srcFile = {&zs->buffWrapper,
|
||||
&ZSTD_seekable_read_buff,
|
||||
&ZSTD_seekable_seek_buff};
|
||||
return ZSTD_seekable_initAdvanced(zs, srcFile); }
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_initFile(ZSTD_seekable* zs, FILE* src)
|
||||
{
|
||||
ZSTD_seekable_customFile srcFile = {src, &ZSTD_seekable_read_FILE,
|
||||
&ZSTD_seekable_seek_FILE};
|
||||
return ZSTD_seekable_initAdvanced(zs, srcFile);
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_initAdvanced(ZSTD_seekable* zs, ZSTD_seekable_customFile src)
|
||||
{
|
||||
zs->src = src;
|
||||
|
||||
{ const size_t seekTableInit = ZSTD_seekable_loadSeekTable(zs);
|
||||
if (ZSTD_isError(seekTableInit)) return seekTableInit; }
|
||||
|
||||
zs->decompressedOffset = (U64)-1;
|
||||
zs->curFrame = (U32)-1;
|
||||
|
||||
{ const size_t dstreamInit = ZSTD_initDStream(zs->dstream);
|
||||
if (ZSTD_isError(dstreamInit)) return dstreamInit; }
|
||||
return 0;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_decompress(ZSTD_seekable* zs, void* dst, size_t len, U64 offset)
|
||||
{
|
||||
U32 targetFrame = ZSTD_seekable_offsetToFrameIndex(zs, offset);
|
||||
do {
|
||||
/* check if we can continue from a previous decompress job */
|
||||
if (targetFrame != zs->curFrame || offset != zs->decompressedOffset) {
|
||||
zs->decompressedOffset = zs->seekTable.entries[targetFrame].dOffset;
|
||||
zs->curFrame = targetFrame;
|
||||
|
||||
CHECK_IO(zs->src.seek(zs->src.opaque,
|
||||
zs->seekTable.entries[targetFrame].cOffset,
|
||||
SEEK_SET));
|
||||
zs->in = (ZSTD_inBuffer){zs->inBuff, 0, 0};
|
||||
XXH64_reset(&zs->xxhState, 0);
|
||||
ZSTD_resetDStream(zs->dstream);
|
||||
}
|
||||
|
||||
while (zs->decompressedOffset < offset + len) {
|
||||
size_t toRead;
|
||||
ZSTD_outBuffer outTmp;
|
||||
size_t prevOutPos;
|
||||
if (zs->decompressedOffset < offset) {
|
||||
/* dummy decompressions until we get to the target offset */
|
||||
outTmp = (ZSTD_outBuffer){zs->outBuff, MIN(SEEKABLE_BUFF_SIZE, offset - zs->decompressedOffset), 0};
|
||||
} else {
|
||||
outTmp = (ZSTD_outBuffer){dst, len, zs->decompressedOffset - offset};
|
||||
}
|
||||
|
||||
prevOutPos = outTmp.pos;
|
||||
toRead = ZSTD_decompressStream(zs->dstream, &outTmp, &zs->in);
|
||||
if (ZSTD_isError(toRead)) {
|
||||
return toRead;
|
||||
}
|
||||
|
||||
if (zs->seekTable.checksumFlag) {
|
||||
XXH64_update(&zs->xxhState, (BYTE*)outTmp.dst + prevOutPos,
|
||||
outTmp.pos - prevOutPos);
|
||||
}
|
||||
zs->decompressedOffset += outTmp.pos - prevOutPos;
|
||||
|
||||
if (toRead == 0) {
|
||||
/* frame complete */
|
||||
|
||||
/* verify checksum */
|
||||
if (zs->seekTable.checksumFlag &&
|
||||
(XXH64_digest(&zs->xxhState) & 0xFFFFFFFFU) !=
|
||||
zs->seekTable.entries[targetFrame].checksum) {
|
||||
return ERROR(corruption_detected);
|
||||
}
|
||||
|
||||
if (zs->decompressedOffset < offset + len) {
|
||||
/* go back to the start and force a reset of the stream */
|
||||
targetFrame = ZSTD_seekable_offsetToFrameIndex(zs, zs->decompressedOffset);
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
/* read in more data if we're done with this buffer */
|
||||
if (zs->in.pos == zs->in.size) {
|
||||
toRead = MIN(toRead, SEEKABLE_BUFF_SIZE);
|
||||
CHECK_IO(zs->src.read(zs->src.opaque, zs->inBuff, toRead));
|
||||
zs->in.size = toRead;
|
||||
zs->in.pos = 0;
|
||||
}
|
||||
}
|
||||
} while (zs->decompressedOffset != offset + len);
|
||||
|
||||
return len;
|
||||
}
|
||||
|
||||
size_t ZSTD_seekable_decompressFrame(ZSTD_seekable* zs, void* dst, size_t dstSize, U32 frameIndex)
|
||||
{
|
||||
if (frameIndex >= zs->seekTable.tableLen) {
|
||||
return ERROR(frameIndex_tooLarge);
|
||||
}
|
||||
|
||||
{
|
||||
size_t const decompressedSize =
|
||||
zs->seekTable.entries[frameIndex + 1].dOffset -
|
||||
zs->seekTable.entries[frameIndex].dOffset;
|
||||
if (dstSize < decompressedSize) {
|
||||
return ERROR(dstSize_tooSmall);
|
||||
}
|
||||
return ZSTD_seekable_decompress(
|
||||
zs, dst, decompressedSize,
|
||||
zs->seekTable.entries[frameIndex].dOffset);
|
||||
}
|
||||
}
|
34
doc/educational_decoder/Makefile
Normal file
34
doc/educational_decoder/Makefile
Normal file
@ -0,0 +1,34 @@
|
||||
HARNESS_FILES=*.c
|
||||
|
||||
MULTITHREAD_LDFLAGS = -pthread
|
||||
DEBUGFLAGS= -g -DZSTD_DEBUG=1
|
||||
CPPFLAGS += -I$(ZSTDDIR) -I$(ZSTDDIR)/common -I$(ZSTDDIR)/compress \
|
||||
-I$(ZSTDDIR)/dictBuilder -I$(ZSTDDIR)/deprecated -I$(PRGDIR)
|
||||
CFLAGS ?= -O3
|
||||
CFLAGS += -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \
|
||||
-Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \
|
||||
-Wstrict-prototypes -Wundef -Wformat-security \
|
||||
-Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \
|
||||
-Wredundant-decls
|
||||
CFLAGS += $(DEBUGFLAGS)
|
||||
CFLAGS += $(MOREFLAGS)
|
||||
FLAGS = $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(MULTITHREAD_LDFLAGS)
|
||||
|
||||
harness: $(HARNESS_FILES)
|
||||
$(CC) $(FLAGS) $^ -o $@
|
||||
|
||||
clean:
|
||||
@$(RM) -f harness
|
||||
@$(RM) -rf harness.dSYM
|
||||
|
||||
test: harness
|
||||
@zstd README.md -o tmp.zst
|
||||
@./harness tmp.zst tmp
|
||||
@diff -s tmp README.md
|
||||
@$(RM) -f tmp*
|
||||
@zstd --train harness.c zstd_decompress.c zstd_decompress.h README.md
|
||||
@zstd -D dictionary README.md -o tmp.zst
|
||||
@./harness tmp.zst tmp dictionary
|
||||
@diff -s tmp README.md
|
||||
@$(RM) -f tmp* dictionary
|
||||
@make clean
|
@ -2,9 +2,9 @@
|
||||
* Copyright (c) 2017-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
#include <stdio.h>
|
||||
|
@ -2,9 +2,9 @@
|
||||
* Copyright (c) 2017-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
/// Zstandard educational decoder implementation
|
||||
@ -1289,7 +1289,7 @@ static void execute_sequences(frame_context_t *const ctx, ostream_t *const out,
|
||||
// Copy any leftover literals
|
||||
{
|
||||
size_t len = IO_istream_len(&litstream);
|
||||
copy_literals(len, &litstream, out);
|
||||
copy_literals(len, &litstream, out);
|
||||
total_output += len;
|
||||
}
|
||||
|
||||
|
@ -1,10 +1,10 @@
|
||||
/*
|
||||
* Copyright (c) 2017-present, Facebook, Inc.
|
||||
* Copyright (c) 2016-present, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
/******* EXPOSED TYPES ********************************************************/
|
||||
|
@ -1,10 +1,10 @@
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
|
||||
<title>zstd 1.3.1 Manual</title>
|
||||
<title>zstd 1.3.2 Manual</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>zstd 1.3.1 Manual</h1>
|
||||
<h1>zstd 1.3.2 Manual</h1>
|
||||
<hr>
|
||||
<a name="Contents"></a><h2>Contents</h2>
|
||||
<ol>
|
||||
@ -27,7 +27,8 @@
|
||||
<li><a href="#Chapter17">Buffer-less and synchronous inner streaming functions</a></li>
|
||||
<li><a href="#Chapter18">Buffer-less streaming compression (synchronous mode)</a></li>
|
||||
<li><a href="#Chapter19">Buffer-less streaming decompression (synchronous mode)</a></li>
|
||||
<li><a href="#Chapter20">Block functions</a></li>
|
||||
<li><a href="#Chapter20">New advanced API (experimental)</a></li>
|
||||
<li><a href="#Chapter21">Block level API</a></li>
|
||||
</ol>
|
||||
<hr>
|
||||
<a name="Chapter1"></a><h2>Introduction</h2><pre>
|
||||
@ -110,10 +111,11 @@ unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize);
|
||||
@return : content size to be decompressed, as a 64-bits value _if known and not empty_, 0 otherwise.
|
||||
</p></pre><BR>
|
||||
|
||||
<h3>Helper functions</h3><pre></pre><b><pre>int ZSTD_maxCLevel(void); </b>/*!< maximum compression level available */<b>
|
||||
<h3>Helper functions</h3><pre></pre><b><pre>#define ZSTD_COMPRESSBOUND(srcSize) ((srcSize) + ((srcSize)>>8) + (((srcSize) < 128 KB) ? ((128 KB - (srcSize)) >> 11) </b>/* margin, from 64 to 0 */ : 0)) /* this formula ensures that bound(A) + bound(B) <= bound(A+B) as long as A and B >= 128 KB */<b>
|
||||
size_t ZSTD_compressBound(size_t srcSize); </b>/*!< maximum compressed size in worst case scenario */<b>
|
||||
unsigned ZSTD_isError(size_t code); </b>/*!< tells if a `size_t` function result is an error code */<b>
|
||||
const char* ZSTD_getErrorName(size_t code); </b>/*!< provides readable string from an error code */<b>
|
||||
int ZSTD_maxCLevel(void); </b>/*!< maximum compression level available */<b>
|
||||
</pre></b><BR>
|
||||
<a name="Chapter4"></a><h2>Explicit memory management</h2><pre></pre>
|
||||
|
||||
@ -398,29 +400,33 @@ size_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds);
|
||||
size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict);
|
||||
size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
|
||||
</b><p> These functions give the current memory usage of selected object.
|
||||
Object memory usage can evolve if it's re-used multiple times.
|
||||
Object memory usage can evolve when re-used multiple times.
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_estimateCCtxSize(int compressionLevel);
|
||||
size_t ZSTD_estimateCCtxSize_advanced(ZSTD_compressionParameters cParams);
|
||||
size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams);
|
||||
size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params);
|
||||
size_t ZSTD_estimateDCtxSize(void);
|
||||
</b><p> These functions make it possible to estimate memory usage
|
||||
of a future {D,C}Ctx, before its creation.
|
||||
ZSTD_estimateCCtxSize() will provide a budget large enough for any compression level up to selected one.
|
||||
It will also consider src size to be arbitrarily "large", which is worst case.
|
||||
If srcSize is known to always be small, ZSTD_estimateCCtxSize_advanced() can provide a tighter estimation.
|
||||
ZSTD_estimateCCtxSize_advanced() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel.
|
||||
If srcSize is known to always be small, ZSTD_estimateCCtxSize_usingCParams() can provide a tighter estimation.
|
||||
ZSTD_estimateCCtxSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel.
|
||||
ZSTD_estimateCCtxSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParam_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_p_nbThreads is > 1.
|
||||
Note : CCtx estimation is only correct for single-threaded compression
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_estimateCStreamSize(int compressionLevel);
|
||||
size_t ZSTD_estimateCStreamSize_advanced(ZSTD_compressionParameters cParams);
|
||||
size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams);
|
||||
size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params);
|
||||
size_t ZSTD_estimateDStreamSize(size_t windowSize);
|
||||
size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize);
|
||||
</b><p> ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one.
|
||||
It will also consider src size to be arbitrarily "large", which is worst case.
|
||||
If srcSize is known to always be small, ZSTD_estimateCStreamSize_advanced() can provide a tighter estimation.
|
||||
ZSTD_estimateCStreamSize_advanced() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel.
|
||||
If srcSize is known to always be small, ZSTD_estimateCStreamSize_usingCParams() can provide a tighter estimation.
|
||||
ZSTD_estimateCStreamSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel.
|
||||
ZSTD_estimateCStreamSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParam_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_p_nbThreads is set to a value > 1.
|
||||
Note : CStream estimation is only correct for single-threaded compression.
|
||||
ZSTD_DStream memory budget depends on window Size.
|
||||
This information can be passed manually, using ZSTD_estimateDStreamSize,
|
||||
@ -430,12 +436,18 @@ size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize);
|
||||
In this case, get total size by adding ZSTD_estimate?DictSize
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>typedef enum {
|
||||
ZSTD_dlm_byCopy = 0, </b>/**< Copy dictionary content internally */<b>
|
||||
ZSTD_dlm_byRef, </b>/**< Reference dictionary content -- the dictionary buffer must outlive its users. */<b>
|
||||
} ZSTD_dictLoadMethod_e;
|
||||
</b></pre><BR>
|
||||
<pre><b>size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel);
|
||||
size_t ZSTD_estimateCDictSize_advanced(size_t dictSize, ZSTD_compressionParameters cParams, unsigned byReference);
|
||||
size_t ZSTD_estimateDDictSize(size_t dictSize, unsigned byReference);
|
||||
size_t ZSTD_estimateCDictSize_advanced(size_t dictSize, ZSTD_compressionParameters cParams, ZSTD_dictLoadMethod_e dictLoadMethod);
|
||||
size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod);
|
||||
</b><p> ZSTD_estimateCDictSize() will bet that src size is relatively "small", and content is copied, like ZSTD_createCDict().
|
||||
ZSTD_estimateCStreamSize_advanced() makes it possible to control precisely compression parameters, like ZSTD_createCDict_advanced().
|
||||
Note : dictionary created "byReference" are smaller
|
||||
ZSTD_estimateCStreamSize_advanced_usingCParams() makes it possible to control precisely compression parameters, like ZSTD_createCDict_advanced().
|
||||
Note : dictionary created by reference using ZSTD_dlm_byRef are smaller
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<a name="Chapter14"></a><h2>Advanced compression functions</h2><pre></pre>
|
||||
@ -461,16 +473,6 @@ size_t ZSTD_estimateDDictSize(size_t dictSize, unsigned byReference);
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>typedef enum {
|
||||
ZSTD_p_forceWindow, </b>/* Force back-references to remain < windowSize, even when referencing Dictionary content (default:0) */<b>
|
||||
ZSTD_p_forceRawDict </b>/* Force loading dictionary in "content-only" mode (no header analysis) */<b>
|
||||
} ZSTD_CCtxParameter;
|
||||
</b></pre><BR>
|
||||
<pre><b>size_t ZSTD_setCCtxParameter(ZSTD_CCtx* cctx, ZSTD_CCtxParameter param, unsigned value);
|
||||
</b><p> Set advanced parameters, selected through enum ZSTD_CCtxParameter
|
||||
@result : 0, or an error code (which can be tested with ZSTD_isError())
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>ZSTD_CDict* ZSTD_createCDict_byReference(const void* dictBuffer, size_t dictSize, int compressionLevel);
|
||||
</b><p> Create a digested dictionary for compression
|
||||
Dictionary content is simply referenced, and therefore stays in dictBuffer.
|
||||
@ -483,7 +485,8 @@ size_t ZSTD_estimateDDictSize(size_t dictSize, unsigned byReference);
|
||||
} ZSTD_dictMode_e;
|
||||
</b></pre><BR>
|
||||
<pre><b>ZSTD_CDict* ZSTD_createCDict_advanced(const void* dict, size_t dictSize,
|
||||
unsigned byReference, ZSTD_dictMode_e dictMode,
|
||||
ZSTD_dictLoadMethod_e dictLoadMethod,
|
||||
ZSTD_dictMode_e dictMode,
|
||||
ZSTD_compressionParameters cParams,
|
||||
ZSTD_customMem customMem);
|
||||
</b><p> Create a ZSTD_CDict using external alloc and free, and customized compression parameters
|
||||
@ -492,7 +495,7 @@ size_t ZSTD_estimateDDictSize(size_t dictSize, unsigned byReference);
|
||||
<pre><b>ZSTD_CDict* ZSTD_initStaticCDict(
|
||||
void* workspace, size_t workspaceSize,
|
||||
const void* dict, size_t dictSize,
|
||||
unsigned byReference, ZSTD_dictMode_e dictMode,
|
||||
ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictMode_e dictMode,
|
||||
ZSTD_compressionParameters cParams);
|
||||
</b><p> Generate a digested dictionary in provided memory area.
|
||||
workspace: The memory area to emplace the dictionary into.
|
||||
@ -580,13 +583,14 @@ size_t ZSTD_estimateDDictSize(size_t dictSize, unsigned byReference);
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>ZSTD_DDict* ZSTD_createDDict_advanced(const void* dict, size_t dictSize,
|
||||
unsigned byReference, ZSTD_customMem customMem);
|
||||
ZSTD_dictLoadMethod_e dictLoadMethod,
|
||||
ZSTD_customMem customMem);
|
||||
</b><p> Create a ZSTD_DDict using external alloc and free, optionally by reference
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>ZSTD_DDict* ZSTD_initStaticDDict(void* workspace, size_t workspaceSize,
|
||||
const void* dict, size_t dictSize,
|
||||
unsigned byReference);
|
||||
ZSTD_dictLoadMethod_e dictLoadMethod);
|
||||
</b><p> Generate a digested dictionary in provided memory area.
|
||||
workspace: The memory area to emplace the dictionary into.
|
||||
Provided pointer must 8-bytes aligned.
|
||||
@ -628,9 +632,9 @@ size_t ZSTD_estimateDDictSize(size_t dictSize, unsigned byReference);
|
||||
<h3>Advanced Streaming compression functions</h3><pre></pre><b><pre>ZSTD_CStream* ZSTD_createCStream_advanced(ZSTD_customMem customMem);
|
||||
ZSTD_CStream* ZSTD_initStaticCStream(void* workspace, size_t workspaceSize); </b>/**< same as ZSTD_initStaticCCtx() */<b>
|
||||
size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, int compressionLevel, unsigned long long pledgedSrcSize); </b>/**< pledgedSrcSize must be correct, a size of 0 means unknown. for a frame size of 0 use initCStream_advanced */<b>
|
||||
size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel); </b>/**< creates of an internal CDict (incompatible with static CCtx), except if dict == NULL or dictSize < 8, in which case no dict is used. */<b>
|
||||
size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel); </b>/**< creates of an internal CDict (incompatible with static CCtx), except if dict == NULL or dictSize < 8, in which case no dict is used. Note: dict is loaded with ZSTD_dm_auto (treated as a full zstd dictionary if it begins with ZSTD_MAGIC_DICTIONARY, else as raw content) and ZSTD_dlm_byCopy.*/<b>
|
||||
size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs, const void* dict, size_t dictSize,
|
||||
ZSTD_parameters params, unsigned long long pledgedSrcSize); </b>/**< pledgedSrcSize is optional and can be 0 (meaning unknown). note: if the contentSizeFlag is set, pledgedSrcSize == 0 means the source size is actually 0 */<b>
|
||||
ZSTD_parameters params, unsigned long long pledgedSrcSize); </b>/**< pledgedSrcSize is optional and can be 0 (meaning unknown). note: if the contentSizeFlag is set, pledgedSrcSize == 0 means the source size is actually 0. dict is loaded with ZSTD_dm_auto and ZSTD_dlm_byCopy. */<b>
|
||||
size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict); </b>/**< note : cdict will just be referenced, and must outlive compression session */<b>
|
||||
size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, const ZSTD_CDict* cdict, ZSTD_frameParameters fParams, unsigned long long pledgedSrcSize); </b>/**< same as ZSTD_initCStream_usingCDict(), with control over frame parameters */<b>
|
||||
</pre></b><BR>
|
||||
@ -643,18 +647,18 @@ size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, const ZSTD_CDict*
|
||||
@return : 0, or an error code (which can be tested using ZSTD_isError())
|
||||
</p></pre><BR>
|
||||
|
||||
<h3>Advanced Streaming decompression functions</h3><pre></pre><b><pre>typedef enum { DStream_p_maxWindowSize } ZSTD_DStreamParameter_e;
|
||||
ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem);
|
||||
<h3>Advanced Streaming decompression functions</h3><pre></pre><b><pre>ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem);
|
||||
ZSTD_DStream* ZSTD_initStaticDStream(void* workspace, size_t workspaceSize); </b>/**< same as ZSTD_initStaticDCtx() */<b>
|
||||
size_t ZSTD_setDStreamParameter(ZSTD_DStream* zds, ZSTD_DStreamParameter_e paramType, unsigned paramValue);
|
||||
size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize); </b>/**< note: a dict will not be used if dict == NULL or dictSize < 8 */<b>
|
||||
size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict); </b>/**< note : ddict will just be referenced, and must outlive decompression session */<b>
|
||||
typedef enum { DStream_p_maxWindowSize } ZSTD_DStreamParameter_e;
|
||||
size_t ZSTD_setDStreamParameter(ZSTD_DStream* zds, ZSTD_DStreamParameter_e paramType, unsigned paramValue); </b>/* obsolete : this API will be removed in a future version */<b>
|
||||
size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize); </b>/**< note: no dictionary will be used if dict == NULL or dictSize < 8 */<b>
|
||||
size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict); </b>/**< note : ddict is referenced, it must outlive decompression session */<b>
|
||||
size_t ZSTD_resetDStream(ZSTD_DStream* zds); </b>/**< re-use decompression parameters from previous init; saves dictionary loading */<b>
|
||||
</pre></b><BR>
|
||||
<a name="Chapter17"></a><h2>Buffer-less and synchronous inner streaming functions</h2><pre>
|
||||
This is an advanced API, giving full control over buffer management, for users which need direct control over memory.
|
||||
But it's also a complex one, with many restrictions (documented below).
|
||||
Prefer using normal streaming API for an easier experience
|
||||
But it's also a complex one, with several restrictions, documented below.
|
||||
Prefer normal streaming API for an easier experience.
|
||||
|
||||
<BR></pre>
|
||||
|
||||
@ -670,8 +674,8 @@ size_t ZSTD_resetDStream(ZSTD_DStream* zds); </b>/**< re-use decompression para
|
||||
|
||||
Then, consume your input using ZSTD_compressContinue().
|
||||
There are some important considerations to keep in mind when using this advanced function :
|
||||
- ZSTD_compressContinue() has no internal buffer. It uses externally provided buffer only.
|
||||
- Interface is synchronous : input is consumed entirely and produce 1+ (or more) compressed blocks.
|
||||
- ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
|
||||
- Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
|
||||
- Caller must ensure there is enough space in `dst` to store compressed data under worst case scenario.
|
||||
Worst case evaluation is provided by ZSTD_compressBound().
|
||||
ZSTD_compressContinue() doesn't guarantee recover after a failed compression.
|
||||
@ -682,9 +686,9 @@ size_t ZSTD_resetDStream(ZSTD_DStream* zds); </b>/**< re-use decompression para
|
||||
|
||||
Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum.
|
||||
It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame.
|
||||
Without last block mark, frames will be considered unfinished (corrupted) by decoders.
|
||||
Without last block mark, frames are considered unfinished (hence corrupted) by compliant decoders.
|
||||
|
||||
`ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress some new frame.
|
||||
`ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
|
||||
<BR></pre>
|
||||
|
||||
<h3>Buffer-less streaming compression functions</h3><pre></pre><b><pre>size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel);
|
||||
@ -700,40 +704,53 @@ size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned lo
|
||||
A ZSTD_DCtx object can be re-used multiple times.
|
||||
|
||||
First typical operation is to retrieve frame parameters, using ZSTD_getFrameHeader().
|
||||
It fills a ZSTD_frameHeader structure with important information to correctly decode the frame,
|
||||
such as minimum rolling buffer size to allocate to decompress data (`windowSize`),
|
||||
and the dictionary ID in use.
|
||||
(Note : content size is optional, it may not be present. 0 means : content size unknown).
|
||||
Note that these values could be wrong, either because of data malformation, or because an attacker is spoofing deliberate false information.
|
||||
As a consequence, check that values remain within valid application range, especially `windowSize`, before allocation.
|
||||
Each application can set its own limit, depending on local restrictions.
|
||||
For extended interoperability, it is recommended to support windowSize of at least 8 MB.
|
||||
Frame header is extracted from the beginning of compressed frame, so providing only the frame's beginning is enough.
|
||||
Data fragment must be large enough to ensure successful decoding.
|
||||
`ZSTD_frameHeaderSize_max` bytes is guaranteed to always be large enough.
|
||||
`ZSTD_frameHeaderSize_max` bytes is guaranteed to always be large enough.
|
||||
@result : 0 : successful decoding, the `ZSTD_frameHeader` structure is correctly filled.
|
||||
>0 : `srcSize` is too small, please provide at least @result bytes on next attempt.
|
||||
errorCode, which can be tested using ZSTD_isError().
|
||||
|
||||
Start decompression, with ZSTD_decompressBegin().
|
||||
It fills a ZSTD_frameHeader structure with important information to correctly decode the frame,
|
||||
such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`).
|
||||
Note that these values could be wrong, either because of data corruption, or because a 3rd party deliberately spoofs false information.
|
||||
As a consequence, check that values remain within valid application range.
|
||||
For example, do not allocate memory blindly, check that `windowSize` is within expectation.
|
||||
Each application can set its own limits, depending on local restrictions.
|
||||
For extended interoperability, it is recommended to support `windowSize` of at least 8 MB.
|
||||
|
||||
ZSTD_decompressContinue() needs previous data blocks during decompression, up to `windowSize` bytes.
|
||||
ZSTD_decompressContinue() is very sensitive to contiguity,
|
||||
if 2 blocks don't follow each other, make sure that either the compressor breaks contiguity at the same place,
|
||||
or that previous contiguous segment is large enough to properly handle maximum back-reference distance.
|
||||
There are multiple ways to guarantee this condition.
|
||||
|
||||
The most memory efficient way is to use a round buffer of sufficient size.
|
||||
Sufficient size is determined by invoking ZSTD_decodingBufferSize_min(),
|
||||
which can @return an error code if required value is too large for current system (in 32-bits mode).
|
||||
In a round buffer methodology, ZSTD_decompressContinue() decompresses each block next to previous one,
|
||||
up to the moment there is not enough room left in the buffer to guarantee decoding another full block,
|
||||
which maximum size is provided in `ZSTD_frameHeader` structure, field `blockSizeMax`.
|
||||
At which point, decoding can resume from the beginning of the buffer.
|
||||
Note that already decoded data stored in the buffer should be flushed before being overwritten.
|
||||
|
||||
There are alternatives possible, for example using two or more buffers of size `windowSize` each, though they consume more memory.
|
||||
|
||||
Finally, if you control the compression process, you can also ignore all buffer size rules,
|
||||
as long as the encoder and decoder progress in "lock-step",
|
||||
aka use exactly the same buffer sizes, break contiguity at the same place, etc.
|
||||
|
||||
Once buffers are setup, start decompression, with ZSTD_decompressBegin().
|
||||
If decompression requires a dictionary, use ZSTD_decompressBegin_usingDict() or ZSTD_decompressBegin_usingDDict().
|
||||
Alternatively, you can copy a prepared context, using ZSTD_copyDCtx().
|
||||
|
||||
Then use ZSTD_nextSrcSizeToDecompress() and ZSTD_decompressContinue() alternatively.
|
||||
ZSTD_nextSrcSizeToDecompress() tells how many bytes to provide as 'srcSize' to ZSTD_decompressContinue().
|
||||
ZSTD_decompressContinue() requires this _exact_ amount of bytes, or it will fail.
|
||||
|
||||
@result of ZSTD_decompressContinue() is the number of bytes regenerated within 'dst' (necessarily <= dstCapacity).
|
||||
It can be zero, which is not an error; it just means ZSTD_decompressContinue() has decoded some metadata item.
|
||||
@result of ZSTD_decompressContinue() is the number of bytes regenerated within 'dst' (necessarily <= dstCapacity).
|
||||
It can be zero : it just means ZSTD_decompressContinue() has decoded some metadata item.
|
||||
It can also be an error code, which can be tested with ZSTD_isError().
|
||||
|
||||
ZSTD_decompressContinue() needs previous data blocks during decompression, up to `windowSize`.
|
||||
They should preferably be located contiguously, prior to current block.
|
||||
Alternatively, a round buffer of sufficient size is also possible. Sufficient size is determined by frame parameters.
|
||||
ZSTD_decompressContinue() is very sensitive to contiguity,
|
||||
if 2 blocks don't follow each other, make sure that either the compressor breaks contiguity at the same place,
|
||||
or that previous contiguous segment is large enough to properly handle maximum back-reference.
|
||||
|
||||
A frame is fully decoded when ZSTD_nextSrcSizeToDecompress() returns zero.
|
||||
Context can then be reset to start a new decompression.
|
||||
|
||||
@ -743,44 +760,62 @@ size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned lo
|
||||
== Special case : skippable frames
|
||||
|
||||
Skippable frames allow integration of user-defined data into a flow of concatenated frames.
|
||||
Skippable frames will be ignored (skipped) by a decompressor. The format of skippable frames is as follows :
|
||||
Skippable frames will be ignored (skipped) by decompressor.
|
||||
The format of skippable frames is as follows :
|
||||
a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
|
||||
b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
|
||||
c) Frame Content - any content (User Data) of length equal to Frame Size
|
||||
For skippable frames ZSTD_decompressContinue() always returns 0.
|
||||
For skippable frames ZSTD_getFrameHeader() returns fparamsPtr->windowLog==0 what means that a frame is skippable.
|
||||
Note : If fparamsPtr->frameContentSize==0, it is ambiguous: the frame might actually be a Zstd encoded frame with no content.
|
||||
For purposes of decompression, it is valid in both cases to skip the frame using
|
||||
ZSTD_findFrameCompressedSize to find its size in bytes.
|
||||
It also returns Frame Size as fparamsPtr->frameContentSize.
|
||||
For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
|
||||
For skippable frames ZSTD_decompressContinue() always returns 0 : it only skips the content.
|
||||
<BR></pre>
|
||||
|
||||
<h3>Buffer-less streaming decompression functions</h3><pre></pre><b><pre>typedef enum { ZSTD_frame, ZSTD_skippableFrame } ZSTD_frameType_e;
|
||||
typedef struct {
|
||||
unsigned long long frameContentSize; </b>/* ZSTD_CONTENTSIZE_UNKNOWN means this field is not available. 0 means "empty" */<b>
|
||||
unsigned long long frameContentSize; </b>/* if == ZSTD_CONTENTSIZE_UNKNOWN, it means this field is not available. 0 means "empty" */<b>
|
||||
unsigned long long windowSize; </b>/* can be very large, up to <= frameContentSize */<b>
|
||||
unsigned blockSizeMax;
|
||||
ZSTD_frameType_e frameType; </b>/* if == ZSTD_skippableFrame, frameContentSize is the size of skippable content */<b>
|
||||
unsigned headerSize;
|
||||
unsigned dictID;
|
||||
unsigned checksumFlag;
|
||||
} ZSTD_frameHeader;
|
||||
size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize); </b>/**< doesn't consume input */<b>
|
||||
size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx);
|
||||
size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);
|
||||
size_t ZSTD_decompressBegin_usingDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict);
|
||||
void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
|
||||
size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long long frameContentSize); </b>/**< when frame content size is not known, pass in frameContentSize == ZSTD_CONTENTSIZE_UNKNOWN */<b>
|
||||
</pre></b><BR>
|
||||
<pre><b>typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e;
|
||||
</b></pre><BR>
|
||||
<h3>New advanced API (experimental, and compression only)</h3><pre></pre><b><pre></pre></b><BR>
|
||||
<a name="Chapter20"></a><h2>New advanced API (experimental)</h2><pre></pre>
|
||||
|
||||
<pre><b>typedef enum {
|
||||
</b>/* Question : should we have a format ZSTD_f_auto ?<b>
|
||||
* For the time being, it would mean exactly the same as ZSTD_f_zstd1.
|
||||
* But, in the future, should several formats be supported,
|
||||
* on the compression side, it would mean "default format".
|
||||
* On the decompression side, it would mean "multi format",
|
||||
* and ZSTD_f_zstd1 could be reserved to mean "accept *only* zstd frames".
|
||||
* Since meaning is a little different, another option could be to define different enums for compression and decompression.
|
||||
* This question could be kept for later, when there are actually multiple formats to support,
|
||||
* but there is also the question of pinning enum values, and pinning value `0` is especially important */
|
||||
ZSTD_f_zstd1 = 0, </b>/* zstd frame format, specified in zstd_compression_format.md (default) */<b>
|
||||
ZSTD_f_zstd1_magicless, </b>/* Variant of zstd frame format, without initial 4-bytes magic number.<b>
|
||||
* Useful to save 4 bytes per generated frame.
|
||||
* Decoder cannot recognise automatically this format, requiring instructions. */
|
||||
} ZSTD_format_e;
|
||||
</b></pre><BR>
|
||||
<pre><b>typedef enum {
|
||||
</b>/* compression format */<b>
|
||||
ZSTD_p_format = 10, </b>/* See ZSTD_format_e enum definition.<b>
|
||||
* Cast selected format as unsigned for ZSTD_CCtx_setParameter() compatibility. */
|
||||
|
||||
</b>/* compression parameters */<b>
|
||||
ZSTD_p_compressionLevel=100, </b>/* Update all compression parameters according to pre-defined cLevel table<b>
|
||||
* Default level is ZSTD_CLEVEL_DEFAULT==3.
|
||||
* Special: value 0 means "do not change cLevel". */
|
||||
ZSTD_p_windowLog, </b>/* Maximum allowed back-reference distance, expressed as power of 2.<b>
|
||||
* Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX.
|
||||
* Special: value 0 means "do not change windowLog". */
|
||||
* Special: value 0 means "do not change windowLog".
|
||||
* Note: Using a window size greater than ZSTD_MAXWINDOWSIZE_DEFAULT (default: 2^27)
|
||||
* requires setting the maximum window size at least as large during decompression. */
|
||||
ZSTD_p_hashLog, </b>/* Size of the probe table, as a power of 2.<b>
|
||||
* Resulting table size is (1 << (hashLog+2)).
|
||||
* Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX.
|
||||
@ -819,12 +854,6 @@ void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
|
||||
ZSTD_p_checksumFlag, </b>/* A 32-bits checksum of content is written at end of frame (default:0) */<b>
|
||||
ZSTD_p_dictIDFlag, </b>/* When applicable, dictID of dictionary is provided in frame header (default:1) */<b>
|
||||
|
||||
</b>/* dictionary parameters (must be set before ZSTD_CCtx_loadDictionary) */<b>
|
||||
ZSTD_p_dictMode=300, </b>/* Select how dictionary content must be interpreted. Value must be from type ZSTD_dictMode_e.<b>
|
||||
* default : 0==auto : dictionary will be "full" if it respects specification, otherwise it will be "rawContent" */
|
||||
ZSTD_p_refDictContent, </b>/* Dictionary content will be referenced, instead of copied (default:0==byCopy).<b>
|
||||
* It requires that dictionary buffer outlives its users */
|
||||
|
||||
</b>/* multi-threading parameters */<b>
|
||||
ZSTD_p_nbThreads=400, </b>/* Select how many threads a compression job can spawn (default:1)<b>
|
||||
* More threads improve speed, but also increase memory usage.
|
||||
@ -840,6 +869,35 @@ void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
|
||||
</b>/* advanced parameters - may not remain available after API update */<b>
|
||||
ZSTD_p_forceMaxWindow=1100, </b>/* Force back-reference distances to remain < windowSize,<b>
|
||||
* even when referencing into Dictionary content (default:0) */
|
||||
ZSTD_p_enableLongDistanceMatching=1200, </b>/* Enable long distance matching.<b>
|
||||
* This parameter is designed to improve the compression
|
||||
* ratio for large inputs with long distance matches.
|
||||
* This increases the memory usage as well as window size.
|
||||
* Note: setting this parameter sets all the LDM parameters
|
||||
* as well as ZSTD_p_windowLog. It should be set after
|
||||
* ZSTD_p_compressionLevel and before ZSTD_p_windowLog and
|
||||
* other LDM parameters. Setting the compression level
|
||||
* after this parameter overrides the window log, though LDM
|
||||
* will remain enabled until explicitly disabled. */
|
||||
ZSTD_p_ldmHashLog, </b>/* Size of the table for long distance matching, as a power of 2.<b>
|
||||
* Larger values increase memory usage and compression ratio, but decrease
|
||||
* compression speed.
|
||||
* Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX
|
||||
* (default: windowlog - 7). */
|
||||
ZSTD_p_ldmMinMatch, </b>/* Minimum size of searched matches for long distance matcher.<b>
|
||||
* Larger/too small values usually decrease compression ratio.
|
||||
* Must be clamped between ZSTD_LDM_MINMATCH_MIN
|
||||
* and ZSTD_LDM_MINMATCH_MAX (default: 64). */
|
||||
ZSTD_p_ldmBucketSizeLog, </b>/* Log size of each bucket in the LDM hash table for collision resolution.<b>
|
||||
* Larger values usually improve collision resolution but may decrease
|
||||
* compression speed.
|
||||
* The maximum value is ZSTD_LDM_BUCKETSIZELOG_MAX (default: 3). */
|
||||
ZSTD_p_ldmHashEveryLog, </b>/* Frequency of inserting/looking up entries in the LDM hash table.<b>
|
||||
* The default is MAX(0, (windowLog - ldmHashLog)) to
|
||||
* optimize hash table usage.
|
||||
* Larger values improve compression speed. Deviating far from the
|
||||
* default value will likely result in a decrease in compression ratio.
|
||||
* Must be clamped between 0 and ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN. */
|
||||
|
||||
} ZSTD_cParameter;
|
||||
</b></pre><BR>
|
||||
@ -861,18 +919,25 @@ void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize);
|
||||
size_t ZSTD_CCtx_loadDictionary_byReference(ZSTD_CCtx* cctx, const void* dict, size_t dictSize);
|
||||
size_t ZSTD_CCtx_loadDictionary_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictMode_e dictMode);
|
||||
</b><p> Create an internal CDict from dict buffer.
|
||||
Decompression will have to use same buffer.
|
||||
@result : 0, or an error code (which can be tested with ZSTD_isError()).
|
||||
Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
|
||||
meaning "return to no-dictionary mode".
|
||||
Note 1 : `dict` content will be copied internally,
|
||||
except if ZSTD_p_refDictContent is set before loading.
|
||||
Note 1 : `dict` content will be copied internally. Use
|
||||
ZSTD_CCtx_loadDictionary_byReference() to reference dictionary
|
||||
content instead. The dictionary buffer must then outlive its
|
||||
users.
|
||||
Note 2 : Loading a dictionary involves building tables, which are dependent on compression parameters.
|
||||
For this reason, compression parameters cannot be changed anymore after loading a dictionary.
|
||||
It's also a CPU-heavy operation, with non-negligible impact on latency.
|
||||
Note 3 : Dictionary will be used for all future compression jobs.
|
||||
To return to "no-dictionary" situation, load a NULL dictionary
|
||||
To return to "no-dictionary" situation, load a NULL dictionary
|
||||
Note 5 : Use ZSTD_CCtx_loadDictionary_advanced() to select how dictionary
|
||||
content will be interpreted.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict);
|
||||
@ -889,6 +954,7 @@ void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize);
|
||||
size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize, ZSTD_dictMode_e dictMode);
|
||||
</b><p> Reference a prefix (single-usage dictionary) for next compression job.
|
||||
Decompression need same prefix to properly regenerate data.
|
||||
Prefix is **only used once**. Tables are discarded at end of compression job.
|
||||
@ -899,13 +965,15 @@ void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
|
||||
Note 1 : Prefix buffer is referenced. It must outlive compression job.
|
||||
Note 2 : Referencing a prefix involves building tables, which are dependent on compression parameters.
|
||||
It's a CPU-heavy operation, with non-negligible impact on latency.
|
||||
Note 3 : it's possible to alter ZSTD_p_dictMode using ZSTD_CCtx_setParameter()
|
||||
Note 3 : By default, the prefix is treated as raw content
|
||||
(ZSTD_dm_rawContent). Use ZSTD_CCtx_refPrefix_advanced() to alter
|
||||
dictMode.
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>typedef enum {
|
||||
ZSTD_e_continue=0, </b>/* collect more data, encoder transparently decides when to output result, for optimal conditions */<b>
|
||||
ZSTD_e_flush, </b>/* flush any data provided so far - frame will continue, future data can still reference previous data for better compression */<b>
|
||||
ZSTD_e_end </b>/* flush any remaining data and ends current frame. Any future compression starts a new frame. */<b>
|
||||
ZSTD_e_end </b>/* flush any remaining data and close current frame. Any additional data starts a new frame. */<b>
|
||||
} ZSTD_EndDirective;
|
||||
</b></pre><BR>
|
||||
<pre><b>size_t ZSTD_compress_generic (ZSTD_CCtx* cctx,
|
||||
@ -915,8 +983,8 @@ void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
|
||||
</b><p> Behave about the same as ZSTD_compressStream. To note :
|
||||
- Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_setParameter()
|
||||
- Compression parameters cannot be changed once compression is started.
|
||||
- *dstPos must be <= dstCapacity, *srcPos must be <= srcSize
|
||||
- *dspPos and *srcPos will be updated. They are guaranteed to remain below their respective limit.
|
||||
- outpot->pos must be <= dstCapacity, input->pos must be <= srcSize
|
||||
- outpot->pos and input->pos will be updated. They are guaranteed to remain below their respective limit.
|
||||
- @return provides the minimum amount of data still to flush from internal buffers
|
||||
or an error code, which can be tested using ZSTD_isError().
|
||||
if @return != 0, flush is not fully completed, there is some data left within internal buffers.
|
||||
@ -932,6 +1000,7 @@ void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
|
||||
Useful after an error, or to interrupt an ongoing compression job and start a new one.
|
||||
Any internal data not yet flushed is cancelled.
|
||||
Dictionary (if any) is dropped.
|
||||
All parameters are back to default values.
|
||||
It's possible to modify compression parameters after a reset.
|
||||
|
||||
</p></pre><BR>
|
||||
@ -943,15 +1012,163 @@ void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
|
||||
ZSTD_EndDirective endOp);
|
||||
</b><p> Same as ZSTD_compress_generic(),
|
||||
but using only integral types as arguments.
|
||||
Argument list is larger and less expressive than ZSTD_{in,out}Buffer,
|
||||
Argument list is larger than ZSTD_{in,out}Buffer,
|
||||
but can be helpful for binders from dynamic languages
|
||||
which have troubles handling structures containing memory pointers.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<a name="Chapter20"></a><h2>Block functions</h2><pre>
|
||||
Block functions produce and decode raw zstd blocks, without frame metadata.
|
||||
Frame metadata cost is typically ~18 bytes, which can be non-negligible for very small blocks (< 100 bytes).
|
||||
<pre><b>ZSTD_CCtx_params* ZSTD_createCCtxParams(void);
|
||||
</b><p> Quick howto :
|
||||
- ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
|
||||
- ZSTD_CCtxParam_setParameter() : Push parameters one by one into
|
||||
an existing ZSTD_CCtx_params structure.
|
||||
This is similar to
|
||||
ZSTD_CCtx_setParameter().
|
||||
- ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
|
||||
an existing CCtx.
|
||||
These parameters will be applied to
|
||||
all subsequent compression jobs.
|
||||
- ZSTD_compress_generic() : Do compression using the CCtx.
|
||||
- ZSTD_freeCCtxParams() : Free the memory.
|
||||
|
||||
This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams()
|
||||
for static allocation for single-threaded compression.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_resetCCtxParams(ZSTD_CCtx_params* params);
|
||||
</b><p> Reset params to default, with the default compression level.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_initCCtxParams(ZSTD_CCtx_params* cctxParams, int compressionLevel);
|
||||
</b><p> Initializes the compression parameters of cctxParams according to
|
||||
compression level. All other parameters are reset to their default values.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_initCCtxParams_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params);
|
||||
</b><p> Initializes the compression and frame parameters of cctxParams according to
|
||||
params. All other parameters are reset to their default values.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_CCtxParam_setParameter(ZSTD_CCtx_params* params, ZSTD_cParameter param, unsigned value);
|
||||
</b><p> Similar to ZSTD_CCtx_setParameter.
|
||||
Set one compression parameter, selected by enum ZSTD_cParameter.
|
||||
Parameters must be applied to a ZSTD_CCtx using ZSTD_CCtx_setParametersUsingCCtxParams().
|
||||
Note : when `value` is an enum, cast it to unsigned for proper type checking.
|
||||
@result : 0, or an error code (which can be tested with ZSTD_isError()).
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_CCtx_setParametersUsingCCtxParams(
|
||||
ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params);
|
||||
</b><p> Apply a set of ZSTD_CCtx_params to the compression context.
|
||||
This must be done before the dictionary is loaded.
|
||||
The pledgedSrcSize is treated as unknown.
|
||||
Multithreading parameters are applied only if nbThreads > 1.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<h3>Advanced parameters for decompression API</h3><pre></pre><b><pre></pre></b><BR>
|
||||
<pre><b>size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize); </b>/* not implemented */<b>
|
||||
size_t ZSTD_DCtx_loadDictionary_byReference(ZSTD_DCtx* dctx, const void* dict, size_t dictSize); </b>/* not implemented */<b>
|
||||
size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictMode_e dictMode); </b>/* not implemented */<b>
|
||||
</b><p> Create an internal DDict from dict buffer,
|
||||
to be used to decompress next frames.
|
||||
@result : 0, or an error code (which can be tested with ZSTD_isError()).
|
||||
Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
|
||||
meaning "return to no-dictionary mode".
|
||||
Note 1 : `dict` content will be copied internally.
|
||||
Use ZSTD_DCtx_loadDictionary_byReference()
|
||||
to reference dictionary content instead.
|
||||
In which case, the dictionary buffer must outlive its users.
|
||||
Note 2 : Loading a dictionary involves building tables,
|
||||
which has a non-negligible impact on CPU usage and latency.
|
||||
Note 3 : Use ZSTD_DCtx_loadDictionary_advanced() to select
|
||||
how dictionary content will be interpreted and loaded.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict); </b>/* not implemented */<b>
|
||||
</b><p> Reference a prepared dictionary, to be used to decompress next frames.
|
||||
The dictionary remains active for decompression of future frames using same DCtx.
|
||||
@result : 0, or an error code (which can be tested with ZSTD_isError()).
|
||||
Note 1 : Currently, only one dictionary can be managed.
|
||||
Referencing a new dictionary effectively "discards" any previous one.
|
||||
Special : adding a NULL DDict means "return to no-dictionary mode".
|
||||
Note 2 : DDict is just referenced, its lifetime must outlive its usage from DCtx.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize); </b>/* not implemented */<b>
|
||||
size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictMode_e dictMode); </b>/* not implemented */<b>
|
||||
</b><p> Reference a prefix (single-usage dictionary) for next compression job.
|
||||
Prefix is **only used once**. It must be explicitly referenced before each frame.
|
||||
If there is a need to use same prefix multiple times, consider embedding it into a ZSTD_DDict instead.
|
||||
@result : 0, or an error code (which can be tested with ZSTD_isError()).
|
||||
Note 1 : Adding any prefix (including NULL) invalidates any previously set prefix or dictionary
|
||||
Note 2 : Prefix buffer is referenced. It must outlive compression job.
|
||||
Note 3 : By default, the prefix is treated as raw content (ZSTD_dm_rawContent).
|
||||
Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode.
|
||||
Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize);
|
||||
</b><p> Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
|
||||
This is useful to prevent a decoder context from reserving too much memory for itself (potential attack scenario).
|
||||
This parameter is only useful in streaming mode, since no internal buffer is allocated in direct mode.
|
||||
By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_MAX)
|
||||
@return : 0, or an error code (which can be tested using ZSTD_isError()).
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format);
|
||||
</b><p> Instruct the decoder context about what kind of data to decode next.
|
||||
This instruction is mandatory to decode data without a fully-formed header,
|
||||
such ZSTD_f_zstd1_magicless for example.
|
||||
@return : 0, or an error code (which can be tested using ZSTD_isError()).
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_decompress_generic(ZSTD_DCtx* dctx,
|
||||
ZSTD_outBuffer* output,
|
||||
ZSTD_inBuffer* input);
|
||||
</b><p> Behave the same as ZSTD_decompressStream.
|
||||
Decompression parameters cannot be changed once decompression is started.
|
||||
@return : an error code, which can be tested using ZSTD_isError()
|
||||
if >0, a hint, nb of expected input bytes for next invocation.
|
||||
`0` means : a frame has just been fully decoded and flushed.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>size_t ZSTD_decompress_generic_simpleArgs (
|
||||
ZSTD_DCtx* dctx,
|
||||
void* dst, size_t dstCapacity, size_t* dstPos,
|
||||
const void* src, size_t srcSize, size_t* srcPos);
|
||||
</b><p> Same as ZSTD_decompress_generic(),
|
||||
but using only integral types as arguments.
|
||||
Argument list is larger than ZSTD_{in,out}Buffer,
|
||||
but can be helpful for binders from dynamic languages
|
||||
which have troubles handling structures containing memory pointers.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<pre><b>void ZSTD_DCtx_reset(ZSTD_DCtx* dctx);
|
||||
</b><p> Return a DCtx to clean state.
|
||||
If a decompression was ongoing, any internal data not yet flushed is cancelled.
|
||||
All parameters are back to default values, including sticky ones.
|
||||
Dictionary (if any) is dropped.
|
||||
Parameters can be modified again after a reset.
|
||||
|
||||
</p></pre><BR>
|
||||
|
||||
<a name="Chapter21"></a><h2>Block level API</h2><pre></pre>
|
||||
|
||||
<pre><b></b><p> Frame metadata cost is typically ~18 bytes, which can be non-negligible for very small blocks (< 100 bytes).
|
||||
User will have to take in charge required information to regenerate data, such as compressed and content sizes.
|
||||
|
||||
A few rules to respect :
|
||||
@ -961,7 +1178,7 @@ void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
|
||||
+ compression : any ZSTD_compressBegin*() variant, including with dictionary
|
||||
+ decompression : any ZSTD_decompressBegin*() variant, including with dictionary
|
||||
+ copyCCtx() and copyDCtx() can be used too
|
||||
- Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX
|
||||
- Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
|
||||
+ If input is larger than a block size, it's necessary to split input data into multiple blocks
|
||||
+ For inputs larger than a single block size, consider using the regular ZSTD_compress() instead.
|
||||
Frame metadata is not that costly, and quickly becomes negligible as source size grows larger.
|
||||
@ -972,12 +1189,12 @@ void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
|
||||
+ In case of multiple successive blocks, should some of them be uncompressed,
|
||||
decoder must be informed of their existence in order to follow proper history.
|
||||
Use ZSTD_insertBlock() for such a case.
|
||||
<BR></pre>
|
||||
</p></pre><BR>
|
||||
|
||||
<h3>Raw zstd block functions</h3><pre></pre><b><pre>size_t ZSTD_getBlockSize (const ZSTD_CCtx* cctx);
|
||||
size_t ZSTD_compressBlock (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
|
||||
size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
|
||||
size_t ZSTD_insertBlock(ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize); </b>/**< insert block into `dctx` history. Useful for uncompressed blocks */<b>
|
||||
size_t ZSTD_insertBlock(ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize); </b>/**< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression */<b>
|
||||
</pre></b><BR>
|
||||
</html>
|
||||
</body>
|
||||
|
1
lib/.gitignore
vendored
1
lib/.gitignore
vendored
@ -1,2 +1,3 @@
|
||||
# make install artefact
|
||||
libzstd.pc
|
||||
libzstd-nomt
|
||||
|
66
lib/Makefile
66
lib/Makefile
@ -1,13 +1,11 @@
|
||||
# ##########################################################################
|
||||
# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
|
||||
# ################################################################
|
||||
# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
|
||||
# All rights reserved.
|
||||
#
|
||||
# This Makefile is validated for Linux, macOS, *BSD, Hurd, Solaris, MSYS2 targets
|
||||
#
|
||||
# This source code is licensed under the BSD-style license found in the
|
||||
# LICENSE file in the root directory of this source tree. An additional grant
|
||||
# of patent rights can be found in the PATENTS file in the same directory.
|
||||
# ##########################################################################
|
||||
# This source code is licensed under both the BSD-style license (found in the
|
||||
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
# in the COPYING file in the root directory of this source tree).
|
||||
# ################################################################
|
||||
|
||||
# Version numbers
|
||||
LIBVER_MAJOR_SCRIPT:=`sed -n '/define ZSTD_VERSION_MAJOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < ./zstd.h`
|
||||
@ -31,7 +29,7 @@ CFLAGS += $(DEBUGFLAGS) $(MOREFLAGS)
|
||||
FLAGS = $(CPPFLAGS) $(CFLAGS)
|
||||
|
||||
|
||||
ZSTD_FILES := $(wildcard common/*.c compress/*.c decompress/*.c dictBuilder/*.c deprecated/*.c)
|
||||
ZSTD_FILES := $(sort $(wildcard common/*.c compress/*.c decompress/*.c dictBuilder/*.c deprecated/*.c))
|
||||
|
||||
ZSTD_LEGACY_SUPPORT ?= 4
|
||||
|
||||
@ -103,10 +101,19 @@ lib-release lib-release-mt: DEBUGFLAGS :=
|
||||
lib-release: lib
|
||||
lib-release-mt: lib-mt
|
||||
|
||||
# Special case : building library in single-thread mode _and_ without zstdmt_compress.c
|
||||
ZSTDMT_FILES = compress/zstdmt_compress.c
|
||||
ZSTD_NOMT_FILES = $(filter-out $(ZSTDMT_FILES),$(ZSTD_FILES))
|
||||
libzstd-nomt: LDFLAGS += -shared -fPIC -fvisibility=hidden
|
||||
libzstd-nomt: $(ZSTD_NOMT_FILES)
|
||||
@echo compiling single-thread dynamic library $(LIBVER)
|
||||
@echo files : $(ZSTD_NOMT_FILES)
|
||||
@$(CC) $(FLAGS) $^ $(LDFLAGS) $(SONAME_FLAGS) -o $@
|
||||
|
||||
clean:
|
||||
@$(RM) -r *.dSYM # Mac OS-X specific
|
||||
@$(RM) core *.o *.a *.gcda *.$(SHARED_EXT) *.$(SHARED_EXT).* libzstd.pc
|
||||
@$(RM) dll/libzstd.dll dll/libzstd.lib
|
||||
@$(RM) dll/libzstd.dll dll/libzstd.lib libzstd-nomt*
|
||||
@$(RM) common/*.o compress/*.o decompress/*.o dictBuilder/*.o legacy/*.o deprecated/*.o
|
||||
@echo Cleaning library completed
|
||||
|
||||
@ -115,16 +122,17 @@ clean:
|
||||
#-----------------------------------------------------------------------------
|
||||
ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD NetBSD DragonFly SunOS))
|
||||
|
||||
ifneq (,$(filter $(shell uname),SunOS))
|
||||
INSTALL ?= ginstall
|
||||
else
|
||||
INSTALL ?= install
|
||||
endif
|
||||
|
||||
PREFIX ?= /usr/local
|
||||
DESTDIR ?=
|
||||
LIBDIR ?= $(PREFIX)/lib
|
||||
INCLUDEDIR ?= $(PREFIX)/include
|
||||
DESTDIR ?=
|
||||
# directory variables : GNU conventions prefer lowercase
|
||||
# see https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
|
||||
# support both lower and uppercase (BSD), use uppercase in script
|
||||
prefix ?= /usr/local
|
||||
PREFIX ?= $(prefix)
|
||||
exec_prefix ?= $(PREFIX)
|
||||
libdir ?= $(exec_prefix)/lib
|
||||
LIBDIR ?= $(libdir)
|
||||
includedir ?= $(PREFIX)/include
|
||||
INCLUDEDIR ?= $(includedir)
|
||||
|
||||
ifneq (,$(filter $(shell uname),OpenBSD FreeBSD NetBSD DragonFly))
|
||||
PKGCONFIGDIR ?= $(PREFIX)/libdata/pkgconfig
|
||||
@ -132,8 +140,14 @@ else
|
||||
PKGCONFIGDIR ?= $(LIBDIR)/pkgconfig
|
||||
endif
|
||||
|
||||
INSTALL_LIB ?= $(INSTALL) -m 755
|
||||
INSTALL_DATA ?= $(INSTALL) -m 644
|
||||
ifneq (,$(filter $(shell uname),SunOS))
|
||||
INSTALL ?= ginstall
|
||||
else
|
||||
INSTALL ?= install
|
||||
endif
|
||||
|
||||
INSTALL_PROGRAM ?= $(INSTALL)
|
||||
INSTALL_DATA ?= $(INSTALL) -m 644
|
||||
|
||||
|
||||
libzstd.pc:
|
||||
@ -150,9 +164,9 @@ install: libzstd.a libzstd libzstd.pc
|
||||
@$(INSTALL_DATA) libzstd.pc $(DESTDIR)$(PKGCONFIGDIR)/
|
||||
@echo Installing libraries
|
||||
@$(INSTALL_DATA) libzstd.a $(DESTDIR)$(LIBDIR)
|
||||
@$(INSTALL_LIB) libzstd.$(SHARED_EXT_VER) $(DESTDIR)$(LIBDIR)
|
||||
@ln -sf libzstd.$(SHARED_EXT_VER) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT_MAJOR)
|
||||
@ln -sf libzstd.$(SHARED_EXT_VER) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT)
|
||||
@$(INSTALL_PROGRAM) $(LIBZSTD) $(DESTDIR)$(LIBDIR)
|
||||
@ln -sf $(LIBZSTD) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT_MAJOR)
|
||||
@ln -sf $(LIBZSTD) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT)
|
||||
@echo Installing includes
|
||||
@$(INSTALL_DATA) zstd.h $(DESTDIR)$(INCLUDEDIR)
|
||||
@$(INSTALL_DATA) common/zstd_errors.h $(DESTDIR)$(INCLUDEDIR)
|
||||
@ -164,7 +178,7 @@ uninstall:
|
||||
@$(RM) $(DESTDIR)$(LIBDIR)/libzstd.a
|
||||
@$(RM) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT)
|
||||
@$(RM) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT_MAJOR)
|
||||
@$(RM) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT_VER)
|
||||
@$(RM) $(DESTDIR)$(LIBDIR)/$(LIBZSTD)
|
||||
@$(RM) $(DESTDIR)$(PKGCONFIGDIR)/libzstd.pc
|
||||
@$(RM) $(DESTDIR)$(INCLUDEDIR)/zstd.h
|
||||
@$(RM) $(DESTDIR)$(INCLUDEDIR)/zstd_errors.h
|
||||
|
109
lib/README.md
109
lib/README.md
@ -1,57 +1,78 @@
|
||||
Zstandard library files
|
||||
================================
|
||||
|
||||
The __lib__ directory contains several directories.
|
||||
Depending on target use case, it's enough to include only files from relevant directories.
|
||||
The __lib__ directory is split into several sub-directories,
|
||||
in order to make it easier to select or exclude specific features.
|
||||
|
||||
|
||||
#### Building
|
||||
|
||||
`Makefile` script is provided, supporting the standard set of commands,
|
||||
directories, and variables (see https://www.gnu.org/prep/standards/html_node/Command-Variables.html).
|
||||
- `make` : generates both static and dynamic libraries
|
||||
- `make install` : install libraries in default system directories
|
||||
|
||||
|
||||
#### API
|
||||
|
||||
Zstandard's stable API is exposed within [zstd.h](zstd.h),
|
||||
at the root of `lib` directory.
|
||||
Zstandard's stable API is exposed within [lib/zstd.h](zstd.h).
|
||||
|
||||
|
||||
#### Advanced API
|
||||
|
||||
Some additional API may be useful if you're looking into advanced features :
|
||||
- common/error_public.h : transforms `size_t` function results into an `enum`,
|
||||
for precise error handling.
|
||||
- ZSTD_STATIC_LINKING_ONLY : if you define this macro _before_ including `zstd.h`,
|
||||
it will give access to advanced and experimental API.
|
||||
Optional advanced features are exposed via :
|
||||
|
||||
- `lib/common/zstd_errors.h` : translates `size_t` function results
|
||||
into an `ZSTD_ErrorCode`, for accurate error handling.
|
||||
- `ZSTD_STATIC_LINKING_ONLY` : if this macro is defined _before_ including `zstd.h`,
|
||||
it unlocks access to advanced experimental API,
|
||||
exposed in second part of `zstd.h`.
|
||||
These APIs shall ___never be used with dynamic library___ !
|
||||
They are not "stable", their definition may change in the future.
|
||||
Only static linking is allowed.
|
||||
|
||||
#### ZSTDMT API
|
||||
|
||||
To enable multithreaded compression within the library, invoke `make lib-mt` target.
|
||||
Prototypes are defined in header file `compress/zstdmt_compress.h`.
|
||||
When linking a program that uses ZSTDMT API against libzstd.a on a POSIX system,
|
||||
`-pthread` flag must be provided to the compiler and linker.
|
||||
Note : ZSTDMT prototypes can still be used with a library built without multithread support,
|
||||
but in this case, they will be single threaded only.
|
||||
|
||||
#### Modular build
|
||||
|
||||
Directory `common/` is required in all circumstances.
|
||||
You can select to support compression only, by just adding files from the `compress/` directory,
|
||||
In a similar way, you can build a decompressor-only library with the `decompress/` directory.
|
||||
|
||||
Other optional functionalities provided are :
|
||||
|
||||
- `dictBuilder/` : source files to create dictionaries.
|
||||
The API can be consulted in `dictBuilder/zdict.h`.
|
||||
This module also depends on `common/` and `compress/` .
|
||||
|
||||
- `legacy/` : source code to decompress previous versions of zstd, starting from `v0.1`.
|
||||
This module also depends on `common/` and `decompress/` .
|
||||
Library compilation must include directive `ZSTD_LEGACY_SUPPORT = 1` .
|
||||
The main API can be consulted in `legacy/zstd_legacy.h`.
|
||||
Advanced API from each version can be found in their relevant header file.
|
||||
For example, advanced API for version `v0.4` is in `legacy/zstd_v04.h` .
|
||||
- Directory `lib/common` is always required, for all variants.
|
||||
- Compression source code lies in `lib/compress`
|
||||
- Decompression source code lies in `lib/decompress`
|
||||
- It's possible to include only `compress` or only `decompress`, they don't depend on each other.
|
||||
- `lib/dictBuilder` : makes it possible to generate dictionaries from a set of samples.
|
||||
The API is exposed in `lib/dictBuilder/zdict.h`.
|
||||
This module depends on both `lib/common` and `lib/compress` .
|
||||
- `lib/legacy` : source code to decompress older zstd formats, starting from `v0.1`.
|
||||
This module depends on `lib/common` and `lib/decompress`.
|
||||
To enable this feature, it's necessary to define `ZSTD_LEGACY_SUPPORT = 1` during compilation.
|
||||
Typically, with `gcc`, add argument `-DZSTD_LEGACY_SUPPORT=1`.
|
||||
Using higher number limits the number of version supported.
|
||||
For example, `ZSTD_LEGACY_SUPPORT=2` means : "support legacy formats starting from v0.2+".
|
||||
The API is exposed in `lib/legacy/zstd_legacy.h`.
|
||||
Each version also provides a (dedicated) set of advanced API.
|
||||
For example, advanced API for version `v0.4` is exposed in `lib/legacy/zstd_v04.h` .
|
||||
|
||||
|
||||
#### Using MinGW+MSYS to create DLL
|
||||
#### Multithreading support
|
||||
|
||||
Multithreading is disabled by default when building with `make`.
|
||||
Enabling multithreading requires 2 conditions :
|
||||
- set macro `ZSTD_MULTITHREAD`
|
||||
- on POSIX systems : compile with pthread (`-pthread` compilation flag for `gcc` for example)
|
||||
|
||||
Both conditions are automatically triggered by invoking `make lib-mt` target.
|
||||
Note that, when linking a POSIX program with a multithreaded version of `libzstd`,
|
||||
it's necessary to trigger `-pthread` flag during link stage.
|
||||
|
||||
Multithreading capabilities are exposed via :
|
||||
- private API `lib/compress/zstdmt_compress.h`.
|
||||
Symbols defined in this header are currently exposed in `libzstd`, hence usable.
|
||||
Note however that this API is planned to be locked and remain strictly internal in the future.
|
||||
- advanced API `ZSTD_compress_generic()`, defined in `lib/zstd.h`, experimental section.
|
||||
This API is still considered experimental, but is designed to be labelled "stable" at some point in the future.
|
||||
It's the recommended entry point for multi-threading operations.
|
||||
|
||||
|
||||
#### Windows : using MinGW+MSYS to create DLL
|
||||
|
||||
DLL can be created using MinGW+MSYS with the `make libzstd` command.
|
||||
This command creates `dll\libzstd.dll` and the import library `dll\libzstd.lib`.
|
||||
@ -67,19 +88,21 @@ file it should be linked with `dll\libzstd.dll`. For example:
|
||||
The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`.
|
||||
|
||||
|
||||
#### Obsolete streaming API
|
||||
#### Deprecated API
|
||||
|
||||
Streaming is now provided within `zstd.h`.
|
||||
Older streaming API is still available within `deprecated/zbuff.h`.
|
||||
It will be removed in a future version.
|
||||
Consider migrating code towards newer streaming API in `zstd.h`.
|
||||
Obsolete API on their way out are stored in directory `lib/deprecated`.
|
||||
At this stage, it contains older streaming prototypes, in `lib/deprecated/zbuff.h`.
|
||||
Presence in this directory is temporary.
|
||||
These prototypes will be removed in some future version.
|
||||
Consider migrating code towards supported streaming API exposed in `zstd.h`.
|
||||
|
||||
|
||||
#### Miscellaneous
|
||||
|
||||
The other files are not source code. There are :
|
||||
|
||||
- LICENSE : contains the BSD license text
|
||||
- Makefile : script to compile or install zstd library (static and dynamic)
|
||||
- libzstd.pc.in : for pkg-config (`make install`)
|
||||
- README.md : this file
|
||||
- `LICENSE` : contains the BSD license text
|
||||
- `Makefile` : `make` script to build and install zstd library (static and dynamic)
|
||||
- `BUCK` : support for `buck` build system (https://buckbuild.com/)
|
||||
- `libzstd.pc.in` : for `pkg-config` (used in `make install`)
|
||||
- `README.md` : this file
|
||||
|
@ -169,33 +169,39 @@ MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits);
|
||||
****************************************************************/
|
||||
MEM_STATIC unsigned BIT_highbit32 (register U32 val)
|
||||
{
|
||||
assert(val != 0);
|
||||
{
|
||||
# if defined(_MSC_VER) /* Visual */
|
||||
unsigned long r=0;
|
||||
_BitScanReverse ( &r, val );
|
||||
return (unsigned) r;
|
||||
unsigned long r=0;
|
||||
_BitScanReverse ( &r, val );
|
||||
return (unsigned) r;
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3) /* Use GCC Intrinsic */
|
||||
return 31 - __builtin_clz (val);
|
||||
return 31 - __builtin_clz (val);
|
||||
# else /* Software version */
|
||||
static const unsigned DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29,
|
||||
11, 14, 16, 18, 22, 25, 3, 30,
|
||||
8, 12, 20, 28, 15, 17, 24, 7,
|
||||
19, 27, 23, 6, 26, 5, 4, 31 };
|
||||
U32 v = val;
|
||||
v |= v >> 1;
|
||||
v |= v >> 2;
|
||||
v |= v >> 4;
|
||||
v |= v >> 8;
|
||||
v |= v >> 16;
|
||||
return DeBruijnClz[ (U32) (v * 0x07C4ACDDU) >> 27];
|
||||
static const unsigned DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29,
|
||||
11, 14, 16, 18, 22, 25, 3, 30,
|
||||
8, 12, 20, 28, 15, 17, 24, 7,
|
||||
19, 27, 23, 6, 26, 5, 4, 31 };
|
||||
U32 v = val;
|
||||
v |= v >> 1;
|
||||
v |= v >> 2;
|
||||
v |= v >> 4;
|
||||
v |= v >> 8;
|
||||
v |= v >> 16;
|
||||
return DeBruijnClz[ (U32) (v * 0x07C4ACDDU) >> 27];
|
||||
# endif
|
||||
}
|
||||
}
|
||||
|
||||
/*===== Local Constants =====*/
|
||||
static const unsigned BIT_mask[] = { 0, 1, 3, 7, 0xF, 0x1F, 0x3F, 0x7F,
|
||||
0xFF, 0x1FF, 0x3FF, 0x7FF, 0xFFF, 0x1FFF, 0x3FFF, 0x7FFF,
|
||||
0xFFFF, 0x1FFFF, 0x3FFFF, 0x7FFFF, 0xFFFFF, 0x1FFFFF, 0x3FFFFF, 0x7FFFFF,
|
||||
0xFFFFFF, 0x1FFFFFF, 0x3FFFFFF }; /* up to 26 bits */
|
||||
|
||||
static const unsigned BIT_mask[] = {
|
||||
0, 1, 3, 7, 0xF, 0x1F,
|
||||
0x3F, 0x7F, 0xFF, 0x1FF, 0x3FF, 0x7FF,
|
||||
0xFFF, 0x1FFF, 0x3FFF, 0x7FFF, 0xFFFF, 0x1FFFF,
|
||||
0x3FFFF, 0x7FFFF, 0xFFFFF, 0x1FFFFF, 0x3FFFFF, 0x7FFFFF,
|
||||
0xFFFFFF, 0x1FFFFFF, 0x3FFFFFF, 0x7FFFFFF, 0xFFFFFFF, 0x1FFFFFFF,
|
||||
0x3FFFFFFF, 0x7FFFFFFF}; /* up to 31 bits */
|
||||
#define BIT_MASK_SIZE (sizeof(BIT_mask) / sizeof(BIT_mask[0]))
|
||||
|
||||
/*-**************************************************************
|
||||
* bitStream encoding
|
||||
@ -217,11 +223,14 @@ MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC,
|
||||
}
|
||||
|
||||
/*! BIT_addBits() :
|
||||
* can add up to 26 bits into `bitC`.
|
||||
* can add up to 31 bits into `bitC`.
|
||||
* Note : does not check for register overflow ! */
|
||||
MEM_STATIC void BIT_addBits(BIT_CStream_t* bitC,
|
||||
size_t value, unsigned nbBits)
|
||||
{
|
||||
MEM_STATIC_ASSERT(BIT_MASK_SIZE == 32);
|
||||
assert(nbBits < BIT_MASK_SIZE);
|
||||
assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8);
|
||||
bitC->bitContainer |= (value & BIT_mask[nbBits]) << bitC->bitPos;
|
||||
bitC->bitPos += nbBits;
|
||||
}
|
||||
@ -232,6 +241,7 @@ MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC,
|
||||
size_t value, unsigned nbBits)
|
||||
{
|
||||
assert((value>>nbBits) == 0);
|
||||
assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8);
|
||||
bitC->bitContainer |= value << bitC->bitPos;
|
||||
bitC->bitPos += nbBits;
|
||||
}
|
||||
@ -242,7 +252,7 @@ MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC,
|
||||
MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC)
|
||||
{
|
||||
size_t const nbBytes = bitC->bitPos >> 3;
|
||||
assert( bitC->bitPos <= (sizeof(bitC->bitContainer)*8) );
|
||||
assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
|
||||
MEM_writeLEST(bitC->ptr, bitC->bitContainer);
|
||||
bitC->ptr += nbBytes;
|
||||
assert(bitC->ptr <= bitC->endPtr);
|
||||
@ -258,7 +268,7 @@ MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC)
|
||||
MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC)
|
||||
{
|
||||
size_t const nbBytes = bitC->bitPos >> 3;
|
||||
assert( bitC->bitPos <= (sizeof(bitC->bitContainer)*8) );
|
||||
assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
|
||||
MEM_writeLEST(bitC->ptr, bitC->bitContainer);
|
||||
bitC->ptr += nbBytes;
|
||||
if (bitC->ptr > bitC->endPtr) bitC->ptr = bitC->endPtr;
|
||||
@ -350,12 +360,14 @@ MEM_STATIC size_t BIT_getMiddleBits(size_t bitContainer, U32 const start, U32 co
|
||||
# endif
|
||||
return _bextr_u32(bitContainer, start, nbBits);
|
||||
#else
|
||||
assert(nbBits < BIT_MASK_SIZE);
|
||||
return (bitContainer >> start) & BIT_mask[nbBits];
|
||||
#endif
|
||||
}
|
||||
|
||||
MEM_STATIC size_t BIT_getLowerBits(size_t bitContainer, U32 const nbBits)
|
||||
{
|
||||
assert(nbBits < BIT_MASK_SIZE);
|
||||
return bitContainer & BIT_mask[nbBits];
|
||||
}
|
||||
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef ZSTD_COMPILER_H
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
/* The purpose of this file is to have a single list of error strings embedded in binary */
|
||||
@ -29,14 +30,15 @@ const char* ERR_getErrorString(ERR_enum code)
|
||||
case PREFIX(init_missing): return "Context should be init first";
|
||||
case PREFIX(memory_allocation): return "Allocation error : not enough memory";
|
||||
case PREFIX(stage_wrong): return "Operation not authorized at current processing stage";
|
||||
case PREFIX(dstSize_tooSmall): return "Destination buffer is too small";
|
||||
case PREFIX(srcSize_wrong): return "Src size is incorrect";
|
||||
case PREFIX(tableLog_tooLarge): return "tableLog requires too much memory : unsupported";
|
||||
case PREFIX(maxSymbolValue_tooLarge): return "Unsupported max Symbol Value : too large";
|
||||
case PREFIX(maxSymbolValue_tooSmall): return "Specified maxSymbolValue is too small";
|
||||
case PREFIX(dictionary_corrupted): return "Dictionary is corrupted";
|
||||
case PREFIX(dictionary_wrong): return "Dictionary mismatch";
|
||||
case PREFIX(dictionaryCreation_failed): return "Cannot create Dictionary from provided samples";
|
||||
case PREFIX(dstSize_tooSmall): return "Destination buffer is too small";
|
||||
case PREFIX(srcSize_wrong): return "Src size is incorrect";
|
||||
/* following error codes are not stable and may be removed or changed in a future version */
|
||||
case PREFIX(frameIndex_tooLarge): return "Frame index is too large";
|
||||
case PREFIX(seekableIO): return "An I/O error occurred when reading/seeking";
|
||||
case PREFIX(maxCode):
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
/* Note : this module is expected to remain private, do not expose it */
|
||||
@ -48,10 +49,9 @@ typedef ZSTD_ErrorCode ERR_enum;
|
||||
/*-****************************************
|
||||
* Error codes handling
|
||||
******************************************/
|
||||
#ifdef ERROR
|
||||
# undef ERROR /* reported already defined on VS 2015 (Rich Geldreich) */
|
||||
#endif
|
||||
#define ERROR(name) ((size_t)-PREFIX(name))
|
||||
#undef ERROR /* reported already defined on VS 2015 (Rich Geldreich) */
|
||||
#define ERROR(name) ZSTD_ERROR(name)
|
||||
#define ZSTD_ERROR(name) ((size_t)-PREFIX(name))
|
||||
|
||||
ERR_STATIC unsigned ERR_isError(size_t code) { return (code > ERROR(maxCode)); }
|
||||
|
||||
|
@ -184,7 +184,7 @@ FSE_PUBLIC_API size_t FSE_writeNCount (void* buffer, size_t bufferSize, const sh
|
||||
/*! Constructor and Destructor of FSE_CTable.
|
||||
Note that FSE_CTable size depends on 'tableLog' and 'maxSymbolValue' */
|
||||
typedef unsigned FSE_CTable; /* don't allocate that. It's only meant to be more restrictive than void* */
|
||||
FSE_PUBLIC_API FSE_CTable* FSE_createCTable (unsigned tableLog, unsigned maxSymbolValue);
|
||||
FSE_PUBLIC_API FSE_CTable* FSE_createCTable (unsigned maxSymbolValue, unsigned tableLog);
|
||||
FSE_PUBLIC_API void FSE_freeCTable (FSE_CTable* ct);
|
||||
|
||||
/*! FSE_buildCTable():
|
||||
|
@ -242,7 +242,7 @@ size_t HUF_readStats(BYTE* huffWeight, size_t hwSize, U32* rankStats,
|
||||
|
||||
/** HUF_readCTable() :
|
||||
* Loading a CTable saved with HUF_writeCTable() */
|
||||
size_t HUF_readCTable (HUF_CElt* CTable, unsigned maxSymbolValue, const void* src, size_t srcSize);
|
||||
size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize);
|
||||
|
||||
|
||||
/*
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef MEM_H_MODULE
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
|
||||
@ -25,13 +26,14 @@
|
||||
|
||||
/* A job is a function and an opaque argument */
|
||||
typedef struct POOL_job_s {
|
||||
POOL_function function;
|
||||
void *opaque;
|
||||
POOL_function function;
|
||||
void *opaque;
|
||||
} POOL_job;
|
||||
|
||||
struct POOL_ctx_s {
|
||||
ZSTD_customMem customMem;
|
||||
/* Keep track of the threads */
|
||||
pthread_t *threads;
|
||||
ZSTD_pthread_t *threads;
|
||||
size_t numThreads;
|
||||
|
||||
/* The queue is a circular buffer */
|
||||
@ -46,11 +48,11 @@ struct POOL_ctx_s {
|
||||
int queueEmpty;
|
||||
|
||||
/* The mutex protects the queue */
|
||||
pthread_mutex_t queueMutex;
|
||||
ZSTD_pthread_mutex_t queueMutex;
|
||||
/* Condition variable for pushers to wait on when the queue is full */
|
||||
pthread_cond_t queuePushCond;
|
||||
ZSTD_pthread_cond_t queuePushCond;
|
||||
/* Condition variables for poppers to wait on when the queue is empty */
|
||||
pthread_cond_t queuePopCond;
|
||||
ZSTD_pthread_cond_t queuePopCond;
|
||||
/* Indicates if the queue is shutting down */
|
||||
int shutdown;
|
||||
};
|
||||
@ -65,14 +67,14 @@ static void* POOL_thread(void* opaque) {
|
||||
if (!ctx) { return NULL; }
|
||||
for (;;) {
|
||||
/* Lock the mutex and wait for a non-empty queue or until shutdown */
|
||||
pthread_mutex_lock(&ctx->queueMutex);
|
||||
ZSTD_pthread_mutex_lock(&ctx->queueMutex);
|
||||
|
||||
while (ctx->queueEmpty && !ctx->shutdown) {
|
||||
pthread_cond_wait(&ctx->queuePopCond, &ctx->queueMutex);
|
||||
ZSTD_pthread_cond_wait(&ctx->queuePopCond, &ctx->queueMutex);
|
||||
}
|
||||
/* empty => shutting down: so stop */
|
||||
if (ctx->queueEmpty) {
|
||||
pthread_mutex_unlock(&ctx->queueMutex);
|
||||
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
|
||||
return opaque;
|
||||
}
|
||||
/* Pop a job off the queue */
|
||||
@ -81,28 +83,32 @@ static void* POOL_thread(void* opaque) {
|
||||
ctx->numThreadsBusy++;
|
||||
ctx->queueEmpty = ctx->queueHead == ctx->queueTail;
|
||||
/* Unlock the mutex, signal a pusher, and run the job */
|
||||
pthread_mutex_unlock(&ctx->queueMutex);
|
||||
pthread_cond_signal(&ctx->queuePushCond);
|
||||
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
|
||||
ZSTD_pthread_cond_signal(&ctx->queuePushCond);
|
||||
|
||||
job.function(job.opaque);
|
||||
|
||||
/* If the intended queue size was 0, signal after finishing job */
|
||||
if (ctx->queueSize == 1) {
|
||||
pthread_mutex_lock(&ctx->queueMutex);
|
||||
ZSTD_pthread_mutex_lock(&ctx->queueMutex);
|
||||
ctx->numThreadsBusy--;
|
||||
pthread_mutex_unlock(&ctx->queueMutex);
|
||||
pthread_cond_signal(&ctx->queuePushCond);
|
||||
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
|
||||
ZSTD_pthread_cond_signal(&ctx->queuePushCond);
|
||||
} }
|
||||
} /* for (;;) */
|
||||
/* Unreachable */
|
||||
}
|
||||
|
||||
POOL_ctx *POOL_create(size_t numThreads, size_t queueSize) {
|
||||
POOL_ctx *ctx;
|
||||
POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) {
|
||||
return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
|
||||
}
|
||||
|
||||
POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem) {
|
||||
POOL_ctx* ctx;
|
||||
/* Check the parameters */
|
||||
if (!numThreads) { return NULL; }
|
||||
/* Allocate the context and zero initialize */
|
||||
ctx = (POOL_ctx *)calloc(1, sizeof(POOL_ctx));
|
||||
ctx = (POOL_ctx*)ZSTD_calloc(sizeof(POOL_ctx), customMem);
|
||||
if (!ctx) { return NULL; }
|
||||
/* Initialize the job queue.
|
||||
* It needs one extra space since one space is wasted to differentiate empty
|
||||
@ -114,19 +120,20 @@ POOL_ctx *POOL_create(size_t numThreads, size_t queueSize) {
|
||||
ctx->queueTail = 0;
|
||||
ctx->numThreadsBusy = 0;
|
||||
ctx->queueEmpty = 1;
|
||||
(void)pthread_mutex_init(&ctx->queueMutex, NULL);
|
||||
(void)pthread_cond_init(&ctx->queuePushCond, NULL);
|
||||
(void)pthread_cond_init(&ctx->queuePopCond, NULL);
|
||||
(void)ZSTD_pthread_mutex_init(&ctx->queueMutex, NULL);
|
||||
(void)ZSTD_pthread_cond_init(&ctx->queuePushCond, NULL);
|
||||
(void)ZSTD_pthread_cond_init(&ctx->queuePopCond, NULL);
|
||||
ctx->shutdown = 0;
|
||||
/* Allocate space for the thread handles */
|
||||
ctx->threads = (pthread_t*)malloc(numThreads * sizeof(pthread_t));
|
||||
ctx->threads = (ZSTD_pthread_t*)ZSTD_malloc(numThreads * sizeof(ZSTD_pthread_t), customMem);
|
||||
ctx->numThreads = 0;
|
||||
ctx->customMem = customMem;
|
||||
/* Check for errors */
|
||||
if (!ctx->threads || !ctx->queue) { POOL_free(ctx); return NULL; }
|
||||
/* Initialize the threads */
|
||||
{ size_t i;
|
||||
for (i = 0; i < numThreads; ++i) {
|
||||
if (pthread_create(&ctx->threads[i], NULL, &POOL_thread, ctx)) {
|
||||
if (ZSTD_pthread_create(&ctx->threads[i], NULL, &POOL_thread, ctx)) {
|
||||
ctx->numThreads = i;
|
||||
POOL_free(ctx);
|
||||
return NULL;
|
||||
@ -139,37 +146,37 @@ POOL_ctx *POOL_create(size_t numThreads, size_t queueSize) {
|
||||
/*! POOL_join() :
|
||||
Shutdown the queue, wake any sleeping threads, and join all of the threads.
|
||||
*/
|
||||
static void POOL_join(POOL_ctx *ctx) {
|
||||
static void POOL_join(POOL_ctx* ctx) {
|
||||
/* Shut down the queue */
|
||||
pthread_mutex_lock(&ctx->queueMutex);
|
||||
ZSTD_pthread_mutex_lock(&ctx->queueMutex);
|
||||
ctx->shutdown = 1;
|
||||
pthread_mutex_unlock(&ctx->queueMutex);
|
||||
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
|
||||
/* Wake up sleeping threads */
|
||||
pthread_cond_broadcast(&ctx->queuePushCond);
|
||||
pthread_cond_broadcast(&ctx->queuePopCond);
|
||||
ZSTD_pthread_cond_broadcast(&ctx->queuePushCond);
|
||||
ZSTD_pthread_cond_broadcast(&ctx->queuePopCond);
|
||||
/* Join all of the threads */
|
||||
{ size_t i;
|
||||
for (i = 0; i < ctx->numThreads; ++i) {
|
||||
pthread_join(ctx->threads[i], NULL);
|
||||
ZSTD_pthread_join(ctx->threads[i], NULL);
|
||||
} }
|
||||
}
|
||||
|
||||
void POOL_free(POOL_ctx *ctx) {
|
||||
if (!ctx) { return; }
|
||||
POOL_join(ctx);
|
||||
pthread_mutex_destroy(&ctx->queueMutex);
|
||||
pthread_cond_destroy(&ctx->queuePushCond);
|
||||
pthread_cond_destroy(&ctx->queuePopCond);
|
||||
if (ctx->queue) free(ctx->queue);
|
||||
if (ctx->threads) free(ctx->threads);
|
||||
free(ctx);
|
||||
ZSTD_pthread_mutex_destroy(&ctx->queueMutex);
|
||||
ZSTD_pthread_cond_destroy(&ctx->queuePushCond);
|
||||
ZSTD_pthread_cond_destroy(&ctx->queuePopCond);
|
||||
ZSTD_free(ctx->queue, ctx->customMem);
|
||||
ZSTD_free(ctx->threads, ctx->customMem);
|
||||
ZSTD_free(ctx, ctx->customMem);
|
||||
}
|
||||
|
||||
size_t POOL_sizeof(POOL_ctx *ctx) {
|
||||
if (ctx==NULL) return 0; /* supports sizeof NULL */
|
||||
return sizeof(*ctx)
|
||||
+ ctx->queueSize * sizeof(POOL_job)
|
||||
+ ctx->numThreads * sizeof(pthread_t);
|
||||
+ ctx->numThreads * sizeof(ZSTD_pthread_t);
|
||||
}
|
||||
|
||||
/**
|
||||
@ -191,12 +198,12 @@ void POOL_add(void* ctxVoid, POOL_function function, void *opaque) {
|
||||
POOL_ctx* const ctx = (POOL_ctx*)ctxVoid;
|
||||
if (!ctx) { return; }
|
||||
|
||||
pthread_mutex_lock(&ctx->queueMutex);
|
||||
ZSTD_pthread_mutex_lock(&ctx->queueMutex);
|
||||
{ POOL_job const job = {function, opaque};
|
||||
|
||||
/* Wait until there is space in the queue for the new job */
|
||||
while (isQueueFull(ctx) && !ctx->shutdown) {
|
||||
pthread_cond_wait(&ctx->queuePushCond, &ctx->queueMutex);
|
||||
ZSTD_pthread_cond_wait(&ctx->queuePushCond, &ctx->queueMutex);
|
||||
}
|
||||
/* The queue is still going => there is space */
|
||||
if (!ctx->shutdown) {
|
||||
@ -205,8 +212,8 @@ void POOL_add(void* ctxVoid, POOL_function function, void *opaque) {
|
||||
ctx->queueTail = (ctx->queueTail + 1) % ctx->queueSize;
|
||||
}
|
||||
}
|
||||
pthread_mutex_unlock(&ctx->queueMutex);
|
||||
pthread_cond_signal(&ctx->queuePopCond);
|
||||
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
|
||||
ZSTD_pthread_cond_signal(&ctx->queuePopCond);
|
||||
}
|
||||
|
||||
#else /* ZSTD_MULTITHREAD not defined */
|
||||
@ -214,26 +221,34 @@ void POOL_add(void* ctxVoid, POOL_function function, void *opaque) {
|
||||
|
||||
/* We don't need any data, but if it is empty malloc() might return NULL. */
|
||||
struct POOL_ctx_s {
|
||||
int data;
|
||||
int dummy;
|
||||
};
|
||||
static POOL_ctx g_ctx;
|
||||
|
||||
POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) {
|
||||
(void)numThreads;
|
||||
(void)queueSize;
|
||||
return (POOL_ctx*)malloc(sizeof(POOL_ctx));
|
||||
return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
|
||||
}
|
||||
|
||||
POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem) {
|
||||
(void)numThreads;
|
||||
(void)queueSize;
|
||||
(void)customMem;
|
||||
return &g_ctx;
|
||||
}
|
||||
|
||||
void POOL_free(POOL_ctx* ctx) {
|
||||
free(ctx);
|
||||
assert(!ctx || ctx == &g_ctx);
|
||||
(void)ctx;
|
||||
}
|
||||
|
||||
void POOL_add(void* ctx, POOL_function function, void* opaque) {
|
||||
(void)ctx;
|
||||
function(opaque);
|
||||
(void)ctx;
|
||||
function(opaque);
|
||||
}
|
||||
|
||||
size_t POOL_sizeof(POOL_ctx* ctx) {
|
||||
if (ctx==NULL) return 0; /* supports sizeof NULL */
|
||||
assert(ctx == &g_ctx);
|
||||
return sizeof(*ctx);
|
||||
}
|
||||
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef POOL_H
|
||||
@ -16,6 +17,7 @@ extern "C" {
|
||||
|
||||
|
||||
#include <stddef.h> /* size_t */
|
||||
#include "zstd_internal.h" /* ZSTD_customMem */
|
||||
|
||||
typedef struct POOL_ctx_s POOL_ctx;
|
||||
|
||||
@ -27,6 +29,8 @@ typedef struct POOL_ctx_s POOL_ctx;
|
||||
*/
|
||||
POOL_ctx *POOL_create(size_t numThreads, size_t queueSize);
|
||||
|
||||
POOL_ctx *POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem);
|
||||
|
||||
/*! POOL_free() :
|
||||
Free a thread pool returned by POOL_create().
|
||||
*/
|
||||
|
@ -2,9 +2,9 @@
|
||||
* Copyright (c) 2016 Tino Reichardt
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*
|
||||
* You can contact the author at:
|
||||
* - zstdmt source repository: https://github.com/mcmilk/zstdmt
|
||||
@ -14,12 +14,8 @@
|
||||
* This file will hold wrapper for systems, which do not support pthreads
|
||||
*/
|
||||
|
||||
/* When ZSTD_MULTITHREAD is not defined, this file would become an empty translation unit.
|
||||
* Include some ISO C header code to prevent this and portably avoid related warnings.
|
||||
* (Visual C++: C4206 / GCC: -Wpedantic / Clang: -Wempty-translation-unit)
|
||||
*/
|
||||
#include <stddef.h>
|
||||
|
||||
/* create fake symbol to avoid empty trnaslation unit warning */
|
||||
int g_ZSTD_threading_useles_symbol;
|
||||
|
||||
#if defined(ZSTD_MULTITHREAD) && defined(_WIN32)
|
||||
|
||||
@ -39,12 +35,12 @@
|
||||
|
||||
static unsigned __stdcall worker(void *arg)
|
||||
{
|
||||
pthread_t* const thread = (pthread_t*) arg;
|
||||
ZSTD_pthread_t* const thread = (ZSTD_pthread_t*) arg;
|
||||
thread->arg = thread->start_routine(thread->arg);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int pthread_create(pthread_t* thread, const void* unused,
|
||||
int ZSTD_pthread_create(ZSTD_pthread_t* thread, const void* unused,
|
||||
void* (*start_routine) (void*), void* arg)
|
||||
{
|
||||
(void)unused;
|
||||
@ -58,16 +54,16 @@ int pthread_create(pthread_t* thread, const void* unused,
|
||||
return 0;
|
||||
}
|
||||
|
||||
int _pthread_join(pthread_t * thread, void **value_ptr)
|
||||
int ZSTD_pthread_join(ZSTD_pthread_t thread, void **value_ptr)
|
||||
{
|
||||
DWORD result;
|
||||
|
||||
if (!thread->handle) return 0;
|
||||
if (!thread.handle) return 0;
|
||||
|
||||
result = WaitForSingleObject(thread->handle, INFINITE);
|
||||
result = WaitForSingleObject(thread.handle, INFINITE);
|
||||
switch (result) {
|
||||
case WAIT_OBJECT_0:
|
||||
if (value_ptr) *value_ptr = thread->arg;
|
||||
if (value_ptr) *value_ptr = thread.arg;
|
||||
return 0;
|
||||
case WAIT_ABANDONED:
|
||||
return EINVAL;
|
||||
|
@ -2,9 +2,9 @@
|
||||
* Copyright (c) 2016 Tino Reichardt
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under the BSD-style license found in the
|
||||
* LICENSE file in the root directory of this source tree. An additional grant
|
||||
* of patent rights can be found in the PATENTS file in the same directory.
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*
|
||||
* You can contact the author at:
|
||||
* - zstdmt source repository: https://github.com/mcmilk/zstdmt
|
||||
@ -37,35 +37,38 @@ extern "C" {
|
||||
# define WIN32_LEAN_AND_MEAN
|
||||
#endif
|
||||
|
||||
#undef ERROR /* reported already defined on VS 2015 (Rich Geldreich) */
|
||||
#include <windows.h>
|
||||
#undef ERROR
|
||||
#define ERROR(name) ZSTD_ERROR(name)
|
||||
|
||||
|
||||
/* mutex */
|
||||
#define pthread_mutex_t CRITICAL_SECTION
|
||||
#define pthread_mutex_init(a,b) (InitializeCriticalSection((a)), 0)
|
||||
#define pthread_mutex_destroy(a) DeleteCriticalSection((a))
|
||||
#define pthread_mutex_lock(a) EnterCriticalSection((a))
|
||||
#define pthread_mutex_unlock(a) LeaveCriticalSection((a))
|
||||
#define ZSTD_pthread_mutex_t CRITICAL_SECTION
|
||||
#define ZSTD_pthread_mutex_init(a, b) (InitializeCriticalSection((a)), 0)
|
||||
#define ZSTD_pthread_mutex_destroy(a) DeleteCriticalSection((a))
|
||||
#define ZSTD_pthread_mutex_lock(a) EnterCriticalSection((a))
|
||||
#define ZSTD_pthread_mutex_unlock(a) LeaveCriticalSection((a))
|
||||
|
||||
/* condition variable */
|
||||
#define pthread_cond_t CONDITION_VARIABLE
|
||||
#define pthread_cond_init(a, b) (InitializeConditionVariable((a)), 0)
|
||||
#define pthread_cond_destroy(a) /* No delete */
|
||||
#define pthread_cond_wait(a, b) SleepConditionVariableCS((a), (b), INFINITE)
|
||||
#define pthread_cond_signal(a) WakeConditionVariable((a))
|
||||
#define pthread_cond_broadcast(a) WakeAllConditionVariable((a))
|
||||
#define ZSTD_pthread_cond_t CONDITION_VARIABLE
|
||||
#define ZSTD_pthread_cond_init(a, b) (InitializeConditionVariable((a)), 0)
|
||||
#define ZSTD_pthread_cond_destroy(a) /* No delete */
|
||||
#define ZSTD_pthread_cond_wait(a, b) SleepConditionVariableCS((a), (b), INFINITE)
|
||||
#define ZSTD_pthread_cond_signal(a) WakeConditionVariable((a))
|
||||
#define ZSTD_pthread_cond_broadcast(a) WakeAllConditionVariable((a))
|
||||
|
||||
/* pthread_create() and pthread_join() */
|
||||
/* ZSTD_pthread_create() and ZSTD_pthread_join() */
|
||||
typedef struct {
|
||||
HANDLE handle;
|
||||
void* (*start_routine)(void*);
|
||||
void* arg;
|
||||
} pthread_t;
|
||||
} ZSTD_pthread_t;
|
||||
|
||||
int pthread_create(pthread_t* thread, const void* unused,
|
||||
int ZSTD_pthread_create(ZSTD_pthread_t* thread, const void* unused,
|
||||
void* (*start_routine) (void*), void* arg);
|
||||
|
||||
#define pthread_join(a, b) _pthread_join(&(a), (b))
|
||||
int _pthread_join(pthread_t* thread, void** value_ptr);
|
||||
int ZSTD_pthread_join(ZSTD_pthread_t thread, void** value_ptr);
|
||||
|
||||
/**
|
||||
* add here more wrappers as required
|
||||
@ -76,23 +79,40 @@ int _pthread_join(pthread_t* thread, void** value_ptr);
|
||||
/* === POSIX Systems === */
|
||||
# include <pthread.h>
|
||||
|
||||
#define ZSTD_pthread_mutex_t pthread_mutex_t
|
||||
#define ZSTD_pthread_mutex_init(a, b) pthread_mutex_init((a), (b))
|
||||
#define ZSTD_pthread_mutex_destroy(a) pthread_mutex_destroy((a))
|
||||
#define ZSTD_pthread_mutex_lock(a) pthread_mutex_lock((a))
|
||||
#define ZSTD_pthread_mutex_unlock(a) pthread_mutex_unlock((a))
|
||||
|
||||
#define ZSTD_pthread_cond_t pthread_cond_t
|
||||
#define ZSTD_pthread_cond_init(a, b) pthread_cond_init((a), (b))
|
||||
#define ZSTD_pthread_cond_destroy(a) pthread_cond_destroy((a))
|
||||
#define ZSTD_pthread_cond_wait(a, b) pthread_cond_wait((a), (b))
|
||||
#define ZSTD_pthread_cond_signal(a) pthread_cond_signal((a))
|
||||
#define ZSTD_pthread_cond_broadcast(a) pthread_cond_broadcast((a))
|
||||
|
||||
#define ZSTD_pthread_t pthread_t
|
||||
#define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d))
|
||||
#define ZSTD_pthread_join(a, b) pthread_join((a),(b))
|
||||
|
||||
#else /* ZSTD_MULTITHREAD not defined */
|
||||
/* No multithreading support */
|
||||
|
||||
#define pthread_mutex_t int /* #define rather than typedef, because sometimes pthread support is implicit, resulting in duplicated symbols */
|
||||
#define pthread_mutex_init(a,b) ((void)a, 0)
|
||||
#define pthread_mutex_destroy(a)
|
||||
#define pthread_mutex_lock(a)
|
||||
#define pthread_mutex_unlock(a)
|
||||
typedef int ZSTD_pthread_mutex_t;
|
||||
#define ZSTD_pthread_mutex_init(a, b) ((void)a, 0)
|
||||
#define ZSTD_pthread_mutex_destroy(a)
|
||||
#define ZSTD_pthread_mutex_lock(a)
|
||||
#define ZSTD_pthread_mutex_unlock(a)
|
||||
|
||||
#define pthread_cond_t int
|
||||
#define pthread_cond_init(a,b) ((void)a, 0)
|
||||
#define pthread_cond_destroy(a)
|
||||
#define pthread_cond_wait(a,b)
|
||||
#define pthread_cond_signal(a)
|
||||
#define pthread_cond_broadcast(a)
|
||||
typedef int ZSTD_pthread_cond_t;
|
||||
#define ZSTD_pthread_cond_init(a, b) ((void)a, 0)
|
||||
#define ZSTD_pthread_cond_destroy(a)
|
||||
#define ZSTD_pthread_cond_wait(a, b)
|
||||
#define ZSTD_pthread_cond_signal(a)
|
||||
#define ZSTD_pthread_cond_broadcast(a)
|
||||
|
||||
/* do not use pthread_t */
|
||||
/* do not use ZSTD_pthread_t */
|
||||
|
||||
#endif /* ZSTD_MULTITHREAD */
|
||||
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
|
||||
@ -15,8 +16,7 @@
|
||||
#include <stdlib.h> /* malloc, calloc, free */
|
||||
#include <string.h> /* memset */
|
||||
#include "error_private.h"
|
||||
#define ZSTD_STATIC_LINKING_ONLY
|
||||
#include "zstd.h"
|
||||
#include "zstd_internal.h"
|
||||
|
||||
|
||||
/*-****************************************
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef ZSTD_ERRORS_H_398273423
|
||||
@ -62,9 +63,10 @@ typedef enum {
|
||||
ZSTD_error_memory_allocation = 64,
|
||||
ZSTD_error_dstSize_tooSmall = 70,
|
||||
ZSTD_error_srcSize_wrong = 72,
|
||||
/* following error codes are not stable and may be removed or changed in a future version */
|
||||
ZSTD_error_frameIndex_tooLarge = 100,
|
||||
ZSTD_error_seekableIO = 102,
|
||||
ZSTD_error_maxCode = 120 /* never EVER use this value directly, it may change in future versions! Use ZSTD_isError() instead */
|
||||
ZSTD_error_maxCode = 120 /* never EVER use this value directly, it can change in future versions! Use ZSTD_isError() instead */
|
||||
} ZSTD_ErrorCode;
|
||||
|
||||
/*! ZSTD_getErrorCode() :
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef ZSTD_CCOMMON_H_MODULE
|
||||
@ -29,6 +30,11 @@
|
||||
#include "xxhash.h" /* XXH_reset, update, digest */
|
||||
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
|
||||
/*-*************************************
|
||||
* Debug
|
||||
***************************************/
|
||||
@ -96,9 +102,13 @@ static const U32 repStartValue[ZSTD_REP_NUM] = { 1, 4, 8 };
|
||||
#define BIT0 1
|
||||
|
||||
#define ZSTD_WINDOWLOG_ABSOLUTEMIN 10
|
||||
#define ZSTD_WINDOWLOG_DEFAULTMAX 27 /* Default maximum allowed window log */
|
||||
static const size_t ZSTD_fcs_fieldSize[4] = { 0, 2, 4, 8 };
|
||||
static const size_t ZSTD_did_fieldSize[4] = { 0, 1, 2, 4 };
|
||||
|
||||
#define ZSTD_FRAMEIDSIZE 4
|
||||
static const size_t ZSTD_frameIdSize = ZSTD_FRAMEIDSIZE; /* magic number size */
|
||||
|
||||
#define ZSTD_BLOCKHEADERSIZE 3 /* C standard doesn't allow `static const` variable to be init using another `static const` variable */
|
||||
static const size_t ZSTD_blockHeaderSize = ZSTD_BLOCKHEADERSIZE;
|
||||
typedef enum { bt_raw, bt_rle, bt_compressed, bt_reserved } blockType_e;
|
||||
@ -117,7 +127,8 @@ typedef enum { set_basic, set_rle, set_compressed, set_repeat } symbolEncodingTy
|
||||
#define MaxLit ((1<<Litbits) - 1)
|
||||
#define MaxML 52
|
||||
#define MaxLL 35
|
||||
#define MaxOff 28
|
||||
#define DefaultMaxOff 28
|
||||
#define MaxOff 31
|
||||
#define MaxSeq MAX(MaxLL, MaxML) /* Assumption : MaxOff < MaxLL,MaxML */
|
||||
#define MLFSELog 9
|
||||
#define LLFSELog 9
|
||||
@ -143,8 +154,8 @@ static const S16 ML_defaultNorm[MaxML+1] = { 1, 4, 3, 2, 2, 2, 2, 2, 2, 1, 1, 1,
|
||||
#define ML_DEFAULTNORMLOG 6 /* for static allocation */
|
||||
static const U32 ML_defaultNormLog = ML_DEFAULTNORMLOG;
|
||||
|
||||
static const S16 OF_defaultNorm[MaxOff+1] = { 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1,
|
||||
1, 1, 1, 1, 1, 1, 1, 1,-1,-1,-1,-1,-1 };
|
||||
static const S16 OF_defaultNorm[DefaultMaxOff+1] = { 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1,
|
||||
1, 1, 1, 1, 1, 1, 1, 1,-1,-1,-1,-1,-1 };
|
||||
#define OF_DEFAULTNORMLOG 5 /* for static allocation */
|
||||
static const U32 OF_defaultNormLog = OF_DEFAULTNORMLOG;
|
||||
|
||||
@ -243,6 +254,26 @@ typedef struct {
|
||||
const BYTE* cachedLiterals;
|
||||
} optState_t;
|
||||
|
||||
typedef struct {
|
||||
U32 offset;
|
||||
U32 checksum;
|
||||
} ldmEntry_t;
|
||||
|
||||
typedef struct {
|
||||
ldmEntry_t* hashTable;
|
||||
BYTE* bucketOffsets; /* Next position in bucket to insert entry */
|
||||
U64 hashPower; /* Used to compute the rolling hash.
|
||||
* Depends on ldmParams.minMatchLength */
|
||||
} ldmState_t;
|
||||
|
||||
typedef struct {
|
||||
U32 enableLdm; /* 1 if enable long distance matching */
|
||||
U32 hashLog; /* Log size of hashTable */
|
||||
U32 bucketSizeLog; /* Log bucket size for collision resolution, at most 8 */
|
||||
U32 minMatchLength; /* Minimum match length */
|
||||
U32 hashEveryLog; /* Log number of entries to skip */
|
||||
} ldmParams_t;
|
||||
|
||||
typedef struct {
|
||||
U32 hufCTable[HUF_CTABLE_SIZE_U32(255)];
|
||||
FSE_CTable offcodeCTable[FSE_CTABLE_SIZE_U32(OffFSELog, MaxOff)];
|
||||
@ -255,6 +286,28 @@ typedef struct {
|
||||
FSE_repeat litlength_repeatMode;
|
||||
} ZSTD_entropyCTables_t;
|
||||
|
||||
struct ZSTD_CCtx_params_s {
|
||||
ZSTD_format_e format;
|
||||
ZSTD_compressionParameters cParams;
|
||||
ZSTD_frameParameters fParams;
|
||||
|
||||
int compressionLevel;
|
||||
U32 forceWindow; /* force back-references to respect limit of
|
||||
* 1<<wLog, even for dictionary */
|
||||
|
||||
/* Multithreading: used to pass parameters to mtctx */
|
||||
U32 nbThreads;
|
||||
unsigned jobSize;
|
||||
unsigned overlapSizeLog;
|
||||
|
||||
/* Long distance matching parameters */
|
||||
ldmParams_t ldmParams;
|
||||
|
||||
/* For use with createCCtxParams() and freeCCtxParams() only */
|
||||
ZSTD_customMem customMem;
|
||||
|
||||
}; /* typedef'd to ZSTD_CCtx_params within "zstd.h" */
|
||||
|
||||
const seqStore_t* ZSTD_getSeqStore(const ZSTD_CCtx* ctx);
|
||||
void ZSTD_seqToCodes(const seqStore_t* seqStorePtr);
|
||||
|
||||
@ -268,24 +321,27 @@ void ZSTD_free(void* ptr, ZSTD_customMem customMem);
|
||||
|
||||
MEM_STATIC U32 ZSTD_highbit32(U32 val)
|
||||
{
|
||||
assert(val != 0);
|
||||
{
|
||||
# if defined(_MSC_VER) /* Visual */
|
||||
unsigned long r=0;
|
||||
_BitScanReverse(&r, val);
|
||||
return (unsigned)r;
|
||||
unsigned long r=0;
|
||||
_BitScanReverse(&r, val);
|
||||
return (unsigned)r;
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3) /* GCC Intrinsic */
|
||||
return 31 - __builtin_clz(val);
|
||||
return 31 - __builtin_clz(val);
|
||||
# else /* Software version */
|
||||
static const int DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, 8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31 };
|
||||
U32 v = val;
|
||||
int r;
|
||||
v |= v >> 1;
|
||||
v |= v >> 2;
|
||||
v |= v >> 4;
|
||||
v |= v >> 8;
|
||||
v |= v >> 16;
|
||||
r = DeBruijnClz[(U32)(v * 0x07C4ACDDU) >> 27];
|
||||
return r;
|
||||
static const int DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, 8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31 };
|
||||
U32 v = val;
|
||||
int r;
|
||||
v |= v >> 1;
|
||||
v |= v >> 2;
|
||||
v |= v >> 4;
|
||||
v |= v >> 8;
|
||||
v |= v >> 16;
|
||||
r = DeBruijnClz[(U32)(v * 0x07C4ACDDU) >> 27];
|
||||
return r;
|
||||
# endif
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -306,7 +362,7 @@ void ZSTD_invalidateRepCodes(ZSTD_CCtx* cctx);
|
||||
size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs,
|
||||
const void* dict, size_t dictSize,
|
||||
const ZSTD_CDict* cdict,
|
||||
ZSTD_parameters params, unsigned long long pledgedSrcSize);
|
||||
ZSTD_CCtx_params params, unsigned long long pledgedSrcSize);
|
||||
|
||||
/*! ZSTD_compressStream_generic() :
|
||||
* Private use only. To be called from zstdmt_compress.c in single-thread mode. */
|
||||
@ -315,10 +371,25 @@ size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs,
|
||||
ZSTD_inBuffer* input,
|
||||
ZSTD_EndDirective const flushMode);
|
||||
|
||||
/*! ZSTD_getParamsFromCDict() :
|
||||
/*! ZSTD_getCParamsFromCDict() :
|
||||
* as the name implies */
|
||||
ZSTD_parameters ZSTD_getParamsFromCDict(const ZSTD_CDict* cdict);
|
||||
ZSTD_compressionParameters ZSTD_getCParamsFromCDict(const ZSTD_CDict* cdict);
|
||||
|
||||
/* ZSTD_compressBegin_advanced_internal() :
|
||||
* Private use only. To be called from zstdmt_compress.c. */
|
||||
size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx,
|
||||
const void* dict, size_t dictSize,
|
||||
ZSTD_dictMode_e dictMode,
|
||||
ZSTD_CCtx_params params,
|
||||
unsigned long long pledgedSrcSize);
|
||||
|
||||
/* ZSTD_compress_advanced_internal() :
|
||||
* Private use only. To be called from zstdmt_compress.c. */
|
||||
size_t ZSTD_compress_advanced_internal(ZSTD_CCtx* cctx,
|
||||
void* dst, size_t dstCapacity,
|
||||
const void* src, size_t srcSize,
|
||||
const void* dict,size_t dictSize,
|
||||
ZSTD_CCtx_params params);
|
||||
|
||||
typedef struct {
|
||||
blockType_e blockType;
|
||||
@ -331,5 +402,8 @@ typedef struct {
|
||||
size_t ZSTD_getcBlockSize(const void* src, size_t srcSize,
|
||||
blockProperties_t* bpPtr);
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* ZSTD_CCOMMON_H_MODULE */
|
||||
|
@ -461,6 +461,7 @@ static unsigned FSE_minTableLog(size_t srcSize, unsigned maxSymbolValue)
|
||||
U32 minBitsSrc = BIT_highbit32((U32)(srcSize - 1)) + 1;
|
||||
U32 minBitsSymbols = BIT_highbit32(maxSymbolValue) + 2;
|
||||
U32 minBits = minBitsSrc < minBitsSymbols ? minBitsSrc : minBitsSymbols;
|
||||
assert(srcSize > 1); /* Not supported, RLE should be used instead */
|
||||
return minBits;
|
||||
}
|
||||
|
||||
@ -469,6 +470,7 @@ unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsi
|
||||
U32 maxBitsSrc = BIT_highbit32((U32)(srcSize - 1)) - minus;
|
||||
U32 tableLog = maxTableLog;
|
||||
U32 minBits = FSE_minTableLog(srcSize, maxSymbolValue);
|
||||
assert(srcSize > 1); /* Not supported, RLE should be used instead */
|
||||
if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG;
|
||||
if (maxBitsSrc < tableLog) tableLog = maxBitsSrc; /* Accuracy can be reduced */
|
||||
if (minBits > tableLog) tableLog = minBits; /* Need a minimum to safely represent all symbol values */
|
||||
@ -580,7 +582,7 @@ size_t FSE_normalizeCount (short* normalizedCounter, unsigned tableLog,
|
||||
if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge); /* Unsupported size */
|
||||
if (tableLog < FSE_minTableLog(total, maxSymbolValue)) return ERROR(GENERIC); /* Too small tableLog, compression potentially impossible */
|
||||
|
||||
{ U32 const rtbTable[] = { 0, 473195, 504333, 520860, 550000, 700000, 750000, 830000 };
|
||||
{ static U32 const rtbTable[] = { 0, 473195, 504333, 520860, 550000, 700000, 750000, 830000 };
|
||||
U64 const scale = 62 - tableLog;
|
||||
U64 const step = ((U64)1<<62) / total; /* <== here, one division ! */
|
||||
U64 const vStep = 1ULL<<(scale-20);
|
||||
|
@ -167,7 +167,7 @@ size_t HUF_writeCTable (void* dst, size_t maxDstSize,
|
||||
}
|
||||
|
||||
|
||||
size_t HUF_readCTable (HUF_CElt* CTable, U32 maxSymbolValue, const void* src, size_t srcSize)
|
||||
size_t HUF_readCTable (HUF_CElt* CTable, U32* maxSymbolValuePtr, const void* src, size_t srcSize)
|
||||
{
|
||||
BYTE huffWeight[HUF_SYMBOLVALUE_MAX + 1]; /* init not required, even though some static analyzer may complain */
|
||||
U32 rankVal[HUF_TABLELOG_ABSOLUTEMAX + 1]; /* large enough for values from 0 to 16 */
|
||||
@ -179,7 +179,7 @@ size_t HUF_readCTable (HUF_CElt* CTable, U32 maxSymbolValue, const void* src, si
|
||||
|
||||
/* check result */
|
||||
if (tableLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge);
|
||||
if (nbSymbols > maxSymbolValue+1) return ERROR(maxSymbolValue_tooSmall);
|
||||
if (nbSymbols > *maxSymbolValuePtr+1) return ERROR(maxSymbolValue_tooSmall);
|
||||
|
||||
/* Prepare base value per rank */
|
||||
{ U32 n, nextRankStart = 0;
|
||||
@ -208,9 +208,10 @@ size_t HUF_readCTable (HUF_CElt* CTable, U32 maxSymbolValue, const void* src, si
|
||||
min >>= 1;
|
||||
} }
|
||||
/* assign value within rank, symbol order */
|
||||
{ U32 n; for (n=0; n<=maxSymbolValue; n++) CTable[n].val = valPerRank[CTable[n].nbBits]++; }
|
||||
{ U32 n; for (n=0; n<nbSymbols; n++) CTable[n].val = valPerRank[CTable[n].nbBits]++; }
|
||||
}
|
||||
|
||||
*maxSymbolValuePtr = nbSymbols - 1;
|
||||
return readSize;
|
||||
}
|
||||
|
||||
|
File diff suppressed because it is too large
Load Diff
307
lib/compress/zstd_compress.h
Normal file
307
lib/compress/zstd_compress.h
Normal file
@ -0,0 +1,307 @@
|
||||
/*
|
||||
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
|
||||
#ifndef ZSTD_COMPRESS_H
|
||||
#define ZSTD_COMPRESS_H
|
||||
|
||||
/*-*************************************
|
||||
* Dependencies
|
||||
***************************************/
|
||||
#include "zstd_internal.h"
|
||||
#ifdef ZSTD_MULTITHREAD
|
||||
# include "zstdmt_compress.h"
|
||||
#endif
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/*-*************************************
|
||||
* Constants
|
||||
***************************************/
|
||||
static const U32 g_searchStrength = 8;
|
||||
#define HASH_READ_SIZE 8
|
||||
|
||||
|
||||
/*-*************************************
|
||||
* Context memory management
|
||||
***************************************/
|
||||
typedef enum { ZSTDcs_created=0, ZSTDcs_init, ZSTDcs_ongoing, ZSTDcs_ending } ZSTD_compressionStage_e;
|
||||
typedef enum { zcss_init=0, zcss_load, zcss_flush } ZSTD_cStreamStage;
|
||||
|
||||
typedef struct ZSTD_prefixDict_s {
|
||||
const void* dict;
|
||||
size_t dictSize;
|
||||
ZSTD_dictMode_e dictMode;
|
||||
} ZSTD_prefixDict;
|
||||
|
||||
struct ZSTD_CCtx_s {
|
||||
const BYTE* nextSrc; /* next block here to continue on current prefix */
|
||||
const BYTE* base; /* All regular indexes relative to this position */
|
||||
const BYTE* dictBase; /* extDict indexes relative to this position */
|
||||
U32 dictLimit; /* below that point, need extDict */
|
||||
U32 lowLimit; /* below that point, no more data */
|
||||
U32 nextToUpdate; /* index from which to continue dictionary update */
|
||||
U32 nextToUpdate3; /* index from which to continue dictionary update */
|
||||
U32 hashLog3; /* dispatch table : larger == faster, more memory */
|
||||
U32 loadedDictEnd; /* index of end of dictionary */
|
||||
ZSTD_compressionStage_e stage;
|
||||
U32 dictID;
|
||||
ZSTD_CCtx_params requestedParams;
|
||||
ZSTD_CCtx_params appliedParams;
|
||||
void* workSpace;
|
||||
size_t workSpaceSize;
|
||||
size_t blockSize;
|
||||
U64 pledgedSrcSizePlusOne; /* this way, 0 (default) == unknown */
|
||||
U64 consumedSrcSize;
|
||||
XXH64_state_t xxhState;
|
||||
ZSTD_customMem customMem;
|
||||
size_t staticSize;
|
||||
|
||||
seqStore_t seqStore; /* sequences storage ptrs */
|
||||
optState_t optState;
|
||||
ldmState_t ldmState; /* long distance matching state */
|
||||
U32* hashTable;
|
||||
U32* hashTable3;
|
||||
U32* chainTable;
|
||||
ZSTD_entropyCTables_t* entropy;
|
||||
|
||||
/* streaming */
|
||||
char* inBuff;
|
||||
size_t inBuffSize;
|
||||
size_t inToCompress;
|
||||
size_t inBuffPos;
|
||||
size_t inBuffTarget;
|
||||
char* outBuff;
|
||||
size_t outBuffSize;
|
||||
size_t outBuffContentSize;
|
||||
size_t outBuffFlushedSize;
|
||||
ZSTD_cStreamStage streamStage;
|
||||
U32 frameEnded;
|
||||
|
||||
/* Dictionary */
|
||||
ZSTD_CDict* cdictLocal;
|
||||
const ZSTD_CDict* cdict;
|
||||
ZSTD_prefixDict prefixDict; /* single-usage dictionary */
|
||||
|
||||
/* Multi-threading */
|
||||
#ifdef ZSTD_MULTITHREAD
|
||||
ZSTDMT_CCtx* mtctx;
|
||||
#endif
|
||||
};
|
||||
|
||||
|
||||
static const BYTE LL_Code[64] = { 0, 1, 2, 3, 4, 5, 6, 7,
|
||||
8, 9, 10, 11, 12, 13, 14, 15,
|
||||
16, 16, 17, 17, 18, 18, 19, 19,
|
||||
20, 20, 20, 20, 21, 21, 21, 21,
|
||||
22, 22, 22, 22, 22, 22, 22, 22,
|
||||
23, 23, 23, 23, 23, 23, 23, 23,
|
||||
24, 24, 24, 24, 24, 24, 24, 24,
|
||||
24, 24, 24, 24, 24, 24, 24, 24 };
|
||||
|
||||
static const BYTE ML_Code[128] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
|
||||
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
|
||||
32, 32, 33, 33, 34, 34, 35, 35, 36, 36, 36, 36, 37, 37, 37, 37,
|
||||
38, 38, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39,
|
||||
40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40,
|
||||
41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41,
|
||||
42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42,
|
||||
42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42 };
|
||||
|
||||
/*! ZSTD_storeSeq() :
|
||||
Store a sequence (literal length, literals, offset code and match length code) into seqStore_t.
|
||||
`offsetCode` : distance to match, or 0 == repCode.
|
||||
`matchCode` : matchLength - MINMATCH
|
||||
*/
|
||||
MEM_STATIC void ZSTD_storeSeq(seqStore_t* seqStorePtr, size_t litLength, const void* literals, U32 offsetCode, size_t matchCode)
|
||||
{
|
||||
#if defined(ZSTD_DEBUG) && (ZSTD_DEBUG >= 6)
|
||||
static const BYTE* g_start = NULL;
|
||||
U32 const pos = (U32)((const BYTE*)literals - g_start);
|
||||
if (g_start==NULL) g_start = (const BYTE*)literals;
|
||||
if ((pos > 0) && (pos < 1000000000))
|
||||
DEBUGLOG(6, "Cpos %6u :%5u literals & match %3u bytes at distance %6u",
|
||||
pos, (U32)litLength, (U32)matchCode+MINMATCH, (U32)offsetCode);
|
||||
#endif
|
||||
/* copy Literals */
|
||||
assert(seqStorePtr->lit + litLength <= seqStorePtr->litStart + 128 KB);
|
||||
ZSTD_wildcopy(seqStorePtr->lit, literals, litLength);
|
||||
seqStorePtr->lit += litLength;
|
||||
|
||||
/* literal Length */
|
||||
if (litLength>0xFFFF) {
|
||||
seqStorePtr->longLengthID = 1;
|
||||
seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
|
||||
}
|
||||
seqStorePtr->sequences[0].litLength = (U16)litLength;
|
||||
|
||||
/* match offset */
|
||||
seqStorePtr->sequences[0].offset = offsetCode + 1;
|
||||
|
||||
/* match Length */
|
||||
if (matchCode>0xFFFF) {
|
||||
seqStorePtr->longLengthID = 2;
|
||||
seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
|
||||
}
|
||||
seqStorePtr->sequences[0].matchLength = (U16)matchCode;
|
||||
|
||||
seqStorePtr->sequences++;
|
||||
}
|
||||
|
||||
|
||||
/*-*************************************
|
||||
* Match length counter
|
||||
***************************************/
|
||||
static unsigned ZSTD_NbCommonBytes (register size_t val)
|
||||
{
|
||||
if (MEM_isLittleEndian()) {
|
||||
if (MEM_64bits()) {
|
||||
# if defined(_MSC_VER) && defined(_WIN64)
|
||||
unsigned long r = 0;
|
||||
_BitScanForward64( &r, (U64)val );
|
||||
return (unsigned)(r>>3);
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 4)
|
||||
return (__builtin_ctzll((U64)val) >> 3);
|
||||
# else
|
||||
static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2,
|
||||
0, 3, 1, 3, 1, 4, 2, 7,
|
||||
0, 2, 3, 6, 1, 5, 3, 5,
|
||||
1, 3, 4, 4, 2, 5, 6, 7,
|
||||
7, 0, 1, 2, 3, 3, 4, 6,
|
||||
2, 6, 5, 5, 3, 4, 5, 6,
|
||||
7, 1, 2, 4, 6, 4, 4, 5,
|
||||
7, 2, 6, 5, 7, 6, 7, 7 };
|
||||
return DeBruijnBytePos[((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58];
|
||||
# endif
|
||||
} else { /* 32 bits */
|
||||
# if defined(_MSC_VER)
|
||||
unsigned long r=0;
|
||||
_BitScanForward( &r, (U32)val );
|
||||
return (unsigned)(r>>3);
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3)
|
||||
return (__builtin_ctz((U32)val) >> 3);
|
||||
# else
|
||||
static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0,
|
||||
3, 2, 2, 1, 3, 2, 0, 1,
|
||||
3, 3, 1, 2, 2, 2, 2, 0,
|
||||
3, 1, 2, 0, 1, 0, 1, 1 };
|
||||
return DeBruijnBytePos[((U32)((val & -(S32)val) * 0x077CB531U)) >> 27];
|
||||
# endif
|
||||
}
|
||||
} else { /* Big Endian CPU */
|
||||
if (MEM_64bits()) {
|
||||
# if defined(_MSC_VER) && defined(_WIN64)
|
||||
unsigned long r = 0;
|
||||
_BitScanReverse64( &r, val );
|
||||
return (unsigned)(r>>3);
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 4)
|
||||
return (__builtin_clzll(val) >> 3);
|
||||
# else
|
||||
unsigned r;
|
||||
const unsigned n32 = sizeof(size_t)*4; /* calculate this way due to compiler complaining in 32-bits mode */
|
||||
if (!(val>>n32)) { r=4; } else { r=0; val>>=n32; }
|
||||
if (!(val>>16)) { r+=2; val>>=8; } else { val>>=24; }
|
||||
r += (!val);
|
||||
return r;
|
||||
# endif
|
||||
} else { /* 32 bits */
|
||||
# if defined(_MSC_VER)
|
||||
unsigned long r = 0;
|
||||
_BitScanReverse( &r, (unsigned long)val );
|
||||
return (unsigned)(r>>3);
|
||||
# elif defined(__GNUC__) && (__GNUC__ >= 3)
|
||||
return (__builtin_clz((U32)val) >> 3);
|
||||
# else
|
||||
unsigned r;
|
||||
if (!(val>>16)) { r=2; val>>=8; } else { r=0; val>>=24; }
|
||||
r += (!val);
|
||||
return r;
|
||||
# endif
|
||||
} }
|
||||
}
|
||||
|
||||
|
||||
MEM_STATIC size_t ZSTD_count(const BYTE* pIn, const BYTE* pMatch, const BYTE* const pInLimit)
|
||||
{
|
||||
const BYTE* const pStart = pIn;
|
||||
const BYTE* const pInLoopLimit = pInLimit - (sizeof(size_t)-1);
|
||||
|
||||
while (pIn < pInLoopLimit) {
|
||||
size_t const diff = MEM_readST(pMatch) ^ MEM_readST(pIn);
|
||||
if (!diff) { pIn+=sizeof(size_t); pMatch+=sizeof(size_t); continue; }
|
||||
pIn += ZSTD_NbCommonBytes(diff);
|
||||
return (size_t)(pIn - pStart);
|
||||
}
|
||||
if (MEM_64bits()) if ((pIn<(pInLimit-3)) && (MEM_read32(pMatch) == MEM_read32(pIn))) { pIn+=4; pMatch+=4; }
|
||||
if ((pIn<(pInLimit-1)) && (MEM_read16(pMatch) == MEM_read16(pIn))) { pIn+=2; pMatch+=2; }
|
||||
if ((pIn<pInLimit) && (*pMatch == *pIn)) pIn++;
|
||||
return (size_t)(pIn - pStart);
|
||||
}
|
||||
|
||||
/** ZSTD_count_2segments() :
|
||||
* can count match length with `ip` & `match` in 2 different segments.
|
||||
* convention : on reaching mEnd, match count continue starting from iStart
|
||||
*/
|
||||
MEM_STATIC size_t ZSTD_count_2segments(const BYTE* ip, const BYTE* match, const BYTE* iEnd, const BYTE* mEnd, const BYTE* iStart)
|
||||
{
|
||||
const BYTE* const vEnd = MIN( ip + (mEnd - match), iEnd);
|
||||
size_t const matchLength = ZSTD_count(ip, match, vEnd);
|
||||
if (match + matchLength != mEnd) return matchLength;
|
||||
return matchLength + ZSTD_count(ip+matchLength, iStart, iEnd);
|
||||
}
|
||||
|
||||
|
||||
/*-*************************************
|
||||
* Hashes
|
||||
***************************************/
|
||||
static const U32 prime3bytes = 506832829U;
|
||||
static U32 ZSTD_hash3(U32 u, U32 h) { return ((u << (32-24)) * prime3bytes) >> (32-h) ; }
|
||||
MEM_STATIC size_t ZSTD_hash3Ptr(const void* ptr, U32 h) { return ZSTD_hash3(MEM_readLE32(ptr), h); } /* only in zstd_opt.h */
|
||||
|
||||
static const U32 prime4bytes = 2654435761U;
|
||||
static U32 ZSTD_hash4(U32 u, U32 h) { return (u * prime4bytes) >> (32-h) ; }
|
||||
static size_t ZSTD_hash4Ptr(const void* ptr, U32 h) { return ZSTD_hash4(MEM_read32(ptr), h); }
|
||||
|
||||
static const U64 prime5bytes = 889523592379ULL;
|
||||
static size_t ZSTD_hash5(U64 u, U32 h) { return (size_t)(((u << (64-40)) * prime5bytes) >> (64-h)) ; }
|
||||
static size_t ZSTD_hash5Ptr(const void* p, U32 h) { return ZSTD_hash5(MEM_readLE64(p), h); }
|
||||
|
||||
static const U64 prime6bytes = 227718039650203ULL;
|
||||
static size_t ZSTD_hash6(U64 u, U32 h) { return (size_t)(((u << (64-48)) * prime6bytes) >> (64-h)) ; }
|
||||
static size_t ZSTD_hash6Ptr(const void* p, U32 h) { return ZSTD_hash6(MEM_readLE64(p), h); }
|
||||
|
||||
static const U64 prime7bytes = 58295818150454627ULL;
|
||||
static size_t ZSTD_hash7(U64 u, U32 h) { return (size_t)(((u << (64-56)) * prime7bytes) >> (64-h)) ; }
|
||||
static size_t ZSTD_hash7Ptr(const void* p, U32 h) { return ZSTD_hash7(MEM_readLE64(p), h); }
|
||||
|
||||
static const U64 prime8bytes = 0xCF1BBCDCB7A56463ULL;
|
||||
static size_t ZSTD_hash8(U64 u, U32 h) { return (size_t)(((u) * prime8bytes) >> (64-h)) ; }
|
||||
static size_t ZSTD_hash8Ptr(const void* p, U32 h) { return ZSTD_hash8(MEM_readLE64(p), h); }
|
||||
|
||||
MEM_STATIC size_t ZSTD_hashPtr(const void* p, U32 hBits, U32 mls)
|
||||
{
|
||||
switch(mls)
|
||||
{
|
||||
default:
|
||||
case 4: return ZSTD_hash4Ptr(p, hBits);
|
||||
case 5: return ZSTD_hash5Ptr(p, hBits);
|
||||
case 6: return ZSTD_hash6Ptr(p, hBits);
|
||||
case 7: return ZSTD_hash7Ptr(p, hBits);
|
||||
case 8: return ZSTD_hash8Ptr(p, hBits);
|
||||
}
|
||||
}
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* ZSTD_COMPRESS_H */
|
308
lib/compress/zstd_double_fast.c
Normal file
308
lib/compress/zstd_double_fast.c
Normal file
@ -0,0 +1,308 @@
|
||||
/*
|
||||
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#include "zstd_double_fast.h"
|
||||
|
||||
|
||||
void ZSTD_fillDoubleHashTable(ZSTD_CCtx* cctx, const void* end, const U32 mls)
|
||||
{
|
||||
U32* const hashLarge = cctx->hashTable;
|
||||
U32 const hBitsL = cctx->appliedParams.cParams.hashLog;
|
||||
U32* const hashSmall = cctx->chainTable;
|
||||
U32 const hBitsS = cctx->appliedParams.cParams.chainLog;
|
||||
const BYTE* const base = cctx->base;
|
||||
const BYTE* ip = base + cctx->nextToUpdate;
|
||||
const BYTE* const iend = ((const BYTE*)end) - HASH_READ_SIZE;
|
||||
const size_t fastHashFillStep = 3;
|
||||
|
||||
while(ip <= iend) {
|
||||
hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = (U32)(ip - base);
|
||||
hashLarge[ZSTD_hashPtr(ip, hBitsL, 8)] = (U32)(ip - base);
|
||||
ip += fastHashFillStep;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
FORCE_INLINE_TEMPLATE
|
||||
size_t ZSTD_compressBlock_doubleFast_generic(ZSTD_CCtx* cctx,
|
||||
const void* src, size_t srcSize,
|
||||
const U32 mls)
|
||||
{
|
||||
U32* const hashLong = cctx->hashTable;
|
||||
const U32 hBitsL = cctx->appliedParams.cParams.hashLog;
|
||||
U32* const hashSmall = cctx->chainTable;
|
||||
const U32 hBitsS = cctx->appliedParams.cParams.chainLog;
|
||||
seqStore_t* seqStorePtr = &(cctx->seqStore);
|
||||
const BYTE* const base = cctx->base;
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const U32 lowestIndex = cctx->dictLimit;
|
||||
const BYTE* const lowest = base + lowestIndex;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - HASH_READ_SIZE;
|
||||
U32 offset_1=seqStorePtr->rep[0], offset_2=seqStorePtr->rep[1];
|
||||
U32 offsetSaved = 0;
|
||||
|
||||
/* init */
|
||||
ip += (ip==lowest);
|
||||
{ U32 const maxRep = (U32)(ip-lowest);
|
||||
if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0;
|
||||
if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0;
|
||||
}
|
||||
|
||||
/* Main Search Loop */
|
||||
while (ip < ilimit) { /* < instead of <=, because repcode check at (ip+1) */
|
||||
size_t mLength;
|
||||
size_t const h2 = ZSTD_hashPtr(ip, hBitsL, 8);
|
||||
size_t const h = ZSTD_hashPtr(ip, hBitsS, mls);
|
||||
U32 const current = (U32)(ip-base);
|
||||
U32 const matchIndexL = hashLong[h2];
|
||||
U32 const matchIndexS = hashSmall[h];
|
||||
const BYTE* matchLong = base + matchIndexL;
|
||||
const BYTE* match = base + matchIndexS;
|
||||
hashLong[h2] = hashSmall[h] = current; /* update hash tables */
|
||||
|
||||
assert(offset_1 <= current); /* supposed guaranteed by construction */
|
||||
if ((offset_1 > 0) & (MEM_read32(ip+1-offset_1) == MEM_read32(ip+1))) {
|
||||
/* favor repcode */
|
||||
mLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4;
|
||||
ip++;
|
||||
ZSTD_storeSeq(seqStorePtr, ip-anchor, anchor, 0, mLength-MINMATCH);
|
||||
} else {
|
||||
U32 offset;
|
||||
if ( (matchIndexL > lowestIndex) && (MEM_read64(matchLong) == MEM_read64(ip)) ) {
|
||||
mLength = ZSTD_count(ip+8, matchLong+8, iend) + 8;
|
||||
offset = (U32)(ip-matchLong);
|
||||
while (((ip>anchor) & (matchLong>lowest)) && (ip[-1] == matchLong[-1])) { ip--; matchLong--; mLength++; } /* catch up */
|
||||
} else if ( (matchIndexS > lowestIndex) && (MEM_read32(match) == MEM_read32(ip)) ) {
|
||||
size_t const hl3 = ZSTD_hashPtr(ip+1, hBitsL, 8);
|
||||
U32 const matchIndexL3 = hashLong[hl3];
|
||||
const BYTE* matchL3 = base + matchIndexL3;
|
||||
hashLong[hl3] = current + 1;
|
||||
if ( (matchIndexL3 > lowestIndex) && (MEM_read64(matchL3) == MEM_read64(ip+1)) ) {
|
||||
mLength = ZSTD_count(ip+9, matchL3+8, iend) + 8;
|
||||
ip++;
|
||||
offset = (U32)(ip-matchL3);
|
||||
while (((ip>anchor) & (matchL3>lowest)) && (ip[-1] == matchL3[-1])) { ip--; matchL3--; mLength++; } /* catch up */
|
||||
} else {
|
||||
mLength = ZSTD_count(ip+4, match+4, iend) + 4;
|
||||
offset = (U32)(ip-match);
|
||||
while (((ip>anchor) & (match>lowest)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
|
||||
}
|
||||
} else {
|
||||
ip += ((ip-anchor) >> g_searchStrength) + 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
offset_2 = offset_1;
|
||||
offset_1 = offset;
|
||||
|
||||
ZSTD_storeSeq(seqStorePtr, ip-anchor, anchor, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
|
||||
}
|
||||
|
||||
/* match found */
|
||||
ip += mLength;
|
||||
anchor = ip;
|
||||
|
||||
if (ip <= ilimit) {
|
||||
/* Fill Table */
|
||||
hashLong[ZSTD_hashPtr(base+current+2, hBitsL, 8)] =
|
||||
hashSmall[ZSTD_hashPtr(base+current+2, hBitsS, mls)] = current+2; /* here because current+2 could be > iend-8 */
|
||||
hashLong[ZSTD_hashPtr(ip-2, hBitsL, 8)] =
|
||||
hashSmall[ZSTD_hashPtr(ip-2, hBitsS, mls)] = (U32)(ip-2-base);
|
||||
|
||||
/* check immediate repcode */
|
||||
while ( (ip <= ilimit)
|
||||
&& ( (offset_2>0)
|
||||
& (MEM_read32(ip) == MEM_read32(ip - offset_2)) )) {
|
||||
/* store sequence */
|
||||
size_t const rLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4;
|
||||
{ U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; } /* swap offset_2 <=> offset_1 */
|
||||
hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = (U32)(ip-base);
|
||||
hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = (U32)(ip-base);
|
||||
ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, rLength-MINMATCH);
|
||||
ip += rLength;
|
||||
anchor = ip;
|
||||
continue; /* faster when present ... (?) */
|
||||
} } }
|
||||
|
||||
/* save reps for next block */
|
||||
seqStorePtr->repToConfirm[0] = offset_1 ? offset_1 : offsetSaved;
|
||||
seqStorePtr->repToConfirm[1] = offset_2 ? offset_2 : offsetSaved;
|
||||
|
||||
/* Return the last literals size */
|
||||
return iend - anchor;
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTD_compressBlock_doubleFast(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
const U32 mls = ctx->appliedParams.cParams.searchLength;
|
||||
switch(mls)
|
||||
{
|
||||
default: /* includes case 3 */
|
||||
case 4 :
|
||||
return ZSTD_compressBlock_doubleFast_generic(ctx, src, srcSize, 4);
|
||||
case 5 :
|
||||
return ZSTD_compressBlock_doubleFast_generic(ctx, src, srcSize, 5);
|
||||
case 6 :
|
||||
return ZSTD_compressBlock_doubleFast_generic(ctx, src, srcSize, 6);
|
||||
case 7 :
|
||||
return ZSTD_compressBlock_doubleFast_generic(ctx, src, srcSize, 7);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
static size_t ZSTD_compressBlock_doubleFast_extDict_generic(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize,
|
||||
const U32 mls)
|
||||
{
|
||||
U32* const hashLong = ctx->hashTable;
|
||||
U32 const hBitsL = ctx->appliedParams.cParams.hashLog;
|
||||
U32* const hashSmall = ctx->chainTable;
|
||||
U32 const hBitsS = ctx->appliedParams.cParams.chainLog;
|
||||
seqStore_t* seqStorePtr = &(ctx->seqStore);
|
||||
const BYTE* const base = ctx->base;
|
||||
const BYTE* const dictBase = ctx->dictBase;
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const U32 lowestIndex = ctx->lowLimit;
|
||||
const BYTE* const dictStart = dictBase + lowestIndex;
|
||||
const U32 dictLimit = ctx->dictLimit;
|
||||
const BYTE* const lowPrefixPtr = base + dictLimit;
|
||||
const BYTE* const dictEnd = dictBase + dictLimit;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - 8;
|
||||
U32 offset_1=seqStorePtr->rep[0], offset_2=seqStorePtr->rep[1];
|
||||
|
||||
/* Search Loop */
|
||||
while (ip < ilimit) { /* < instead of <=, because (ip+1) */
|
||||
const size_t hSmall = ZSTD_hashPtr(ip, hBitsS, mls);
|
||||
const U32 matchIndex = hashSmall[hSmall];
|
||||
const BYTE* matchBase = matchIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* match = matchBase + matchIndex;
|
||||
|
||||
const size_t hLong = ZSTD_hashPtr(ip, hBitsL, 8);
|
||||
const U32 matchLongIndex = hashLong[hLong];
|
||||
const BYTE* matchLongBase = matchLongIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* matchLong = matchLongBase + matchLongIndex;
|
||||
|
||||
const U32 current = (U32)(ip-base);
|
||||
const U32 repIndex = current + 1 - offset_1; /* offset_1 expected <= current +1 */
|
||||
const BYTE* repBase = repIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* repMatch = repBase + repIndex;
|
||||
size_t mLength;
|
||||
hashSmall[hSmall] = hashLong[hLong] = current; /* update hash table */
|
||||
|
||||
if ( (((U32)((dictLimit-1) - repIndex) >= 3) /* intentional underflow */ & (repIndex > lowestIndex))
|
||||
&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
|
||||
const BYTE* repMatchEnd = repIndex < dictLimit ? dictEnd : iend;
|
||||
mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, lowPrefixPtr) + 4;
|
||||
ip++;
|
||||
ZSTD_storeSeq(seqStorePtr, ip-anchor, anchor, 0, mLength-MINMATCH);
|
||||
} else {
|
||||
if ((matchLongIndex > lowestIndex) && (MEM_read64(matchLong) == MEM_read64(ip))) {
|
||||
const BYTE* matchEnd = matchLongIndex < dictLimit ? dictEnd : iend;
|
||||
const BYTE* lowMatchPtr = matchLongIndex < dictLimit ? dictStart : lowPrefixPtr;
|
||||
U32 offset;
|
||||
mLength = ZSTD_count_2segments(ip+8, matchLong+8, iend, matchEnd, lowPrefixPtr) + 8;
|
||||
offset = current - matchLongIndex;
|
||||
while (((ip>anchor) & (matchLong>lowMatchPtr)) && (ip[-1] == matchLong[-1])) { ip--; matchLong--; mLength++; } /* catch up */
|
||||
offset_2 = offset_1;
|
||||
offset_1 = offset;
|
||||
ZSTD_storeSeq(seqStorePtr, ip-anchor, anchor, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
|
||||
|
||||
} else if ((matchIndex > lowestIndex) && (MEM_read32(match) == MEM_read32(ip))) {
|
||||
size_t const h3 = ZSTD_hashPtr(ip+1, hBitsL, 8);
|
||||
U32 const matchIndex3 = hashLong[h3];
|
||||
const BYTE* const match3Base = matchIndex3 < dictLimit ? dictBase : base;
|
||||
const BYTE* match3 = match3Base + matchIndex3;
|
||||
U32 offset;
|
||||
hashLong[h3] = current + 1;
|
||||
if ( (matchIndex3 > lowestIndex) && (MEM_read64(match3) == MEM_read64(ip+1)) ) {
|
||||
const BYTE* matchEnd = matchIndex3 < dictLimit ? dictEnd : iend;
|
||||
const BYTE* lowMatchPtr = matchIndex3 < dictLimit ? dictStart : lowPrefixPtr;
|
||||
mLength = ZSTD_count_2segments(ip+9, match3+8, iend, matchEnd, lowPrefixPtr) + 8;
|
||||
ip++;
|
||||
offset = current+1 - matchIndex3;
|
||||
while (((ip>anchor) & (match3>lowMatchPtr)) && (ip[-1] == match3[-1])) { ip--; match3--; mLength++; } /* catch up */
|
||||
} else {
|
||||
const BYTE* matchEnd = matchIndex < dictLimit ? dictEnd : iend;
|
||||
const BYTE* lowMatchPtr = matchIndex < dictLimit ? dictStart : lowPrefixPtr;
|
||||
mLength = ZSTD_count_2segments(ip+4, match+4, iend, matchEnd, lowPrefixPtr) + 4;
|
||||
offset = current - matchIndex;
|
||||
while (((ip>anchor) & (match>lowMatchPtr)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
|
||||
}
|
||||
offset_2 = offset_1;
|
||||
offset_1 = offset;
|
||||
ZSTD_storeSeq(seqStorePtr, ip-anchor, anchor, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
|
||||
|
||||
} else {
|
||||
ip += ((ip-anchor) >> g_searchStrength) + 1;
|
||||
continue;
|
||||
} }
|
||||
|
||||
/* found a match : store it */
|
||||
ip += mLength;
|
||||
anchor = ip;
|
||||
|
||||
if (ip <= ilimit) {
|
||||
/* Fill Table */
|
||||
hashSmall[ZSTD_hashPtr(base+current+2, hBitsS, mls)] = current+2;
|
||||
hashLong[ZSTD_hashPtr(base+current+2, hBitsL, 8)] = current+2;
|
||||
hashSmall[ZSTD_hashPtr(ip-2, hBitsS, mls)] = (U32)(ip-2-base);
|
||||
hashLong[ZSTD_hashPtr(ip-2, hBitsL, 8)] = (U32)(ip-2-base);
|
||||
/* check immediate repcode */
|
||||
while (ip <= ilimit) {
|
||||
U32 const current2 = (U32)(ip-base);
|
||||
U32 const repIndex2 = current2 - offset_2;
|
||||
const BYTE* repMatch2 = repIndex2 < dictLimit ? dictBase + repIndex2 : base + repIndex2;
|
||||
if ( (((U32)((dictLimit-1) - repIndex2) >= 3) & (repIndex2 > lowestIndex)) /* intentional overflow */
|
||||
&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
|
||||
const BYTE* const repEnd2 = repIndex2 < dictLimit ? dictEnd : iend;
|
||||
size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, lowPrefixPtr) + 4;
|
||||
U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
|
||||
ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, repLength2-MINMATCH);
|
||||
hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2;
|
||||
hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2;
|
||||
ip += repLength2;
|
||||
anchor = ip;
|
||||
continue;
|
||||
}
|
||||
break;
|
||||
} } }
|
||||
|
||||
/* save reps for next block */
|
||||
seqStorePtr->repToConfirm[0] = offset_1; seqStorePtr->repToConfirm[1] = offset_2;
|
||||
|
||||
/* Return the last literals size */
|
||||
return iend - anchor;
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTD_compressBlock_doubleFast_extDict(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize)
|
||||
{
|
||||
U32 const mls = ctx->appliedParams.cParams.searchLength;
|
||||
switch(mls)
|
||||
{
|
||||
default: /* includes case 3 */
|
||||
case 4 :
|
||||
return ZSTD_compressBlock_doubleFast_extDict_generic(ctx, src, srcSize, 4);
|
||||
case 5 :
|
||||
return ZSTD_compressBlock_doubleFast_extDict_generic(ctx, src, srcSize, 5);
|
||||
case 6 :
|
||||
return ZSTD_compressBlock_doubleFast_extDict_generic(ctx, src, srcSize, 6);
|
||||
case 7 :
|
||||
return ZSTD_compressBlock_doubleFast_extDict_generic(ctx, src, srcSize, 7);
|
||||
}
|
||||
}
|
28
lib/compress/zstd_double_fast.h
Normal file
28
lib/compress/zstd_double_fast.h
Normal file
@ -0,0 +1,28 @@
|
||||
/*
|
||||
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef ZSTD_DOUBLE_FAST_H
|
||||
#define ZSTD_DOUBLE_FAST_H
|
||||
|
||||
#include "zstd_compress.h"
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
void ZSTD_fillDoubleHashTable(ZSTD_CCtx* cctx, const void* end, const U32 mls);
|
||||
size_t ZSTD_compressBlock_doubleFast(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
size_t ZSTD_compressBlock_doubleFast_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* ZSTD_DOUBLE_FAST_H */
|
242
lib/compress/zstd_fast.c
Normal file
242
lib/compress/zstd_fast.c
Normal file
@ -0,0 +1,242 @@
|
||||
/*
|
||||
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#include "zstd_fast.h"
|
||||
|
||||
|
||||
void ZSTD_fillHashTable (ZSTD_CCtx* zc, const void* end, const U32 mls)
|
||||
{
|
||||
U32* const hashTable = zc->hashTable;
|
||||
U32 const hBits = zc->appliedParams.cParams.hashLog;
|
||||
const BYTE* const base = zc->base;
|
||||
const BYTE* ip = base + zc->nextToUpdate;
|
||||
const BYTE* const iend = ((const BYTE*)end) - HASH_READ_SIZE;
|
||||
const size_t fastHashFillStep = 3;
|
||||
|
||||
while(ip <= iend) {
|
||||
hashTable[ZSTD_hashPtr(ip, hBits, mls)] = (U32)(ip - base);
|
||||
ip += fastHashFillStep;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
FORCE_INLINE_TEMPLATE
|
||||
size_t ZSTD_compressBlock_fast_generic(ZSTD_CCtx* cctx,
|
||||
const void* src, size_t srcSize,
|
||||
const U32 mls)
|
||||
{
|
||||
U32* const hashTable = cctx->hashTable;
|
||||
U32 const hBits = cctx->appliedParams.cParams.hashLog;
|
||||
seqStore_t* seqStorePtr = &(cctx->seqStore);
|
||||
const BYTE* const base = cctx->base;
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const U32 lowestIndex = cctx->dictLimit;
|
||||
const BYTE* const lowest = base + lowestIndex;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - HASH_READ_SIZE;
|
||||
U32 offset_1=seqStorePtr->rep[0], offset_2=seqStorePtr->rep[1];
|
||||
U32 offsetSaved = 0;
|
||||
|
||||
/* init */
|
||||
ip += (ip==lowest);
|
||||
{ U32 const maxRep = (U32)(ip-lowest);
|
||||
if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0;
|
||||
if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0;
|
||||
}
|
||||
|
||||
/* Main Search Loop */
|
||||
while (ip < ilimit) { /* < instead of <=, because repcode check at (ip+1) */
|
||||
size_t mLength;
|
||||
size_t const h = ZSTD_hashPtr(ip, hBits, mls);
|
||||
U32 const current = (U32)(ip-base);
|
||||
U32 const matchIndex = hashTable[h];
|
||||
const BYTE* match = base + matchIndex;
|
||||
hashTable[h] = current; /* update hash table */
|
||||
|
||||
if ((offset_1 > 0) & (MEM_read32(ip+1-offset_1) == MEM_read32(ip+1))) {
|
||||
mLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4;
|
||||
ip++;
|
||||
ZSTD_storeSeq(seqStorePtr, ip-anchor, anchor, 0, mLength-MINMATCH);
|
||||
} else {
|
||||
U32 offset;
|
||||
if ( (matchIndex <= lowestIndex) || (MEM_read32(match) != MEM_read32(ip)) ) {
|
||||
ip += ((ip-anchor) >> g_searchStrength) + 1;
|
||||
continue;
|
||||
}
|
||||
mLength = ZSTD_count(ip+4, match+4, iend) + 4;
|
||||
offset = (U32)(ip-match);
|
||||
while (((ip>anchor) & (match>lowest)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
|
||||
offset_2 = offset_1;
|
||||
offset_1 = offset;
|
||||
|
||||
ZSTD_storeSeq(seqStorePtr, ip-anchor, anchor, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
|
||||
}
|
||||
|
||||
/* match found */
|
||||
ip += mLength;
|
||||
anchor = ip;
|
||||
|
||||
if (ip <= ilimit) {
|
||||
/* Fill Table */
|
||||
hashTable[ZSTD_hashPtr(base+current+2, hBits, mls)] = current+2; /* here because current+2 could be > iend-8 */
|
||||
hashTable[ZSTD_hashPtr(ip-2, hBits, mls)] = (U32)(ip-2-base);
|
||||
/* check immediate repcode */
|
||||
while ( (ip <= ilimit)
|
||||
&& ( (offset_2>0)
|
||||
& (MEM_read32(ip) == MEM_read32(ip - offset_2)) )) {
|
||||
/* store sequence */
|
||||
size_t const rLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4;
|
||||
{ U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; } /* swap offset_2 <=> offset_1 */
|
||||
hashTable[ZSTD_hashPtr(ip, hBits, mls)] = (U32)(ip-base);
|
||||
ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, rLength-MINMATCH);
|
||||
ip += rLength;
|
||||
anchor = ip;
|
||||
continue; /* faster when present ... (?) */
|
||||
} } }
|
||||
|
||||
/* save reps for next block */
|
||||
seqStorePtr->repToConfirm[0] = offset_1 ? offset_1 : offsetSaved;
|
||||
seqStorePtr->repToConfirm[1] = offset_2 ? offset_2 : offsetSaved;
|
||||
|
||||
/* Return the last literals size */
|
||||
return iend - anchor;
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTD_compressBlock_fast(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize)
|
||||
{
|
||||
const U32 mls = ctx->appliedParams.cParams.searchLength;
|
||||
switch(mls)
|
||||
{
|
||||
default: /* includes case 3 */
|
||||
case 4 :
|
||||
return ZSTD_compressBlock_fast_generic(ctx, src, srcSize, 4);
|
||||
case 5 :
|
||||
return ZSTD_compressBlock_fast_generic(ctx, src, srcSize, 5);
|
||||
case 6 :
|
||||
return ZSTD_compressBlock_fast_generic(ctx, src, srcSize, 6);
|
||||
case 7 :
|
||||
return ZSTD_compressBlock_fast_generic(ctx, src, srcSize, 7);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
static size_t ZSTD_compressBlock_fast_extDict_generic(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize,
|
||||
const U32 mls)
|
||||
{
|
||||
U32* hashTable = ctx->hashTable;
|
||||
const U32 hBits = ctx->appliedParams.cParams.hashLog;
|
||||
seqStore_t* seqStorePtr = &(ctx->seqStore);
|
||||
const BYTE* const base = ctx->base;
|
||||
const BYTE* const dictBase = ctx->dictBase;
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const U32 lowestIndex = ctx->lowLimit;
|
||||
const BYTE* const dictStart = dictBase + lowestIndex;
|
||||
const U32 dictLimit = ctx->dictLimit;
|
||||
const BYTE* const lowPrefixPtr = base + dictLimit;
|
||||
const BYTE* const dictEnd = dictBase + dictLimit;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - 8;
|
||||
U32 offset_1=seqStorePtr->rep[0], offset_2=seqStorePtr->rep[1];
|
||||
|
||||
/* Search Loop */
|
||||
while (ip < ilimit) { /* < instead of <=, because (ip+1) */
|
||||
const size_t h = ZSTD_hashPtr(ip, hBits, mls);
|
||||
const U32 matchIndex = hashTable[h];
|
||||
const BYTE* matchBase = matchIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* match = matchBase + matchIndex;
|
||||
const U32 current = (U32)(ip-base);
|
||||
const U32 repIndex = current + 1 - offset_1; /* offset_1 expected <= current +1 */
|
||||
const BYTE* repBase = repIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* repMatch = repBase + repIndex;
|
||||
size_t mLength;
|
||||
hashTable[h] = current; /* update hash table */
|
||||
|
||||
if ( (((U32)((dictLimit-1) - repIndex) >= 3) /* intentional underflow */ & (repIndex > lowestIndex))
|
||||
&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
|
||||
const BYTE* repMatchEnd = repIndex < dictLimit ? dictEnd : iend;
|
||||
mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, lowPrefixPtr) + 4;
|
||||
ip++;
|
||||
ZSTD_storeSeq(seqStorePtr, ip-anchor, anchor, 0, mLength-MINMATCH);
|
||||
} else {
|
||||
if ( (matchIndex < lowestIndex) ||
|
||||
(MEM_read32(match) != MEM_read32(ip)) ) {
|
||||
ip += ((ip-anchor) >> g_searchStrength) + 1;
|
||||
continue;
|
||||
}
|
||||
{ const BYTE* matchEnd = matchIndex < dictLimit ? dictEnd : iend;
|
||||
const BYTE* lowMatchPtr = matchIndex < dictLimit ? dictStart : lowPrefixPtr;
|
||||
U32 offset;
|
||||
mLength = ZSTD_count_2segments(ip+4, match+4, iend, matchEnd, lowPrefixPtr) + 4;
|
||||
while (((ip>anchor) & (match>lowMatchPtr)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
|
||||
offset = current - matchIndex;
|
||||
offset_2 = offset_1;
|
||||
offset_1 = offset;
|
||||
ZSTD_storeSeq(seqStorePtr, ip-anchor, anchor, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
|
||||
} }
|
||||
|
||||
/* found a match : store it */
|
||||
ip += mLength;
|
||||
anchor = ip;
|
||||
|
||||
if (ip <= ilimit) {
|
||||
/* Fill Table */
|
||||
hashTable[ZSTD_hashPtr(base+current+2, hBits, mls)] = current+2;
|
||||
hashTable[ZSTD_hashPtr(ip-2, hBits, mls)] = (U32)(ip-2-base);
|
||||
/* check immediate repcode */
|
||||
while (ip <= ilimit) {
|
||||
U32 const current2 = (U32)(ip-base);
|
||||
U32 const repIndex2 = current2 - offset_2;
|
||||
const BYTE* repMatch2 = repIndex2 < dictLimit ? dictBase + repIndex2 : base + repIndex2;
|
||||
if ( (((U32)((dictLimit-1) - repIndex2) >= 3) & (repIndex2 > lowestIndex)) /* intentional overflow */
|
||||
&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
|
||||
const BYTE* const repEnd2 = repIndex2 < dictLimit ? dictEnd : iend;
|
||||
size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, lowPrefixPtr) + 4;
|
||||
U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
|
||||
ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, repLength2-MINMATCH);
|
||||
hashTable[ZSTD_hashPtr(ip, hBits, mls)] = current2;
|
||||
ip += repLength2;
|
||||
anchor = ip;
|
||||
continue;
|
||||
}
|
||||
break;
|
||||
} } }
|
||||
|
||||
/* save reps for next block */
|
||||
seqStorePtr->repToConfirm[0] = offset_1; seqStorePtr->repToConfirm[1] = offset_2;
|
||||
|
||||
/* Return the last literals size */
|
||||
return iend - anchor;
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTD_compressBlock_fast_extDict(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize)
|
||||
{
|
||||
U32 const mls = ctx->appliedParams.cParams.searchLength;
|
||||
switch(mls)
|
||||
{
|
||||
default: /* includes case 3 */
|
||||
case 4 :
|
||||
return ZSTD_compressBlock_fast_extDict_generic(ctx, src, srcSize, 4);
|
||||
case 5 :
|
||||
return ZSTD_compressBlock_fast_extDict_generic(ctx, src, srcSize, 5);
|
||||
case 6 :
|
||||
return ZSTD_compressBlock_fast_extDict_generic(ctx, src, srcSize, 6);
|
||||
case 7 :
|
||||
return ZSTD_compressBlock_fast_extDict_generic(ctx, src, srcSize, 7);
|
||||
}
|
||||
}
|
30
lib/compress/zstd_fast.h
Normal file
30
lib/compress/zstd_fast.h
Normal file
@ -0,0 +1,30 @@
|
||||
/*
|
||||
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef ZSTD_FAST_H
|
||||
#define ZSTD_FAST_H
|
||||
|
||||
#include "zstd_compress.h"
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
void ZSTD_fillHashTable(ZSTD_CCtx* zc, const void* end, const U32 mls);
|
||||
size_t ZSTD_compressBlock_fast(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize);
|
||||
size_t ZSTD_compressBlock_fast_extDict(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize);
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* ZSTD_FAST_H */
|
749
lib/compress/zstd_lazy.c
Normal file
749
lib/compress/zstd_lazy.c
Normal file
@ -0,0 +1,749 @@
|
||||
/*
|
||||
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#include "zstd_lazy.h"
|
||||
|
||||
|
||||
/*-*************************************
|
||||
* Binary Tree search
|
||||
***************************************/
|
||||
/** ZSTD_insertBt1() : add one or multiple positions to tree.
|
||||
* ip : assumed <= iend-8 .
|
||||
* @return : nb of positions added */
|
||||
static U32 ZSTD_insertBt1(ZSTD_CCtx* zc, const BYTE* const ip, const U32 mls, const BYTE* const iend, U32 nbCompares,
|
||||
U32 extDict)
|
||||
{
|
||||
U32* const hashTable = zc->hashTable;
|
||||
U32 const hashLog = zc->appliedParams.cParams.hashLog;
|
||||
size_t const h = ZSTD_hashPtr(ip, hashLog, mls);
|
||||
U32* const bt = zc->chainTable;
|
||||
U32 const btLog = zc->appliedParams.cParams.chainLog - 1;
|
||||
U32 const btMask = (1 << btLog) - 1;
|
||||
U32 matchIndex = hashTable[h];
|
||||
size_t commonLengthSmaller=0, commonLengthLarger=0;
|
||||
const BYTE* const base = zc->base;
|
||||
const BYTE* const dictBase = zc->dictBase;
|
||||
const U32 dictLimit = zc->dictLimit;
|
||||
const BYTE* const dictEnd = dictBase + dictLimit;
|
||||
const BYTE* const prefixStart = base + dictLimit;
|
||||
const BYTE* match;
|
||||
const U32 current = (U32)(ip-base);
|
||||
const U32 btLow = btMask >= current ? 0 : current - btMask;
|
||||
U32* smallerPtr = bt + 2*(current&btMask);
|
||||
U32* largerPtr = smallerPtr + 1;
|
||||
U32 dummy32; /* to be nullified at the end */
|
||||
U32 const windowLow = zc->lowLimit;
|
||||
U32 matchEndIdx = current+8;
|
||||
size_t bestLength = 8;
|
||||
#ifdef ZSTD_C_PREDICT
|
||||
U32 predictedSmall = *(bt + 2*((current-1)&btMask) + 0);
|
||||
U32 predictedLarge = *(bt + 2*((current-1)&btMask) + 1);
|
||||
predictedSmall += (predictedSmall>0);
|
||||
predictedLarge += (predictedLarge>0);
|
||||
#endif /* ZSTD_C_PREDICT */
|
||||
|
||||
assert(ip <= iend-8); /* required for h calculation */
|
||||
hashTable[h] = current; /* Update Hash Table */
|
||||
|
||||
while (nbCompares-- && (matchIndex > windowLow)) {
|
||||
U32* const nextPtr = bt + 2*(matchIndex & btMask);
|
||||
size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
|
||||
|
||||
#ifdef ZSTD_C_PREDICT /* note : can create issues when hlog small <= 11 */
|
||||
const U32* predictPtr = bt + 2*((matchIndex-1) & btMask); /* written this way, as bt is a roll buffer */
|
||||
if (matchIndex == predictedSmall) {
|
||||
/* no need to check length, result known */
|
||||
*smallerPtr = matchIndex;
|
||||
if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */
|
||||
smallerPtr = nextPtr+1; /* new "smaller" => larger of match */
|
||||
matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
|
||||
predictedSmall = predictPtr[1] + (predictPtr[1]>0);
|
||||
continue;
|
||||
}
|
||||
if (matchIndex == predictedLarge) {
|
||||
*largerPtr = matchIndex;
|
||||
if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */
|
||||
largerPtr = nextPtr;
|
||||
matchIndex = nextPtr[0];
|
||||
predictedLarge = predictPtr[0] + (predictPtr[0]>0);
|
||||
continue;
|
||||
}
|
||||
#endif
|
||||
if ((!extDict) || (matchIndex+matchLength >= dictLimit)) {
|
||||
match = base + matchIndex;
|
||||
if (match[matchLength] == ip[matchLength])
|
||||
matchLength += ZSTD_count(ip+matchLength+1, match+matchLength+1, iend) +1;
|
||||
} else {
|
||||
match = dictBase + matchIndex;
|
||||
matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart);
|
||||
if (matchIndex+matchLength >= dictLimit)
|
||||
match = base + matchIndex; /* to prepare for next usage of match[matchLength] */
|
||||
}
|
||||
|
||||
if (matchLength > bestLength) {
|
||||
bestLength = matchLength;
|
||||
if (matchLength > matchEndIdx - matchIndex)
|
||||
matchEndIdx = matchIndex + (U32)matchLength;
|
||||
}
|
||||
|
||||
if (ip+matchLength == iend) /* equal : no way to know if inf or sup */
|
||||
break; /* drop , to guarantee consistency ; miss a bit of compression, but other solutions can corrupt tree */
|
||||
|
||||
if (match[matchLength] < ip[matchLength]) { /* necessarily within buffer */
|
||||
/* match+1 is smaller than current */
|
||||
*smallerPtr = matchIndex; /* update smaller idx */
|
||||
commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
|
||||
if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop searching */
|
||||
smallerPtr = nextPtr+1; /* new "smaller" => larger of match */
|
||||
matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
|
||||
} else {
|
||||
/* match is larger than current */
|
||||
*largerPtr = matchIndex;
|
||||
commonLengthLarger = matchLength;
|
||||
if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop searching */
|
||||
largerPtr = nextPtr;
|
||||
matchIndex = nextPtr[0];
|
||||
} }
|
||||
|
||||
*smallerPtr = *largerPtr = 0;
|
||||
if (bestLength > 384) return MIN(192, (U32)(bestLength - 384)); /* speed optimization */
|
||||
if (matchEndIdx > current + 8) return matchEndIdx - (current + 8);
|
||||
return 1;
|
||||
}
|
||||
|
||||
|
||||
static size_t ZSTD_insertBtAndFindBestMatch (
|
||||
ZSTD_CCtx* zc,
|
||||
const BYTE* const ip, const BYTE* const iend,
|
||||
size_t* offsetPtr,
|
||||
U32 nbCompares, const U32 mls,
|
||||
U32 extDict)
|
||||
{
|
||||
U32* const hashTable = zc->hashTable;
|
||||
U32 const hashLog = zc->appliedParams.cParams.hashLog;
|
||||
size_t const h = ZSTD_hashPtr(ip, hashLog, mls);
|
||||
U32* const bt = zc->chainTable;
|
||||
U32 const btLog = zc->appliedParams.cParams.chainLog - 1;
|
||||
U32 const btMask = (1 << btLog) - 1;
|
||||
U32 matchIndex = hashTable[h];
|
||||
size_t commonLengthSmaller=0, commonLengthLarger=0;
|
||||
const BYTE* const base = zc->base;
|
||||
const BYTE* const dictBase = zc->dictBase;
|
||||
const U32 dictLimit = zc->dictLimit;
|
||||
const BYTE* const dictEnd = dictBase + dictLimit;
|
||||
const BYTE* const prefixStart = base + dictLimit;
|
||||
const U32 current = (U32)(ip-base);
|
||||
const U32 btLow = btMask >= current ? 0 : current - btMask;
|
||||
const U32 windowLow = zc->lowLimit;
|
||||
U32* smallerPtr = bt + 2*(current&btMask);
|
||||
U32* largerPtr = bt + 2*(current&btMask) + 1;
|
||||
U32 matchEndIdx = current+8;
|
||||
U32 dummy32; /* to be nullified at the end */
|
||||
size_t bestLength = 0;
|
||||
|
||||
assert(ip <= iend-8); /* required for h calculation */
|
||||
hashTable[h] = current; /* Update Hash Table */
|
||||
|
||||
while (nbCompares-- && (matchIndex > windowLow)) {
|
||||
U32* const nextPtr = bt + 2*(matchIndex & btMask);
|
||||
size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
|
||||
const BYTE* match;
|
||||
|
||||
if ((!extDict) || (matchIndex+matchLength >= dictLimit)) {
|
||||
match = base + matchIndex;
|
||||
if (match[matchLength] == ip[matchLength])
|
||||
matchLength += ZSTD_count(ip+matchLength+1, match+matchLength+1, iend) +1;
|
||||
} else {
|
||||
match = dictBase + matchIndex;
|
||||
matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart);
|
||||
if (matchIndex+matchLength >= dictLimit)
|
||||
match = base + matchIndex; /* to prepare for next usage of match[matchLength] */
|
||||
}
|
||||
|
||||
if (matchLength > bestLength) {
|
||||
if (matchLength > matchEndIdx - matchIndex)
|
||||
matchEndIdx = matchIndex + (U32)matchLength;
|
||||
if ( (4*(int)(matchLength-bestLength)) > (int)(ZSTD_highbit32(current-matchIndex+1) - ZSTD_highbit32((U32)offsetPtr[0]+1)) )
|
||||
bestLength = matchLength, *offsetPtr = ZSTD_REP_MOVE + current - matchIndex;
|
||||
if (ip+matchLength == iend) /* equal : no way to know if inf or sup */
|
||||
break; /* drop, to guarantee consistency (miss a little bit of compression) */
|
||||
}
|
||||
|
||||
if (match[matchLength] < ip[matchLength]) {
|
||||
/* match is smaller than current */
|
||||
*smallerPtr = matchIndex; /* update smaller idx */
|
||||
commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
|
||||
if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */
|
||||
smallerPtr = nextPtr+1; /* new "smaller" => larger of match */
|
||||
matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
|
||||
} else {
|
||||
/* match is larger than current */
|
||||
*largerPtr = matchIndex;
|
||||
commonLengthLarger = matchLength;
|
||||
if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */
|
||||
largerPtr = nextPtr;
|
||||
matchIndex = nextPtr[0];
|
||||
} }
|
||||
|
||||
*smallerPtr = *largerPtr = 0;
|
||||
|
||||
zc->nextToUpdate = (matchEndIdx > current + 8) ? matchEndIdx - 8 : current+1;
|
||||
return bestLength;
|
||||
}
|
||||
|
||||
|
||||
void ZSTD_updateTree(ZSTD_CCtx* zc, const BYTE* const ip, const BYTE* const iend, const U32 nbCompares, const U32 mls)
|
||||
{
|
||||
const BYTE* const base = zc->base;
|
||||
const U32 target = (U32)(ip - base);
|
||||
U32 idx = zc->nextToUpdate;
|
||||
|
||||
while(idx < target)
|
||||
idx += ZSTD_insertBt1(zc, base+idx, mls, iend, nbCompares, 0);
|
||||
}
|
||||
|
||||
/** ZSTD_BtFindBestMatch() : Tree updater, providing best match */
|
||||
static size_t ZSTD_BtFindBestMatch (
|
||||
ZSTD_CCtx* zc,
|
||||
const BYTE* const ip, const BYTE* const iLimit,
|
||||
size_t* offsetPtr,
|
||||
const U32 maxNbAttempts, const U32 mls)
|
||||
{
|
||||
if (ip < zc->base + zc->nextToUpdate) return 0; /* skipped area */
|
||||
ZSTD_updateTree(zc, ip, iLimit, maxNbAttempts, mls);
|
||||
return ZSTD_insertBtAndFindBestMatch(zc, ip, iLimit, offsetPtr, maxNbAttempts, mls, 0);
|
||||
}
|
||||
|
||||
|
||||
static size_t ZSTD_BtFindBestMatch_selectMLS (
|
||||
ZSTD_CCtx* zc, /* Index table will be updated */
|
||||
const BYTE* ip, const BYTE* const iLimit,
|
||||
size_t* offsetPtr,
|
||||
const U32 maxNbAttempts, const U32 matchLengthSearch)
|
||||
{
|
||||
switch(matchLengthSearch)
|
||||
{
|
||||
default : /* includes case 3 */
|
||||
case 4 : return ZSTD_BtFindBestMatch(zc, ip, iLimit, offsetPtr, maxNbAttempts, 4);
|
||||
case 5 : return ZSTD_BtFindBestMatch(zc, ip, iLimit, offsetPtr, maxNbAttempts, 5);
|
||||
case 7 :
|
||||
case 6 : return ZSTD_BtFindBestMatch(zc, ip, iLimit, offsetPtr, maxNbAttempts, 6);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
void ZSTD_updateTree_extDict(ZSTD_CCtx* zc, const BYTE* const ip, const BYTE* const iend, const U32 nbCompares, const U32 mls)
|
||||
{
|
||||
const BYTE* const base = zc->base;
|
||||
const U32 target = (U32)(ip - base);
|
||||
U32 idx = zc->nextToUpdate;
|
||||
|
||||
while (idx < target) idx += ZSTD_insertBt1(zc, base+idx, mls, iend, nbCompares, 1);
|
||||
}
|
||||
|
||||
|
||||
/** Tree updater, providing best match */
|
||||
static size_t ZSTD_BtFindBestMatch_extDict (
|
||||
ZSTD_CCtx* zc,
|
||||
const BYTE* const ip, const BYTE* const iLimit,
|
||||
size_t* offsetPtr,
|
||||
const U32 maxNbAttempts, const U32 mls)
|
||||
{
|
||||
if (ip < zc->base + zc->nextToUpdate) return 0; /* skipped area */
|
||||
ZSTD_updateTree_extDict(zc, ip, iLimit, maxNbAttempts, mls);
|
||||
return ZSTD_insertBtAndFindBestMatch(zc, ip, iLimit, offsetPtr, maxNbAttempts, mls, 1);
|
||||
}
|
||||
|
||||
|
||||
static size_t ZSTD_BtFindBestMatch_selectMLS_extDict (
|
||||
ZSTD_CCtx* zc, /* Index table will be updated */
|
||||
const BYTE* ip, const BYTE* const iLimit,
|
||||
size_t* offsetPtr,
|
||||
const U32 maxNbAttempts, const U32 matchLengthSearch)
|
||||
{
|
||||
switch(matchLengthSearch)
|
||||
{
|
||||
default : /* includes case 3 */
|
||||
case 4 : return ZSTD_BtFindBestMatch_extDict(zc, ip, iLimit, offsetPtr, maxNbAttempts, 4);
|
||||
case 5 : return ZSTD_BtFindBestMatch_extDict(zc, ip, iLimit, offsetPtr, maxNbAttempts, 5);
|
||||
case 7 :
|
||||
case 6 : return ZSTD_BtFindBestMatch_extDict(zc, ip, iLimit, offsetPtr, maxNbAttempts, 6);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/* *********************************
|
||||
* Hash Chain
|
||||
***********************************/
|
||||
#define NEXT_IN_CHAIN(d, mask) chainTable[(d) & mask]
|
||||
|
||||
/* Update chains up to ip (excluded)
|
||||
Assumption : always within prefix (i.e. not within extDict) */
|
||||
U32 ZSTD_insertAndFindFirstIndex (ZSTD_CCtx* zc, const BYTE* ip, U32 mls)
|
||||
{
|
||||
U32* const hashTable = zc->hashTable;
|
||||
const U32 hashLog = zc->appliedParams.cParams.hashLog;
|
||||
U32* const chainTable = zc->chainTable;
|
||||
const U32 chainMask = (1 << zc->appliedParams.cParams.chainLog) - 1;
|
||||
const BYTE* const base = zc->base;
|
||||
const U32 target = (U32)(ip - base);
|
||||
U32 idx = zc->nextToUpdate;
|
||||
|
||||
while(idx < target) { /* catch up */
|
||||
size_t const h = ZSTD_hashPtr(base+idx, hashLog, mls);
|
||||
NEXT_IN_CHAIN(idx, chainMask) = hashTable[h];
|
||||
hashTable[h] = idx;
|
||||
idx++;
|
||||
}
|
||||
|
||||
zc->nextToUpdate = target;
|
||||
return hashTable[ZSTD_hashPtr(ip, hashLog, mls)];
|
||||
}
|
||||
|
||||
|
||||
/* inlining is important to hardwire a hot branch (template emulation) */
|
||||
FORCE_INLINE_TEMPLATE
|
||||
size_t ZSTD_HcFindBestMatch_generic (
|
||||
ZSTD_CCtx* zc, /* Index table will be updated */
|
||||
const BYTE* const ip, const BYTE* const iLimit,
|
||||
size_t* offsetPtr,
|
||||
const U32 maxNbAttempts, const U32 mls, const U32 extDict)
|
||||
{
|
||||
U32* const chainTable = zc->chainTable;
|
||||
const U32 chainSize = (1 << zc->appliedParams.cParams.chainLog);
|
||||
const U32 chainMask = chainSize-1;
|
||||
const BYTE* const base = zc->base;
|
||||
const BYTE* const dictBase = zc->dictBase;
|
||||
const U32 dictLimit = zc->dictLimit;
|
||||
const BYTE* const prefixStart = base + dictLimit;
|
||||
const BYTE* const dictEnd = dictBase + dictLimit;
|
||||
const U32 lowLimit = zc->lowLimit;
|
||||
const U32 current = (U32)(ip-base);
|
||||
const U32 minChain = current > chainSize ? current - chainSize : 0;
|
||||
int nbAttempts=maxNbAttempts;
|
||||
size_t ml=4-1;
|
||||
|
||||
/* HC4 match finder */
|
||||
U32 matchIndex = ZSTD_insertAndFindFirstIndex (zc, ip, mls);
|
||||
|
||||
for ( ; (matchIndex>lowLimit) & (nbAttempts>0) ; nbAttempts--) {
|
||||
const BYTE* match;
|
||||
size_t currentMl=0;
|
||||
if ((!extDict) || matchIndex >= dictLimit) {
|
||||
match = base + matchIndex;
|
||||
if (match[ml] == ip[ml]) /* potentially better */
|
||||
currentMl = ZSTD_count(ip, match, iLimit);
|
||||
} else {
|
||||
match = dictBase + matchIndex;
|
||||
if (MEM_read32(match) == MEM_read32(ip)) /* assumption : matchIndex <= dictLimit-4 (by table construction) */
|
||||
currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, dictEnd, prefixStart) + 4;
|
||||
}
|
||||
|
||||
/* save best solution */
|
||||
if (currentMl > ml) {
|
||||
ml = currentMl;
|
||||
*offsetPtr = current - matchIndex + ZSTD_REP_MOVE;
|
||||
if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */
|
||||
}
|
||||
|
||||
if (matchIndex <= minChain) break;
|
||||
matchIndex = NEXT_IN_CHAIN(matchIndex, chainMask);
|
||||
}
|
||||
|
||||
return ml;
|
||||
}
|
||||
|
||||
|
||||
FORCE_INLINE_TEMPLATE size_t ZSTD_HcFindBestMatch_selectMLS (
|
||||
ZSTD_CCtx* zc,
|
||||
const BYTE* ip, const BYTE* const iLimit,
|
||||
size_t* offsetPtr,
|
||||
const U32 maxNbAttempts, const U32 matchLengthSearch)
|
||||
{
|
||||
switch(matchLengthSearch)
|
||||
{
|
||||
default : /* includes case 3 */
|
||||
case 4 : return ZSTD_HcFindBestMatch_generic(zc, ip, iLimit, offsetPtr, maxNbAttempts, 4, 0);
|
||||
case 5 : return ZSTD_HcFindBestMatch_generic(zc, ip, iLimit, offsetPtr, maxNbAttempts, 5, 0);
|
||||
case 7 :
|
||||
case 6 : return ZSTD_HcFindBestMatch_generic(zc, ip, iLimit, offsetPtr, maxNbAttempts, 6, 0);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
FORCE_INLINE_TEMPLATE size_t ZSTD_HcFindBestMatch_extDict_selectMLS (
|
||||
ZSTD_CCtx* zc,
|
||||
const BYTE* ip, const BYTE* const iLimit,
|
||||
size_t* offsetPtr,
|
||||
const U32 maxNbAttempts, const U32 matchLengthSearch)
|
||||
{
|
||||
switch(matchLengthSearch)
|
||||
{
|
||||
default : /* includes case 3 */
|
||||
case 4 : return ZSTD_HcFindBestMatch_generic(zc, ip, iLimit, offsetPtr, maxNbAttempts, 4, 1);
|
||||
case 5 : return ZSTD_HcFindBestMatch_generic(zc, ip, iLimit, offsetPtr, maxNbAttempts, 5, 1);
|
||||
case 7 :
|
||||
case 6 : return ZSTD_HcFindBestMatch_generic(zc, ip, iLimit, offsetPtr, maxNbAttempts, 6, 1);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/* *******************************
|
||||
* Common parser - lazy strategy
|
||||
*********************************/
|
||||
FORCE_INLINE_TEMPLATE
|
||||
size_t ZSTD_compressBlock_lazy_generic(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize,
|
||||
const U32 searchMethod, const U32 depth)
|
||||
{
|
||||
seqStore_t* seqStorePtr = &(ctx->seqStore);
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - 8;
|
||||
const BYTE* const base = ctx->base + ctx->dictLimit;
|
||||
|
||||
U32 const maxSearches = 1 << ctx->appliedParams.cParams.searchLog;
|
||||
U32 const mls = ctx->appliedParams.cParams.searchLength;
|
||||
|
||||
typedef size_t (*searchMax_f)(ZSTD_CCtx* zc, const BYTE* ip, const BYTE* iLimit,
|
||||
size_t* offsetPtr,
|
||||
U32 maxNbAttempts, U32 matchLengthSearch);
|
||||
searchMax_f const searchMax = searchMethod ? ZSTD_BtFindBestMatch_selectMLS : ZSTD_HcFindBestMatch_selectMLS;
|
||||
U32 offset_1 = seqStorePtr->rep[0], offset_2 = seqStorePtr->rep[1], savedOffset=0;
|
||||
|
||||
/* init */
|
||||
ip += (ip==base);
|
||||
ctx->nextToUpdate3 = ctx->nextToUpdate;
|
||||
{ U32 const maxRep = (U32)(ip-base);
|
||||
if (offset_2 > maxRep) savedOffset = offset_2, offset_2 = 0;
|
||||
if (offset_1 > maxRep) savedOffset = offset_1, offset_1 = 0;
|
||||
}
|
||||
|
||||
/* Match Loop */
|
||||
while (ip < ilimit) {
|
||||
size_t matchLength=0;
|
||||
size_t offset=0;
|
||||
const BYTE* start=ip+1;
|
||||
|
||||
/* check repCode */
|
||||
if ((offset_1>0) & (MEM_read32(ip+1) == MEM_read32(ip+1 - offset_1))) {
|
||||
/* repcode : we take it */
|
||||
matchLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4;
|
||||
if (depth==0) goto _storeSequence;
|
||||
}
|
||||
|
||||
/* first search (depth 0) */
|
||||
{ size_t offsetFound = 99999999;
|
||||
size_t const ml2 = searchMax(ctx, ip, iend, &offsetFound, maxSearches, mls);
|
||||
if (ml2 > matchLength)
|
||||
matchLength = ml2, start = ip, offset=offsetFound;
|
||||
}
|
||||
|
||||
if (matchLength < 4) {
|
||||
ip += ((ip-anchor) >> g_searchStrength) + 1; /* jump faster over incompressible sections */
|
||||
continue;
|
||||
}
|
||||
|
||||
/* let's try to find a better solution */
|
||||
if (depth>=1)
|
||||
while (ip<ilimit) {
|
||||
ip ++;
|
||||
if ((offset) && ((offset_1>0) & (MEM_read32(ip) == MEM_read32(ip - offset_1)))) {
|
||||
size_t const mlRep = ZSTD_count(ip+4, ip+4-offset_1, iend) + 4;
|
||||
int const gain2 = (int)(mlRep * 3);
|
||||
int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offset+1) + 1);
|
||||
if ((mlRep >= 4) && (gain2 > gain1))
|
||||
matchLength = mlRep, offset = 0, start = ip;
|
||||
}
|
||||
{ size_t offset2=99999999;
|
||||
size_t const ml2 = searchMax(ctx, ip, iend, &offset2, maxSearches, mls);
|
||||
int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)offset2+1)); /* raw approx */
|
||||
int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 4);
|
||||
if ((ml2 >= 4) && (gain2 > gain1)) {
|
||||
matchLength = ml2, offset = offset2, start = ip;
|
||||
continue; /* search a better one */
|
||||
} }
|
||||
|
||||
/* let's find an even better one */
|
||||
if ((depth==2) && (ip<ilimit)) {
|
||||
ip ++;
|
||||
if ((offset) && ((offset_1>0) & (MEM_read32(ip) == MEM_read32(ip - offset_1)))) {
|
||||
size_t const ml2 = ZSTD_count(ip+4, ip+4-offset_1, iend) + 4;
|
||||
int const gain2 = (int)(ml2 * 4);
|
||||
int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 1);
|
||||
if ((ml2 >= 4) && (gain2 > gain1))
|
||||
matchLength = ml2, offset = 0, start = ip;
|
||||
}
|
||||
{ size_t offset2=99999999;
|
||||
size_t const ml2 = searchMax(ctx, ip, iend, &offset2, maxSearches, mls);
|
||||
int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)offset2+1)); /* raw approx */
|
||||
int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 7);
|
||||
if ((ml2 >= 4) && (gain2 > gain1)) {
|
||||
matchLength = ml2, offset = offset2, start = ip;
|
||||
continue;
|
||||
} } }
|
||||
break; /* nothing found : store previous solution */
|
||||
}
|
||||
|
||||
/* NOTE:
|
||||
* start[-offset+ZSTD_REP_MOVE-1] is undefined behavior.
|
||||
* (-offset+ZSTD_REP_MOVE-1) is unsigned, and is added to start, which
|
||||
* overflows the pointer, which is undefined behavior.
|
||||
*/
|
||||
/* catch up */
|
||||
if (offset) {
|
||||
while ( (start > anchor)
|
||||
&& (start > base+offset-ZSTD_REP_MOVE)
|
||||
&& (start[-1] == (start-offset+ZSTD_REP_MOVE)[-1]) ) /* only search for offset within prefix */
|
||||
{ start--; matchLength++; }
|
||||
offset_2 = offset_1; offset_1 = (U32)(offset - ZSTD_REP_MOVE);
|
||||
}
|
||||
/* store sequence */
|
||||
_storeSequence:
|
||||
{ size_t const litLength = start - anchor;
|
||||
ZSTD_storeSeq(seqStorePtr, litLength, anchor, (U32)offset, matchLength-MINMATCH);
|
||||
anchor = ip = start + matchLength;
|
||||
}
|
||||
|
||||
/* check immediate repcode */
|
||||
while ( (ip <= ilimit)
|
||||
&& ((offset_2>0)
|
||||
& (MEM_read32(ip) == MEM_read32(ip - offset_2)) )) {
|
||||
/* store sequence */
|
||||
matchLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4;
|
||||
offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap repcodes */
|
||||
ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, matchLength-MINMATCH);
|
||||
ip += matchLength;
|
||||
anchor = ip;
|
||||
continue; /* faster when present ... (?) */
|
||||
} }
|
||||
|
||||
/* Save reps for next block */
|
||||
seqStorePtr->repToConfirm[0] = offset_1 ? offset_1 : savedOffset;
|
||||
seqStorePtr->repToConfirm[1] = offset_2 ? offset_2 : savedOffset;
|
||||
|
||||
/* Return the last literals size */
|
||||
return iend - anchor;
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTD_compressBlock_btlazy2(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 1, 2);
|
||||
}
|
||||
|
||||
size_t ZSTD_compressBlock_lazy2(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 2);
|
||||
}
|
||||
|
||||
size_t ZSTD_compressBlock_lazy(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 1);
|
||||
}
|
||||
|
||||
size_t ZSTD_compressBlock_greedy(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 0);
|
||||
}
|
||||
|
||||
|
||||
FORCE_INLINE_TEMPLATE
|
||||
size_t ZSTD_compressBlock_lazy_extDict_generic(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize,
|
||||
const U32 searchMethod, const U32 depth)
|
||||
{
|
||||
seqStore_t* seqStorePtr = &(ctx->seqStore);
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - 8;
|
||||
const BYTE* const base = ctx->base;
|
||||
const U32 dictLimit = ctx->dictLimit;
|
||||
const U32 lowestIndex = ctx->lowLimit;
|
||||
const BYTE* const prefixStart = base + dictLimit;
|
||||
const BYTE* const dictBase = ctx->dictBase;
|
||||
const BYTE* const dictEnd = dictBase + dictLimit;
|
||||
const BYTE* const dictStart = dictBase + ctx->lowLimit;
|
||||
|
||||
const U32 maxSearches = 1 << ctx->appliedParams.cParams.searchLog;
|
||||
const U32 mls = ctx->appliedParams.cParams.searchLength;
|
||||
|
||||
typedef size_t (*searchMax_f)(ZSTD_CCtx* zc, const BYTE* ip, const BYTE* iLimit,
|
||||
size_t* offsetPtr,
|
||||
U32 maxNbAttempts, U32 matchLengthSearch);
|
||||
searchMax_f searchMax = searchMethod ? ZSTD_BtFindBestMatch_selectMLS_extDict : ZSTD_HcFindBestMatch_extDict_selectMLS;
|
||||
|
||||
U32 offset_1 = seqStorePtr->rep[0], offset_2 = seqStorePtr->rep[1];
|
||||
|
||||
/* init */
|
||||
ctx->nextToUpdate3 = ctx->nextToUpdate;
|
||||
ip += (ip == prefixStart);
|
||||
|
||||
/* Match Loop */
|
||||
while (ip < ilimit) {
|
||||
size_t matchLength=0;
|
||||
size_t offset=0;
|
||||
const BYTE* start=ip+1;
|
||||
U32 current = (U32)(ip-base);
|
||||
|
||||
/* check repCode */
|
||||
{ const U32 repIndex = (U32)(current+1 - offset_1);
|
||||
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* const repMatch = repBase + repIndex;
|
||||
if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > lowestIndex)) /* intentional overflow */
|
||||
if (MEM_read32(ip+1) == MEM_read32(repMatch)) {
|
||||
/* repcode detected we should take it */
|
||||
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
|
||||
matchLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repEnd, prefixStart) + 4;
|
||||
if (depth==0) goto _storeSequence;
|
||||
} }
|
||||
|
||||
/* first search (depth 0) */
|
||||
{ size_t offsetFound = 99999999;
|
||||
size_t const ml2 = searchMax(ctx, ip, iend, &offsetFound, maxSearches, mls);
|
||||
if (ml2 > matchLength)
|
||||
matchLength = ml2, start = ip, offset=offsetFound;
|
||||
}
|
||||
|
||||
if (matchLength < 4) {
|
||||
ip += ((ip-anchor) >> g_searchStrength) + 1; /* jump faster over incompressible sections */
|
||||
continue;
|
||||
}
|
||||
|
||||
/* let's try to find a better solution */
|
||||
if (depth>=1)
|
||||
while (ip<ilimit) {
|
||||
ip ++;
|
||||
current++;
|
||||
/* check repCode */
|
||||
if (offset) {
|
||||
const U32 repIndex = (U32)(current - offset_1);
|
||||
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* const repMatch = repBase + repIndex;
|
||||
if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > lowestIndex)) /* intentional overflow */
|
||||
if (MEM_read32(ip) == MEM_read32(repMatch)) {
|
||||
/* repcode detected */
|
||||
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
|
||||
size_t const repLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
|
||||
int const gain2 = (int)(repLength * 3);
|
||||
int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offset+1) + 1);
|
||||
if ((repLength >= 4) && (gain2 > gain1))
|
||||
matchLength = repLength, offset = 0, start = ip;
|
||||
} }
|
||||
|
||||
/* search match, depth 1 */
|
||||
{ size_t offset2=99999999;
|
||||
size_t const ml2 = searchMax(ctx, ip, iend, &offset2, maxSearches, mls);
|
||||
int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)offset2+1)); /* raw approx */
|
||||
int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 4);
|
||||
if ((ml2 >= 4) && (gain2 > gain1)) {
|
||||
matchLength = ml2, offset = offset2, start = ip;
|
||||
continue; /* search a better one */
|
||||
} }
|
||||
|
||||
/* let's find an even better one */
|
||||
if ((depth==2) && (ip<ilimit)) {
|
||||
ip ++;
|
||||
current++;
|
||||
/* check repCode */
|
||||
if (offset) {
|
||||
const U32 repIndex = (U32)(current - offset_1);
|
||||
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* const repMatch = repBase + repIndex;
|
||||
if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > lowestIndex)) /* intentional overflow */
|
||||
if (MEM_read32(ip) == MEM_read32(repMatch)) {
|
||||
/* repcode detected */
|
||||
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
|
||||
size_t const repLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
|
||||
int const gain2 = (int)(repLength * 4);
|
||||
int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 1);
|
||||
if ((repLength >= 4) && (gain2 > gain1))
|
||||
matchLength = repLength, offset = 0, start = ip;
|
||||
} }
|
||||
|
||||
/* search match, depth 2 */
|
||||
{ size_t offset2=99999999;
|
||||
size_t const ml2 = searchMax(ctx, ip, iend, &offset2, maxSearches, mls);
|
||||
int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)offset2+1)); /* raw approx */
|
||||
int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 7);
|
||||
if ((ml2 >= 4) && (gain2 > gain1)) {
|
||||
matchLength = ml2, offset = offset2, start = ip;
|
||||
continue;
|
||||
} } }
|
||||
break; /* nothing found : store previous solution */
|
||||
}
|
||||
|
||||
/* catch up */
|
||||
if (offset) {
|
||||
U32 const matchIndex = (U32)((start-base) - (offset - ZSTD_REP_MOVE));
|
||||
const BYTE* match = (matchIndex < dictLimit) ? dictBase + matchIndex : base + matchIndex;
|
||||
const BYTE* const mStart = (matchIndex < dictLimit) ? dictStart : prefixStart;
|
||||
while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */
|
||||
offset_2 = offset_1; offset_1 = (U32)(offset - ZSTD_REP_MOVE);
|
||||
}
|
||||
|
||||
/* store sequence */
|
||||
_storeSequence:
|
||||
{ size_t const litLength = start - anchor;
|
||||
ZSTD_storeSeq(seqStorePtr, litLength, anchor, (U32)offset, matchLength-MINMATCH);
|
||||
anchor = ip = start + matchLength;
|
||||
}
|
||||
|
||||
/* check immediate repcode */
|
||||
while (ip <= ilimit) {
|
||||
const U32 repIndex = (U32)((ip-base) - offset_2);
|
||||
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* const repMatch = repBase + repIndex;
|
||||
if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > lowestIndex)) /* intentional overflow */
|
||||
if (MEM_read32(ip) == MEM_read32(repMatch)) {
|
||||
/* repcode detected we should take it */
|
||||
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
|
||||
matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
|
||||
offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap offset history */
|
||||
ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, matchLength-MINMATCH);
|
||||
ip += matchLength;
|
||||
anchor = ip;
|
||||
continue; /* faster when present ... (?) */
|
||||
}
|
||||
break;
|
||||
} }
|
||||
|
||||
/* Save reps for next block */
|
||||
seqStorePtr->repToConfirm[0] = offset_1; seqStorePtr->repToConfirm[1] = offset_2;
|
||||
|
||||
/* Return the last literals size */
|
||||
return iend - anchor;
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTD_compressBlock_greedy_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_lazy_extDict_generic(ctx, src, srcSize, 0, 0);
|
||||
}
|
||||
|
||||
size_t ZSTD_compressBlock_lazy_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_lazy_extDict_generic(ctx, src, srcSize, 0, 1);
|
||||
}
|
||||
|
||||
size_t ZSTD_compressBlock_lazy2_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_lazy_extDict_generic(ctx, src, srcSize, 0, 2);
|
||||
}
|
||||
|
||||
size_t ZSTD_compressBlock_btlazy2_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_lazy_extDict_generic(ctx, src, srcSize, 1, 2);
|
||||
}
|
38
lib/compress/zstd_lazy.h
Normal file
38
lib/compress/zstd_lazy.h
Normal file
@ -0,0 +1,38 @@
|
||||
/*
|
||||
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef ZSTD_LAZY_H
|
||||
#define ZSTD_LAZY_H
|
||||
|
||||
#include "zstd_compress.h"
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
U32 ZSTD_insertAndFindFirstIndex (ZSTD_CCtx* zc, const BYTE* ip, U32 mls);
|
||||
void ZSTD_updateTree(ZSTD_CCtx* zc, const BYTE* const ip, const BYTE* const iend, const U32 nbCompares, const U32 mls);
|
||||
void ZSTD_updateTree_extDict(ZSTD_CCtx* zc, const BYTE* const ip, const BYTE* const iend, const U32 nbCompares, const U32 mls);
|
||||
|
||||
size_t ZSTD_compressBlock_btlazy2(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
size_t ZSTD_compressBlock_lazy2(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
size_t ZSTD_compressBlock_lazy(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
size_t ZSTD_compressBlock_greedy(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
|
||||
size_t ZSTD_compressBlock_greedy_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
size_t ZSTD_compressBlock_lazy_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
size_t ZSTD_compressBlock_lazy2_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
size_t ZSTD_compressBlock_btlazy2_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* ZSTD_LAZY_H */
|
707
lib/compress/zstd_ldm.c
Normal file
707
lib/compress/zstd_ldm.c
Normal file
@ -0,0 +1,707 @@
|
||||
/*
|
||||
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
#include "zstd_ldm.h"
|
||||
|
||||
#include "zstd_fast.h" /* ZSTD_fillHashTable() */
|
||||
#include "zstd_double_fast.h" /* ZSTD_fillDoubleHashTable() */
|
||||
|
||||
#define LDM_BUCKET_SIZE_LOG 3
|
||||
#define LDM_MIN_MATCH_LENGTH 64
|
||||
#define LDM_HASH_RLOG 7
|
||||
#define LDM_HASH_CHAR_OFFSET 10
|
||||
|
||||
size_t ZSTD_ldm_initializeParameters(ldmParams_t* params, U32 enableLdm)
|
||||
{
|
||||
ZSTD_STATIC_ASSERT(LDM_BUCKET_SIZE_LOG <= ZSTD_LDM_BUCKETSIZELOG_MAX);
|
||||
params->enableLdm = enableLdm>0;
|
||||
params->hashLog = 0;
|
||||
params->bucketSizeLog = LDM_BUCKET_SIZE_LOG;
|
||||
params->minMatchLength = LDM_MIN_MATCH_LENGTH;
|
||||
params->hashEveryLog = ZSTD_LDM_HASHEVERYLOG_NOTSET;
|
||||
return 0;
|
||||
}
|
||||
|
||||
void ZSTD_ldm_adjustParameters(ldmParams_t* params, U32 windowLog)
|
||||
{
|
||||
if (params->hashLog == 0) {
|
||||
params->hashLog = MAX(ZSTD_HASHLOG_MIN, windowLog - LDM_HASH_RLOG);
|
||||
assert(params->hashLog <= ZSTD_HASHLOG_MAX);
|
||||
}
|
||||
if (params->hashEveryLog == ZSTD_LDM_HASHEVERYLOG_NOTSET) {
|
||||
params->hashEveryLog =
|
||||
windowLog < params->hashLog ? 0 : windowLog - params->hashLog;
|
||||
}
|
||||
params->bucketSizeLog = MIN(params->bucketSizeLog, params->hashLog);
|
||||
}
|
||||
|
||||
size_t ZSTD_ldm_getTableSize(U32 hashLog, U32 bucketSizeLog) {
|
||||
size_t const ldmHSize = ((size_t)1) << hashLog;
|
||||
size_t const ldmBucketSizeLog = MIN(bucketSizeLog, hashLog);
|
||||
size_t const ldmBucketSize =
|
||||
((size_t)1) << (hashLog - ldmBucketSizeLog);
|
||||
return ldmBucketSize + (ldmHSize * (sizeof(ldmEntry_t)));
|
||||
}
|
||||
|
||||
/** ZSTD_ldm_getSmallHash() :
|
||||
* numBits should be <= 32
|
||||
* If numBits==0, returns 0.
|
||||
* @return : the most significant numBits of value. */
|
||||
static U32 ZSTD_ldm_getSmallHash(U64 value, U32 numBits)
|
||||
{
|
||||
assert(numBits <= 32);
|
||||
return numBits == 0 ? 0 : (U32)(value >> (64 - numBits));
|
||||
}
|
||||
|
||||
/** ZSTD_ldm_getChecksum() :
|
||||
* numBitsToDiscard should be <= 32
|
||||
* @return : the next most significant 32 bits after numBitsToDiscard */
|
||||
static U32 ZSTD_ldm_getChecksum(U64 hash, U32 numBitsToDiscard)
|
||||
{
|
||||
assert(numBitsToDiscard <= 32);
|
||||
return (hash >> (64 - 32 - numBitsToDiscard)) & 0xFFFFFFFF;
|
||||
}
|
||||
|
||||
/** ZSTD_ldm_getTag() ;
|
||||
* Given the hash, returns the most significant numTagBits bits
|
||||
* after (32 + hbits) bits.
|
||||
*
|
||||
* If there are not enough bits remaining, return the last
|
||||
* numTagBits bits. */
|
||||
static U32 ZSTD_ldm_getTag(U64 hash, U32 hbits, U32 numTagBits)
|
||||
{
|
||||
assert(numTagBits < 32 && hbits <= 32);
|
||||
if (32 - hbits < numTagBits) {
|
||||
return hash & (((U32)1 << numTagBits) - 1);
|
||||
} else {
|
||||
return (hash >> (32 - hbits - numTagBits)) & (((U32)1 << numTagBits) - 1);
|
||||
}
|
||||
}
|
||||
|
||||
/** ZSTD_ldm_getBucket() :
|
||||
* Returns a pointer to the start of the bucket associated with hash. */
|
||||
static ldmEntry_t* ZSTD_ldm_getBucket(
|
||||
ldmState_t* ldmState, size_t hash, ldmParams_t const ldmParams)
|
||||
{
|
||||
return ldmState->hashTable + (hash << ldmParams.bucketSizeLog);
|
||||
}
|
||||
|
||||
/** ZSTD_ldm_insertEntry() :
|
||||
* Insert the entry with corresponding hash into the hash table */
|
||||
static void ZSTD_ldm_insertEntry(ldmState_t* ldmState,
|
||||
size_t const hash, const ldmEntry_t entry,
|
||||
ldmParams_t const ldmParams)
|
||||
{
|
||||
BYTE* const bucketOffsets = ldmState->bucketOffsets;
|
||||
*(ZSTD_ldm_getBucket(ldmState, hash, ldmParams) + bucketOffsets[hash]) = entry;
|
||||
bucketOffsets[hash]++;
|
||||
bucketOffsets[hash] &= ((U32)1 << ldmParams.bucketSizeLog) - 1;
|
||||
}
|
||||
|
||||
/** ZSTD_ldm_makeEntryAndInsertByTag() :
|
||||
*
|
||||
* Gets the small hash, checksum, and tag from the rollingHash.
|
||||
*
|
||||
* If the tag matches (1 << ldmParams.hashEveryLog)-1, then
|
||||
* creates an ldmEntry from the offset, and inserts it into the hash table.
|
||||
*
|
||||
* hBits is the length of the small hash, which is the most significant hBits
|
||||
* of rollingHash. The checksum is the next 32 most significant bits, followed
|
||||
* by ldmParams.hashEveryLog bits that make up the tag. */
|
||||
static void ZSTD_ldm_makeEntryAndInsertByTag(ldmState_t* ldmState,
|
||||
U64 const rollingHash,
|
||||
U32 const hBits,
|
||||
U32 const offset,
|
||||
ldmParams_t const ldmParams)
|
||||
{
|
||||
U32 const tag = ZSTD_ldm_getTag(rollingHash, hBits, ldmParams.hashEveryLog);
|
||||
U32 const tagMask = ((U32)1 << ldmParams.hashEveryLog) - 1;
|
||||
if (tag == tagMask) {
|
||||
U32 const hash = ZSTD_ldm_getSmallHash(rollingHash, hBits);
|
||||
U32 const checksum = ZSTD_ldm_getChecksum(rollingHash, hBits);
|
||||
ldmEntry_t entry;
|
||||
entry.offset = offset;
|
||||
entry.checksum = checksum;
|
||||
ZSTD_ldm_insertEntry(ldmState, hash, entry, ldmParams);
|
||||
}
|
||||
}
|
||||
|
||||
/** ZSTD_ldm_getRollingHash() :
|
||||
* Get a 64-bit hash using the first len bytes from buf.
|
||||
*
|
||||
* Giving bytes s = s_1, s_2, ... s_k, the hash is defined to be
|
||||
* H(s) = s_1*(a^(k-1)) + s_2*(a^(k-2)) + ... + s_k*(a^0)
|
||||
*
|
||||
* where the constant a is defined to be prime8bytes.
|
||||
*
|
||||
* The implementation adds an offset to each byte, so
|
||||
* H(s) = (s_1 + HASH_CHAR_OFFSET)*(a^(k-1)) + ... */
|
||||
static U64 ZSTD_ldm_getRollingHash(const BYTE* buf, U32 len)
|
||||
{
|
||||
U64 ret = 0;
|
||||
U32 i;
|
||||
for (i = 0; i < len; i++) {
|
||||
ret *= prime8bytes;
|
||||
ret += buf[i] + LDM_HASH_CHAR_OFFSET;
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
/** ZSTD_ldm_ipow() :
|
||||
* Return base^exp. */
|
||||
static U64 ZSTD_ldm_ipow(U64 base, U64 exp)
|
||||
{
|
||||
U64 ret = 1;
|
||||
while (exp) {
|
||||
if (exp & 1) { ret *= base; }
|
||||
exp >>= 1;
|
||||
base *= base;
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
U64 ZSTD_ldm_getHashPower(U32 minMatchLength) {
|
||||
assert(minMatchLength >= ZSTD_LDM_MINMATCH_MIN);
|
||||
return ZSTD_ldm_ipow(prime8bytes, minMatchLength - 1);
|
||||
}
|
||||
|
||||
/** ZSTD_ldm_updateHash() :
|
||||
* Updates hash by removing toRemove and adding toAdd. */
|
||||
static U64 ZSTD_ldm_updateHash(U64 hash, BYTE toRemove, BYTE toAdd, U64 hashPower)
|
||||
{
|
||||
hash -= ((toRemove + LDM_HASH_CHAR_OFFSET) * hashPower);
|
||||
hash *= prime8bytes;
|
||||
hash += toAdd + LDM_HASH_CHAR_OFFSET;
|
||||
return hash;
|
||||
}
|
||||
|
||||
/** ZSTD_ldm_countBackwardsMatch() :
|
||||
* Returns the number of bytes that match backwards before pIn and pMatch.
|
||||
*
|
||||
* We count only bytes where pMatch >= pBase and pIn >= pAnchor. */
|
||||
static size_t ZSTD_ldm_countBackwardsMatch(
|
||||
const BYTE* pIn, const BYTE* pAnchor,
|
||||
const BYTE* pMatch, const BYTE* pBase)
|
||||
{
|
||||
size_t matchLength = 0;
|
||||
while (pIn > pAnchor && pMatch > pBase && pIn[-1] == pMatch[-1]) {
|
||||
pIn--;
|
||||
pMatch--;
|
||||
matchLength++;
|
||||
}
|
||||
return matchLength;
|
||||
}
|
||||
|
||||
/** ZSTD_ldm_fillFastTables() :
|
||||
*
|
||||
* Fills the relevant tables for the ZSTD_fast and ZSTD_dfast strategies.
|
||||
* This is similar to ZSTD_loadDictionaryContent.
|
||||
*
|
||||
* The tables for the other strategies are filled within their
|
||||
* block compressors. */
|
||||
static size_t ZSTD_ldm_fillFastTables(ZSTD_CCtx* zc, const void* end)
|
||||
{
|
||||
const BYTE* const iend = (const BYTE*)end;
|
||||
const U32 mls = zc->appliedParams.cParams.searchLength;
|
||||
|
||||
switch(zc->appliedParams.cParams.strategy)
|
||||
{
|
||||
case ZSTD_fast:
|
||||
ZSTD_fillHashTable(zc, iend, mls);
|
||||
zc->nextToUpdate = (U32)(iend - zc->base);
|
||||
break;
|
||||
|
||||
case ZSTD_dfast:
|
||||
ZSTD_fillDoubleHashTable(zc, iend, mls);
|
||||
zc->nextToUpdate = (U32)(iend - zc->base);
|
||||
break;
|
||||
|
||||
case ZSTD_greedy:
|
||||
case ZSTD_lazy:
|
||||
case ZSTD_lazy2:
|
||||
case ZSTD_btlazy2:
|
||||
case ZSTD_btopt:
|
||||
case ZSTD_btultra:
|
||||
break;
|
||||
default:
|
||||
assert(0); /* not possible : not a valid strategy id */
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/** ZSTD_ldm_fillLdmHashTable() :
|
||||
*
|
||||
* Fills hashTable from (lastHashed + 1) to iend (non-inclusive).
|
||||
* lastHash is the rolling hash that corresponds to lastHashed.
|
||||
*
|
||||
* Returns the rolling hash corresponding to position iend-1. */
|
||||
static U64 ZSTD_ldm_fillLdmHashTable(ldmState_t* state,
|
||||
U64 lastHash, const BYTE* lastHashed,
|
||||
const BYTE* iend, const BYTE* base,
|
||||
U32 hBits, ldmParams_t const ldmParams)
|
||||
{
|
||||
U64 rollingHash = lastHash;
|
||||
const BYTE* cur = lastHashed + 1;
|
||||
|
||||
while (cur < iend) {
|
||||
rollingHash = ZSTD_ldm_updateHash(rollingHash, cur[-1],
|
||||
cur[ldmParams.minMatchLength-1],
|
||||
state->hashPower);
|
||||
ZSTD_ldm_makeEntryAndInsertByTag(state,
|
||||
rollingHash, hBits,
|
||||
(U32)(cur - base), ldmParams);
|
||||
++cur;
|
||||
}
|
||||
return rollingHash;
|
||||
}
|
||||
|
||||
|
||||
/** ZSTD_ldm_limitTableUpdate() :
|
||||
*
|
||||
* Sets cctx->nextToUpdate to a position corresponding closer to anchor
|
||||
* if it is far way
|
||||
* (after a long match, only update tables a limited amount). */
|
||||
static void ZSTD_ldm_limitTableUpdate(ZSTD_CCtx* cctx, const BYTE* anchor)
|
||||
{
|
||||
U32 const current = (U32)(anchor - cctx->base);
|
||||
if (current > cctx->nextToUpdate + 1024) {
|
||||
cctx->nextToUpdate =
|
||||
current - MIN(512, current - cctx->nextToUpdate - 1024);
|
||||
}
|
||||
}
|
||||
|
||||
typedef size_t (*ZSTD_blockCompressor) (ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
/* defined in zstd_compress.c */
|
||||
ZSTD_blockCompressor ZSTD_selectBlockCompressor(ZSTD_strategy strat, int extDict);
|
||||
|
||||
FORCE_INLINE_TEMPLATE
|
||||
size_t ZSTD_compressBlock_ldm_generic(ZSTD_CCtx* cctx,
|
||||
const void* src, size_t srcSize)
|
||||
{
|
||||
ldmState_t* const ldmState = &(cctx->ldmState);
|
||||
const ldmParams_t ldmParams = cctx->appliedParams.ldmParams;
|
||||
const U64 hashPower = ldmState->hashPower;
|
||||
const U32 hBits = ldmParams.hashLog - ldmParams.bucketSizeLog;
|
||||
const U32 ldmBucketSize = ((U32)1 << ldmParams.bucketSizeLog);
|
||||
const U32 ldmTagMask = ((U32)1 << ldmParams.hashEveryLog) - 1;
|
||||
seqStore_t* const seqStorePtr = &(cctx->seqStore);
|
||||
const BYTE* const base = cctx->base;
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const U32 lowestIndex = cctx->dictLimit;
|
||||
const BYTE* const lowest = base + lowestIndex;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - MAX(ldmParams.minMatchLength, HASH_READ_SIZE);
|
||||
|
||||
const ZSTD_blockCompressor blockCompressor =
|
||||
ZSTD_selectBlockCompressor(cctx->appliedParams.cParams.strategy, 0);
|
||||
U32* const repToConfirm = seqStorePtr->repToConfirm;
|
||||
U32 savedRep[ZSTD_REP_NUM];
|
||||
U64 rollingHash = 0;
|
||||
const BYTE* lastHashed = NULL;
|
||||
size_t i, lastLiterals;
|
||||
|
||||
/* Save seqStorePtr->rep and copy repToConfirm */
|
||||
for (i = 0; i < ZSTD_REP_NUM; i++)
|
||||
savedRep[i] = repToConfirm[i] = seqStorePtr->rep[i];
|
||||
|
||||
/* Main Search Loop */
|
||||
while (ip < ilimit) { /* < instead of <=, because repcode check at (ip+1) */
|
||||
size_t mLength;
|
||||
U32 const current = (U32)(ip - base);
|
||||
size_t forwardMatchLength = 0, backwardMatchLength = 0;
|
||||
ldmEntry_t* bestEntry = NULL;
|
||||
if (ip != istart) {
|
||||
rollingHash = ZSTD_ldm_updateHash(rollingHash, lastHashed[0],
|
||||
lastHashed[ldmParams.minMatchLength],
|
||||
hashPower);
|
||||
} else {
|
||||
rollingHash = ZSTD_ldm_getRollingHash(ip, ldmParams.minMatchLength);
|
||||
}
|
||||
lastHashed = ip;
|
||||
|
||||
/* Do not insert and do not look for a match */
|
||||
if (ZSTD_ldm_getTag(rollingHash, hBits, ldmParams.hashEveryLog) !=
|
||||
ldmTagMask) {
|
||||
ip++;
|
||||
continue;
|
||||
}
|
||||
|
||||
/* Get the best entry and compute the match lengths */
|
||||
{
|
||||
ldmEntry_t* const bucket =
|
||||
ZSTD_ldm_getBucket(ldmState,
|
||||
ZSTD_ldm_getSmallHash(rollingHash, hBits),
|
||||
ldmParams);
|
||||
ldmEntry_t* cur;
|
||||
size_t bestMatchLength = 0;
|
||||
U32 const checksum = ZSTD_ldm_getChecksum(rollingHash, hBits);
|
||||
|
||||
for (cur = bucket; cur < bucket + ldmBucketSize; ++cur) {
|
||||
const BYTE* const pMatch = cur->offset + base;
|
||||
size_t curForwardMatchLength, curBackwardMatchLength,
|
||||
curTotalMatchLength;
|
||||
if (cur->checksum != checksum || cur->offset <= lowestIndex) {
|
||||
continue;
|
||||
}
|
||||
|
||||
curForwardMatchLength = ZSTD_count(ip, pMatch, iend);
|
||||
if (curForwardMatchLength < ldmParams.minMatchLength) {
|
||||
continue;
|
||||
}
|
||||
curBackwardMatchLength = ZSTD_ldm_countBackwardsMatch(
|
||||
ip, anchor, pMatch, lowest);
|
||||
curTotalMatchLength = curForwardMatchLength +
|
||||
curBackwardMatchLength;
|
||||
|
||||
if (curTotalMatchLength > bestMatchLength) {
|
||||
bestMatchLength = curTotalMatchLength;
|
||||
forwardMatchLength = curForwardMatchLength;
|
||||
backwardMatchLength = curBackwardMatchLength;
|
||||
bestEntry = cur;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* No match found -- continue searching */
|
||||
if (bestEntry == NULL) {
|
||||
ZSTD_ldm_makeEntryAndInsertByTag(ldmState, rollingHash,
|
||||
hBits, current,
|
||||
ldmParams);
|
||||
ip++;
|
||||
continue;
|
||||
}
|
||||
|
||||
/* Match found */
|
||||
mLength = forwardMatchLength + backwardMatchLength;
|
||||
ip -= backwardMatchLength;
|
||||
|
||||
/* Call the block compressor on the remaining literals */
|
||||
{
|
||||
U32 const matchIndex = bestEntry->offset;
|
||||
const BYTE* const match = base + matchIndex - backwardMatchLength;
|
||||
U32 const offset = (U32)(ip - match);
|
||||
|
||||
/* Overwrite rep codes */
|
||||
for (i = 0; i < ZSTD_REP_NUM; i++)
|
||||
seqStorePtr->rep[i] = repToConfirm[i];
|
||||
|
||||
/* Fill tables for block compressor */
|
||||
ZSTD_ldm_limitTableUpdate(cctx, anchor);
|
||||
ZSTD_ldm_fillFastTables(cctx, anchor);
|
||||
|
||||
/* Call block compressor and get remaining literals */
|
||||
lastLiterals = blockCompressor(cctx, anchor, ip - anchor);
|
||||
cctx->nextToUpdate = (U32)(ip - base);
|
||||
|
||||
/* Update repToConfirm with the new offset */
|
||||
for (i = ZSTD_REP_NUM - 1; i > 0; i--)
|
||||
repToConfirm[i] = repToConfirm[i-1];
|
||||
repToConfirm[0] = offset;
|
||||
|
||||
/* Store the sequence with the leftover literals */
|
||||
ZSTD_storeSeq(seqStorePtr, lastLiterals, ip - lastLiterals,
|
||||
offset + ZSTD_REP_MOVE, mLength - MINMATCH);
|
||||
}
|
||||
|
||||
/* Insert the current entry into the hash table */
|
||||
ZSTD_ldm_makeEntryAndInsertByTag(ldmState, rollingHash, hBits,
|
||||
(U32)(lastHashed - base),
|
||||
ldmParams);
|
||||
|
||||
assert(ip + backwardMatchLength == lastHashed);
|
||||
|
||||
/* Fill the hash table from lastHashed+1 to ip+mLength*/
|
||||
/* Heuristic: don't need to fill the entire table at end of block */
|
||||
if (ip + mLength < ilimit) {
|
||||
rollingHash = ZSTD_ldm_fillLdmHashTable(
|
||||
ldmState, rollingHash, lastHashed,
|
||||
ip + mLength, base, hBits, ldmParams);
|
||||
lastHashed = ip + mLength - 1;
|
||||
}
|
||||
ip += mLength;
|
||||
anchor = ip;
|
||||
/* Check immediate repcode */
|
||||
while ( (ip < ilimit)
|
||||
&& ( (repToConfirm[1] > 0) && (repToConfirm[1] <= (U32)(ip-lowest))
|
||||
&& (MEM_read32(ip) == MEM_read32(ip - repToConfirm[1])) )) {
|
||||
|
||||
size_t const rLength = ZSTD_count(ip+4, ip+4-repToConfirm[1],
|
||||
iend) + 4;
|
||||
/* Swap repToConfirm[1] <=> repToConfirm[0] */
|
||||
{
|
||||
U32 const tmpOff = repToConfirm[1];
|
||||
repToConfirm[1] = repToConfirm[0];
|
||||
repToConfirm[0] = tmpOff;
|
||||
}
|
||||
|
||||
ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, rLength-MINMATCH);
|
||||
|
||||
/* Fill the hash table from lastHashed+1 to ip+rLength*/
|
||||
if (ip + rLength < ilimit) {
|
||||
rollingHash = ZSTD_ldm_fillLdmHashTable(
|
||||
ldmState, rollingHash, lastHashed,
|
||||
ip + rLength, base, hBits, ldmParams);
|
||||
lastHashed = ip + rLength - 1;
|
||||
}
|
||||
ip += rLength;
|
||||
anchor = ip;
|
||||
}
|
||||
}
|
||||
|
||||
/* Overwrite rep */
|
||||
for (i = 0; i < ZSTD_REP_NUM; i++)
|
||||
seqStorePtr->rep[i] = repToConfirm[i];
|
||||
|
||||
ZSTD_ldm_limitTableUpdate(cctx, anchor);
|
||||
ZSTD_ldm_fillFastTables(cctx, anchor);
|
||||
|
||||
lastLiterals = blockCompressor(cctx, anchor, iend - anchor);
|
||||
cctx->nextToUpdate = (U32)(iend - base);
|
||||
|
||||
/* Restore seqStorePtr->rep */
|
||||
for (i = 0; i < ZSTD_REP_NUM; i++)
|
||||
seqStorePtr->rep[i] = savedRep[i];
|
||||
|
||||
/* Return the last literals size */
|
||||
return lastLiterals;
|
||||
}
|
||||
|
||||
size_t ZSTD_compressBlock_ldm(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_ldm_generic(ctx, src, srcSize);
|
||||
}
|
||||
|
||||
static size_t ZSTD_compressBlock_ldm_extDict_generic(
|
||||
ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize)
|
||||
{
|
||||
ldmState_t* const ldmState = &(ctx->ldmState);
|
||||
const ldmParams_t ldmParams = ctx->appliedParams.ldmParams;
|
||||
const U64 hashPower = ldmState->hashPower;
|
||||
const U32 hBits = ldmParams.hashLog - ldmParams.bucketSizeLog;
|
||||
const U32 ldmBucketSize = ((U32)1 << ldmParams.bucketSizeLog);
|
||||
const U32 ldmTagMask = ((U32)1 << ldmParams.hashEveryLog) - 1;
|
||||
seqStore_t* const seqStorePtr = &(ctx->seqStore);
|
||||
const BYTE* const base = ctx->base;
|
||||
const BYTE* const dictBase = ctx->dictBase;
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const U32 lowestIndex = ctx->lowLimit;
|
||||
const BYTE* const dictStart = dictBase + lowestIndex;
|
||||
const U32 dictLimit = ctx->dictLimit;
|
||||
const BYTE* const lowPrefixPtr = base + dictLimit;
|
||||
const BYTE* const dictEnd = dictBase + dictLimit;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - MAX(ldmParams.minMatchLength, HASH_READ_SIZE);
|
||||
|
||||
const ZSTD_blockCompressor blockCompressor =
|
||||
ZSTD_selectBlockCompressor(ctx->appliedParams.cParams.strategy, 1);
|
||||
U32* const repToConfirm = seqStorePtr->repToConfirm;
|
||||
U32 savedRep[ZSTD_REP_NUM];
|
||||
U64 rollingHash = 0;
|
||||
const BYTE* lastHashed = NULL;
|
||||
size_t i, lastLiterals;
|
||||
|
||||
/* Save seqStorePtr->rep and copy repToConfirm */
|
||||
for (i = 0; i < ZSTD_REP_NUM; i++) {
|
||||
savedRep[i] = repToConfirm[i] = seqStorePtr->rep[i];
|
||||
}
|
||||
|
||||
/* Search Loop */
|
||||
while (ip < ilimit) { /* < instead of <=, because (ip+1) */
|
||||
size_t mLength;
|
||||
const U32 current = (U32)(ip-base);
|
||||
size_t forwardMatchLength = 0, backwardMatchLength = 0;
|
||||
ldmEntry_t* bestEntry = NULL;
|
||||
if (ip != istart) {
|
||||
rollingHash = ZSTD_ldm_updateHash(rollingHash, lastHashed[0],
|
||||
lastHashed[ldmParams.minMatchLength],
|
||||
hashPower);
|
||||
} else {
|
||||
rollingHash = ZSTD_ldm_getRollingHash(ip, ldmParams.minMatchLength);
|
||||
}
|
||||
lastHashed = ip;
|
||||
|
||||
if (ZSTD_ldm_getTag(rollingHash, hBits, ldmParams.hashEveryLog) !=
|
||||
ldmTagMask) {
|
||||
/* Don't insert and don't look for a match */
|
||||
ip++;
|
||||
continue;
|
||||
}
|
||||
|
||||
/* Get the best entry and compute the match lengths */
|
||||
{
|
||||
ldmEntry_t* const bucket =
|
||||
ZSTD_ldm_getBucket(ldmState,
|
||||
ZSTD_ldm_getSmallHash(rollingHash, hBits),
|
||||
ldmParams);
|
||||
ldmEntry_t* cur;
|
||||
size_t bestMatchLength = 0;
|
||||
U32 const checksum = ZSTD_ldm_getChecksum(rollingHash, hBits);
|
||||
|
||||
for (cur = bucket; cur < bucket + ldmBucketSize; ++cur) {
|
||||
const BYTE* const curMatchBase =
|
||||
cur->offset < dictLimit ? dictBase : base;
|
||||
const BYTE* const pMatch = curMatchBase + cur->offset;
|
||||
const BYTE* const matchEnd =
|
||||
cur->offset < dictLimit ? dictEnd : iend;
|
||||
const BYTE* const lowMatchPtr =
|
||||
cur->offset < dictLimit ? dictStart : lowPrefixPtr;
|
||||
size_t curForwardMatchLength, curBackwardMatchLength,
|
||||
curTotalMatchLength;
|
||||
|
||||
if (cur->checksum != checksum || cur->offset <= lowestIndex) {
|
||||
continue;
|
||||
}
|
||||
|
||||
curForwardMatchLength = ZSTD_count_2segments(
|
||||
ip, pMatch, iend,
|
||||
matchEnd, lowPrefixPtr);
|
||||
if (curForwardMatchLength < ldmParams.minMatchLength) {
|
||||
continue;
|
||||
}
|
||||
curBackwardMatchLength = ZSTD_ldm_countBackwardsMatch(
|
||||
ip, anchor, pMatch, lowMatchPtr);
|
||||
curTotalMatchLength = curForwardMatchLength +
|
||||
curBackwardMatchLength;
|
||||
|
||||
if (curTotalMatchLength > bestMatchLength) {
|
||||
bestMatchLength = curTotalMatchLength;
|
||||
forwardMatchLength = curForwardMatchLength;
|
||||
backwardMatchLength = curBackwardMatchLength;
|
||||
bestEntry = cur;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* No match found -- continue searching */
|
||||
if (bestEntry == NULL) {
|
||||
ZSTD_ldm_makeEntryAndInsertByTag(ldmState, rollingHash, hBits,
|
||||
(U32)(lastHashed - base),
|
||||
ldmParams);
|
||||
ip++;
|
||||
continue;
|
||||
}
|
||||
|
||||
/* Match found */
|
||||
mLength = forwardMatchLength + backwardMatchLength;
|
||||
ip -= backwardMatchLength;
|
||||
|
||||
/* Call the block compressor on the remaining literals */
|
||||
{
|
||||
/* ip = current - backwardMatchLength
|
||||
* The match is at (bestEntry->offset - backwardMatchLength) */
|
||||
U32 const matchIndex = bestEntry->offset;
|
||||
U32 const offset = current - matchIndex;
|
||||
|
||||
/* Overwrite rep codes */
|
||||
for (i = 0; i < ZSTD_REP_NUM; i++)
|
||||
seqStorePtr->rep[i] = repToConfirm[i];
|
||||
|
||||
/* Fill the hash table for the block compressor */
|
||||
ZSTD_ldm_limitTableUpdate(ctx, anchor);
|
||||
ZSTD_ldm_fillFastTables(ctx, anchor);
|
||||
|
||||
/* Call block compressor and get remaining literals */
|
||||
lastLiterals = blockCompressor(ctx, anchor, ip - anchor);
|
||||
ctx->nextToUpdate = (U32)(ip - base);
|
||||
|
||||
/* Update repToConfirm with the new offset */
|
||||
for (i = ZSTD_REP_NUM - 1; i > 0; i--)
|
||||
repToConfirm[i] = repToConfirm[i-1];
|
||||
repToConfirm[0] = offset;
|
||||
|
||||
/* Store the sequence with the leftover literals */
|
||||
ZSTD_storeSeq(seqStorePtr, lastLiterals, ip - lastLiterals,
|
||||
offset + ZSTD_REP_MOVE, mLength - MINMATCH);
|
||||
}
|
||||
|
||||
/* Insert the current entry into the hash table */
|
||||
ZSTD_ldm_makeEntryAndInsertByTag(ldmState, rollingHash, hBits,
|
||||
(U32)(lastHashed - base),
|
||||
ldmParams);
|
||||
|
||||
/* Fill the hash table from lastHashed+1 to ip+mLength */
|
||||
assert(ip + backwardMatchLength == lastHashed);
|
||||
if (ip + mLength < ilimit) {
|
||||
rollingHash = ZSTD_ldm_fillLdmHashTable(
|
||||
ldmState, rollingHash, lastHashed,
|
||||
ip + mLength, base, hBits,
|
||||
ldmParams);
|
||||
lastHashed = ip + mLength - 1;
|
||||
}
|
||||
ip += mLength;
|
||||
anchor = ip;
|
||||
|
||||
/* check immediate repcode */
|
||||
while (ip < ilimit) {
|
||||
U32 const current2 = (U32)(ip-base);
|
||||
U32 const repIndex2 = current2 - repToConfirm[1];
|
||||
const BYTE* repMatch2 = repIndex2 < dictLimit ?
|
||||
dictBase + repIndex2 : base + repIndex2;
|
||||
if ( (((U32)((dictLimit-1) - repIndex2) >= 3) &
|
||||
(repIndex2 > lowestIndex)) /* intentional overflow */
|
||||
&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
|
||||
const BYTE* const repEnd2 = repIndex2 < dictLimit ?
|
||||
dictEnd : iend;
|
||||
size_t const repLength2 =
|
||||
ZSTD_count_2segments(ip+4, repMatch2+4, iend,
|
||||
repEnd2, lowPrefixPtr) + 4;
|
||||
|
||||
U32 tmpOffset = repToConfirm[1];
|
||||
repToConfirm[1] = repToConfirm[0];
|
||||
repToConfirm[0] = tmpOffset;
|
||||
|
||||
ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, repLength2-MINMATCH);
|
||||
|
||||
/* Fill the hash table from lastHashed+1 to ip+repLength2*/
|
||||
if (ip + repLength2 < ilimit) {
|
||||
rollingHash = ZSTD_ldm_fillLdmHashTable(
|
||||
ldmState, rollingHash, lastHashed,
|
||||
ip + repLength2, base, hBits,
|
||||
ldmParams);
|
||||
lastHashed = ip + repLength2 - 1;
|
||||
}
|
||||
ip += repLength2;
|
||||
anchor = ip;
|
||||
continue;
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
/* Overwrite rep */
|
||||
for (i = 0; i < ZSTD_REP_NUM; i++)
|
||||
seqStorePtr->rep[i] = repToConfirm[i];
|
||||
|
||||
ZSTD_ldm_limitTableUpdate(ctx, anchor);
|
||||
ZSTD_ldm_fillFastTables(ctx, anchor);
|
||||
|
||||
/* Call the block compressor one last time on the last literals */
|
||||
lastLiterals = blockCompressor(ctx, anchor, iend - anchor);
|
||||
ctx->nextToUpdate = (U32)(iend - base);
|
||||
|
||||
/* Restore seqStorePtr->rep */
|
||||
for (i = 0; i < ZSTD_REP_NUM; i++)
|
||||
seqStorePtr->rep[i] = savedRep[i];
|
||||
|
||||
/* Return the last literals size */
|
||||
return lastLiterals;
|
||||
}
|
||||
|
||||
size_t ZSTD_compressBlock_ldm_extDict(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_ldm_extDict_generic(ctx, src, srcSize);
|
||||
}
|
67
lib/compress/zstd_ldm.h
Normal file
67
lib/compress/zstd_ldm.h
Normal file
@ -0,0 +1,67 @@
|
||||
/*
|
||||
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
*/
|
||||
|
||||
#ifndef ZSTD_LDM_H
|
||||
#define ZSTD_LDM_H
|
||||
|
||||
#include "zstd_compress.h"
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/*-*************************************
|
||||
* Long distance matching
|
||||
***************************************/
|
||||
|
||||
#define ZSTD_LDM_DEFAULT_WINDOW_LOG ZSTD_WINDOWLOG_DEFAULTMAX
|
||||
#define ZSTD_LDM_HASHEVERYLOG_NOTSET 9999
|
||||
|
||||
/** ZSTD_compressBlock_ldm_generic() :
|
||||
*
|
||||
* This is a block compressor intended for long distance matching.
|
||||
*
|
||||
* The function searches for matches of length at least
|
||||
* ldmParams.minMatchLength using a hash table in cctx->ldmState.
|
||||
* Matches can be at a distance of up to cParams.windowLog.
|
||||
*
|
||||
* Upon finding a match, the unmatched literals are compressed using a
|
||||
* ZSTD_blockCompressor (depending on the strategy in the compression
|
||||
* parameters), which stores the matched sequences. The "long distance"
|
||||
* match is then stored with the remaining literals from the
|
||||
* ZSTD_blockCompressor. */
|
||||
size_t ZSTD_compressBlock_ldm(ZSTD_CCtx* cctx, const void* src, size_t srcSize);
|
||||
size_t ZSTD_compressBlock_ldm_extDict(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize);
|
||||
|
||||
/** ZSTD_ldm_initializeParameters() :
|
||||
* Initialize the long distance matching parameters to their default values. */
|
||||
size_t ZSTD_ldm_initializeParameters(ldmParams_t* params, U32 enableLdm);
|
||||
|
||||
/** ZSTD_ldm_getTableSize() :
|
||||
* Estimate the space needed for long distance matching tables. */
|
||||
size_t ZSTD_ldm_getTableSize(U32 hashLog, U32 bucketSizeLog);
|
||||
|
||||
/** ZSTD_ldm_getTableSize() :
|
||||
* Return prime8bytes^(minMatchLength-1) */
|
||||
U64 ZSTD_ldm_getHashPower(U32 minMatchLength);
|
||||
|
||||
/** ZSTD_ldm_adjustParameters() :
|
||||
* If the params->hashEveryLog is not set, set it to its default value based on
|
||||
* windowLog and params->hashLog.
|
||||
*
|
||||
* Ensures that params->bucketSizeLog is <= params->hashLog (setting it to
|
||||
* params->hashLog if it is not). */
|
||||
void ZSTD_ldm_adjustParameters(ldmParams_t* params, U32 windowLog);
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* ZSTD_FAST_H */
|
957
lib/compress/zstd_opt.c
Normal file
957
lib/compress/zstd_opt.c
Normal file
@ -0,0 +1,957 @@
|
||||
/*
|
||||
* Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
|
||||
* All rights reserved.
|
||||
*
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#include "zstd_opt.h"
|
||||
#include "zstd_lazy.h"
|
||||
|
||||
|
||||
#define ZSTD_LITFREQ_ADD 2
|
||||
#define ZSTD_FREQ_DIV 4
|
||||
#define ZSTD_MAX_PRICE (1<<30)
|
||||
|
||||
/*-*************************************
|
||||
* Price functions for optimal parser
|
||||
***************************************/
|
||||
static void ZSTD_setLog2Prices(optState_t* optPtr)
|
||||
{
|
||||
optPtr->log2matchLengthSum = ZSTD_highbit32(optPtr->matchLengthSum+1);
|
||||
optPtr->log2litLengthSum = ZSTD_highbit32(optPtr->litLengthSum+1);
|
||||
optPtr->log2litSum = ZSTD_highbit32(optPtr->litSum+1);
|
||||
optPtr->log2offCodeSum = ZSTD_highbit32(optPtr->offCodeSum+1);
|
||||
optPtr->factor = 1 + ((optPtr->litSum>>5) / optPtr->litLengthSum) + ((optPtr->litSum<<1) / (optPtr->litSum + optPtr->matchSum));
|
||||
}
|
||||
|
||||
|
||||
static void ZSTD_rescaleFreqs(optState_t* optPtr, const BYTE* src, size_t srcSize)
|
||||
{
|
||||
unsigned u;
|
||||
|
||||
optPtr->cachedLiterals = NULL;
|
||||
optPtr->cachedPrice = optPtr->cachedLitLength = 0;
|
||||
optPtr->staticPrices = 0;
|
||||
|
||||
if (optPtr->litLengthSum == 0) {
|
||||
if (srcSize <= 1024) optPtr->staticPrices = 1;
|
||||
|
||||
assert(optPtr->litFreq!=NULL);
|
||||
for (u=0; u<=MaxLit; u++)
|
||||
optPtr->litFreq[u] = 0;
|
||||
for (u=0; u<srcSize; u++)
|
||||
optPtr->litFreq[src[u]]++;
|
||||
|
||||
optPtr->litSum = 0;
|
||||
optPtr->litLengthSum = MaxLL+1;
|
||||
optPtr->matchLengthSum = MaxML+1;
|
||||
optPtr->offCodeSum = (MaxOff+1);
|
||||
optPtr->matchSum = (ZSTD_LITFREQ_ADD<<Litbits);
|
||||
|
||||
for (u=0; u<=MaxLit; u++) {
|
||||
optPtr->litFreq[u] = 1 + (optPtr->litFreq[u]>>ZSTD_FREQ_DIV);
|
||||
optPtr->litSum += optPtr->litFreq[u];
|
||||
}
|
||||
for (u=0; u<=MaxLL; u++)
|
||||
optPtr->litLengthFreq[u] = 1;
|
||||
for (u=0; u<=MaxML; u++)
|
||||
optPtr->matchLengthFreq[u] = 1;
|
||||
for (u=0; u<=MaxOff; u++)
|
||||
optPtr->offCodeFreq[u] = 1;
|
||||
} else {
|
||||
optPtr->matchLengthSum = 0;
|
||||
optPtr->litLengthSum = 0;
|
||||
optPtr->offCodeSum = 0;
|
||||
optPtr->matchSum = 0;
|
||||
optPtr->litSum = 0;
|
||||
|
||||
for (u=0; u<=MaxLit; u++) {
|
||||
optPtr->litFreq[u] = 1 + (optPtr->litFreq[u]>>(ZSTD_FREQ_DIV+1));
|
||||
optPtr->litSum += optPtr->litFreq[u];
|
||||
}
|
||||
for (u=0; u<=MaxLL; u++) {
|
||||
optPtr->litLengthFreq[u] = 1 + (optPtr->litLengthFreq[u]>>(ZSTD_FREQ_DIV+1));
|
||||
optPtr->litLengthSum += optPtr->litLengthFreq[u];
|
||||
}
|
||||
for (u=0; u<=MaxML; u++) {
|
||||
optPtr->matchLengthFreq[u] = 1 + (optPtr->matchLengthFreq[u]>>ZSTD_FREQ_DIV);
|
||||
optPtr->matchLengthSum += optPtr->matchLengthFreq[u];
|
||||
optPtr->matchSum += optPtr->matchLengthFreq[u] * (u + 3);
|
||||
}
|
||||
optPtr->matchSum *= ZSTD_LITFREQ_ADD;
|
||||
for (u=0; u<=MaxOff; u++) {
|
||||
optPtr->offCodeFreq[u] = 1 + (optPtr->offCodeFreq[u]>>ZSTD_FREQ_DIV);
|
||||
optPtr->offCodeSum += optPtr->offCodeFreq[u];
|
||||
}
|
||||
}
|
||||
|
||||
ZSTD_setLog2Prices(optPtr);
|
||||
}
|
||||
|
||||
|
||||
static U32 ZSTD_getLiteralPrice(optState_t* optPtr, U32 litLength, const BYTE* literals)
|
||||
{
|
||||
U32 price, u;
|
||||
|
||||
if (optPtr->staticPrices)
|
||||
return ZSTD_highbit32((U32)litLength+1) + (litLength*6);
|
||||
|
||||
if (litLength == 0)
|
||||
return optPtr->log2litLengthSum - ZSTD_highbit32(optPtr->litLengthFreq[0]+1);
|
||||
|
||||
/* literals */
|
||||
if (optPtr->cachedLiterals == literals) {
|
||||
U32 const additional = litLength - optPtr->cachedLitLength;
|
||||
const BYTE* literals2 = optPtr->cachedLiterals + optPtr->cachedLitLength;
|
||||
price = optPtr->cachedPrice + additional * optPtr->log2litSum;
|
||||
for (u=0; u < additional; u++)
|
||||
price -= ZSTD_highbit32(optPtr->litFreq[literals2[u]]+1);
|
||||
optPtr->cachedPrice = price;
|
||||
optPtr->cachedLitLength = litLength;
|
||||
} else {
|
||||
price = litLength * optPtr->log2litSum;
|
||||
for (u=0; u < litLength; u++)
|
||||
price -= ZSTD_highbit32(optPtr->litFreq[literals[u]]+1);
|
||||
|
||||
if (litLength >= 12) {
|
||||
optPtr->cachedLiterals = literals;
|
||||
optPtr->cachedPrice = price;
|
||||
optPtr->cachedLitLength = litLength;
|
||||
}
|
||||
}
|
||||
|
||||
/* literal Length */
|
||||
{ const BYTE LL_deltaCode = 19;
|
||||
const BYTE llCode = (litLength>63) ? (BYTE)ZSTD_highbit32(litLength) + LL_deltaCode : LL_Code[litLength];
|
||||
price += LL_bits[llCode] + optPtr->log2litLengthSum - ZSTD_highbit32(optPtr->litLengthFreq[llCode]+1);
|
||||
}
|
||||
|
||||
return price;
|
||||
}
|
||||
|
||||
|
||||
FORCE_INLINE_TEMPLATE U32 ZSTD_getPrice(optState_t* optPtr, U32 litLength, const BYTE* literals, U32 offset, U32 matchLength, const int ultra)
|
||||
{
|
||||
/* offset */
|
||||
U32 price;
|
||||
BYTE const offCode = (BYTE)ZSTD_highbit32(offset+1);
|
||||
|
||||
if (optPtr->staticPrices)
|
||||
return ZSTD_getLiteralPrice(optPtr, litLength, literals) + ZSTD_highbit32((U32)matchLength+1) + 16 + offCode;
|
||||
|
||||
price = offCode + optPtr->log2offCodeSum - ZSTD_highbit32(optPtr->offCodeFreq[offCode]+1);
|
||||
if (!ultra && offCode >= 20) price += (offCode-19)*2;
|
||||
|
||||
/* match Length */
|
||||
{ const BYTE ML_deltaCode = 36;
|
||||
const BYTE mlCode = (matchLength>127) ? (BYTE)ZSTD_highbit32(matchLength) + ML_deltaCode : ML_Code[matchLength];
|
||||
price += ML_bits[mlCode] + optPtr->log2matchLengthSum - ZSTD_highbit32(optPtr->matchLengthFreq[mlCode]+1);
|
||||
}
|
||||
|
||||
return price + ZSTD_getLiteralPrice(optPtr, litLength, literals) + optPtr->factor;
|
||||
}
|
||||
|
||||
|
||||
static void ZSTD_updatePrice(optState_t* optPtr, U32 litLength, const BYTE* literals, U32 offset, U32 matchLength)
|
||||
{
|
||||
U32 u;
|
||||
|
||||
/* literals */
|
||||
optPtr->litSum += litLength*ZSTD_LITFREQ_ADD;
|
||||
for (u=0; u < litLength; u++)
|
||||
optPtr->litFreq[literals[u]] += ZSTD_LITFREQ_ADD;
|
||||
|
||||
/* literal Length */
|
||||
{ const BYTE LL_deltaCode = 19;
|
||||
const BYTE llCode = (litLength>63) ? (BYTE)ZSTD_highbit32(litLength) + LL_deltaCode : LL_Code[litLength];
|
||||
optPtr->litLengthFreq[llCode]++;
|
||||
optPtr->litLengthSum++;
|
||||
}
|
||||
|
||||
/* match offset */
|
||||
{ BYTE const offCode = (BYTE)ZSTD_highbit32(offset+1);
|
||||
optPtr->offCodeSum++;
|
||||
optPtr->offCodeFreq[offCode]++;
|
||||
}
|
||||
|
||||
/* match Length */
|
||||
{ const BYTE ML_deltaCode = 36;
|
||||
const BYTE mlCode = (matchLength>127) ? (BYTE)ZSTD_highbit32(matchLength) + ML_deltaCode : ML_Code[matchLength];
|
||||
optPtr->matchLengthFreq[mlCode]++;
|
||||
optPtr->matchLengthSum++;
|
||||
}
|
||||
|
||||
ZSTD_setLog2Prices(optPtr);
|
||||
}
|
||||
|
||||
|
||||
#define SET_PRICE(pos, mlen_, offset_, litlen_, price_) \
|
||||
{ \
|
||||
while (last_pos < pos) { opt[last_pos+1].price = ZSTD_MAX_PRICE; last_pos++; } \
|
||||
opt[pos].mlen = mlen_; \
|
||||
opt[pos].off = offset_; \
|
||||
opt[pos].litlen = litlen_; \
|
||||
opt[pos].price = price_; \
|
||||
}
|
||||
|
||||
|
||||
/* function safe only for comparisons */
|
||||
static U32 ZSTD_readMINMATCH(const void* memPtr, U32 length)
|
||||
{
|
||||
switch (length)
|
||||
{
|
||||
default :
|
||||
case 4 : return MEM_read32(memPtr);
|
||||
case 3 : if (MEM_isLittleEndian())
|
||||
return MEM_read32(memPtr)<<8;
|
||||
else
|
||||
return MEM_read32(memPtr)>>8;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/* Update hashTable3 up to ip (excluded)
|
||||
Assumption : always within prefix (i.e. not within extDict) */
|
||||
static
|
||||
U32 ZSTD_insertAndFindFirstIndexHash3 (ZSTD_CCtx* zc, const BYTE* ip)
|
||||
{
|
||||
U32* const hashTable3 = zc->hashTable3;
|
||||
U32 const hashLog3 = zc->hashLog3;
|
||||
const BYTE* const base = zc->base;
|
||||
U32 idx = zc->nextToUpdate3;
|
||||
const U32 target = zc->nextToUpdate3 = (U32)(ip - base);
|
||||
const size_t hash3 = ZSTD_hash3Ptr(ip, hashLog3);
|
||||
|
||||
while(idx < target) {
|
||||
hashTable3[ZSTD_hash3Ptr(base+idx, hashLog3)] = idx;
|
||||
idx++;
|
||||
}
|
||||
|
||||
return hashTable3[hash3];
|
||||
}
|
||||
|
||||
|
||||
/*-*************************************
|
||||
* Binary Tree search
|
||||
***************************************/
|
||||
static U32 ZSTD_insertBtAndGetAllMatches (
|
||||
ZSTD_CCtx* zc,
|
||||
const BYTE* const ip, const BYTE* const iLimit,
|
||||
U32 nbCompares, const U32 mls,
|
||||
U32 extDict, ZSTD_match_t* matches, const U32 minMatchLen)
|
||||
{
|
||||
const BYTE* const base = zc->base;
|
||||
const U32 current = (U32)(ip-base);
|
||||
const U32 hashLog = zc->appliedParams.cParams.hashLog;
|
||||
const size_t h = ZSTD_hashPtr(ip, hashLog, mls);
|
||||
U32* const hashTable = zc->hashTable;
|
||||
U32 matchIndex = hashTable[h];
|
||||
U32* const bt = zc->chainTable;
|
||||
const U32 btLog = zc->appliedParams.cParams.chainLog - 1;
|
||||
const U32 btMask= (1U << btLog) - 1;
|
||||
size_t commonLengthSmaller=0, commonLengthLarger=0;
|
||||
const BYTE* const dictBase = zc->dictBase;
|
||||
const U32 dictLimit = zc->dictLimit;
|
||||
const BYTE* const dictEnd = dictBase + dictLimit;
|
||||
const BYTE* const prefixStart = base + dictLimit;
|
||||
const U32 btLow = btMask >= current ? 0 : current - btMask;
|
||||
const U32 windowLow = zc->lowLimit;
|
||||
U32* smallerPtr = bt + 2*(current&btMask);
|
||||
U32* largerPtr = bt + 2*(current&btMask) + 1;
|
||||
U32 matchEndIdx = current+8;
|
||||
U32 dummy32; /* to be nullified at the end */
|
||||
U32 mnum = 0;
|
||||
|
||||
const U32 minMatch = (mls == 3) ? 3 : 4;
|
||||
size_t bestLength = minMatchLen-1;
|
||||
|
||||
if (minMatch == 3) { /* HC3 match finder */
|
||||
U32 const matchIndex3 = ZSTD_insertAndFindFirstIndexHash3 (zc, ip);
|
||||
if (matchIndex3>windowLow && (current - matchIndex3 < (1<<18))) {
|
||||
const BYTE* match;
|
||||
size_t currentMl=0;
|
||||
if ((!extDict) || matchIndex3 >= dictLimit) {
|
||||
match = base + matchIndex3;
|
||||
if (match[bestLength] == ip[bestLength]) currentMl = ZSTD_count(ip, match, iLimit);
|
||||
} else {
|
||||
match = dictBase + matchIndex3;
|
||||
if (ZSTD_readMINMATCH(match, MINMATCH) == ZSTD_readMINMATCH(ip, MINMATCH)) /* assumption : matchIndex3 <= dictLimit-4 (by table construction) */
|
||||
currentMl = ZSTD_count_2segments(ip+MINMATCH, match+MINMATCH, iLimit, dictEnd, prefixStart) + MINMATCH;
|
||||
}
|
||||
|
||||
/* save best solution */
|
||||
if (currentMl > bestLength) {
|
||||
bestLength = currentMl;
|
||||
matches[mnum].off = ZSTD_REP_MOVE_OPT + current - matchIndex3;
|
||||
matches[mnum].len = (U32)currentMl;
|
||||
mnum++;
|
||||
if (currentMl > ZSTD_OPT_NUM) goto update;
|
||||
if (ip+currentMl == iLimit) goto update; /* best possible, and avoid read overflow*/
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
hashTable[h] = current; /* Update Hash Table */
|
||||
|
||||
while (nbCompares-- && (matchIndex > windowLow)) {
|
||||
U32* nextPtr = bt + 2*(matchIndex & btMask);
|
||||
size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
|
||||
const BYTE* match;
|
||||
|
||||
if ((!extDict) || (matchIndex+matchLength >= dictLimit)) {
|
||||
match = base + matchIndex;
|
||||
if (match[matchLength] == ip[matchLength]) {
|
||||
matchLength += ZSTD_count(ip+matchLength+1, match+matchLength+1, iLimit) +1;
|
||||
}
|
||||
} else {
|
||||
match = dictBase + matchIndex;
|
||||
matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iLimit, dictEnd, prefixStart);
|
||||
if (matchIndex+matchLength >= dictLimit)
|
||||
match = base + matchIndex; /* to prepare for next usage of match[matchLength] */
|
||||
}
|
||||
|
||||
if (matchLength > bestLength) {
|
||||
if (matchLength > matchEndIdx - matchIndex) matchEndIdx = matchIndex + (U32)matchLength;
|
||||
bestLength = matchLength;
|
||||
matches[mnum].off = ZSTD_REP_MOVE_OPT + current - matchIndex;
|
||||
matches[mnum].len = (U32)matchLength;
|
||||
mnum++;
|
||||
if (matchLength > ZSTD_OPT_NUM) break;
|
||||
if (ip+matchLength == iLimit) /* equal : no way to know if inf or sup */
|
||||
break; /* drop, to guarantee consistency (miss a little bit of compression) */
|
||||
}
|
||||
|
||||
if (match[matchLength] < ip[matchLength]) {
|
||||
/* match is smaller than current */
|
||||
*smallerPtr = matchIndex; /* update smaller idx */
|
||||
commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
|
||||
if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */
|
||||
smallerPtr = nextPtr+1; /* new "smaller" => larger of match */
|
||||
matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
|
||||
} else {
|
||||
/* match is larger than current */
|
||||
*largerPtr = matchIndex;
|
||||
commonLengthLarger = matchLength;
|
||||
if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */
|
||||
largerPtr = nextPtr;
|
||||
matchIndex = nextPtr[0];
|
||||
} }
|
||||
|
||||
*smallerPtr = *largerPtr = 0;
|
||||
|
||||
update:
|
||||
zc->nextToUpdate = (matchEndIdx > current + 8) ? matchEndIdx - 8 : current+1;
|
||||
return mnum;
|
||||
}
|
||||
|
||||
|
||||
/** Tree updater, providing best match */
|
||||
static U32 ZSTD_BtGetAllMatches (
|
||||
ZSTD_CCtx* zc,
|
||||
const BYTE* const ip, const BYTE* const iLimit,
|
||||
const U32 maxNbAttempts, const U32 mls, ZSTD_match_t* matches, const U32 minMatchLen)
|
||||
{
|
||||
if (ip < zc->base + zc->nextToUpdate) return 0; /* skipped area */
|
||||
ZSTD_updateTree(zc, ip, iLimit, maxNbAttempts, mls);
|
||||
return ZSTD_insertBtAndGetAllMatches(zc, ip, iLimit, maxNbAttempts, mls, 0, matches, minMatchLen);
|
||||
}
|
||||
|
||||
|
||||
static U32 ZSTD_BtGetAllMatches_selectMLS (
|
||||
ZSTD_CCtx* zc, /* Index table will be updated */
|
||||
const BYTE* ip, const BYTE* const iHighLimit,
|
||||
const U32 maxNbAttempts, const U32 matchLengthSearch, ZSTD_match_t* matches, const U32 minMatchLen)
|
||||
{
|
||||
switch(matchLengthSearch)
|
||||
{
|
||||
case 3 : return ZSTD_BtGetAllMatches(zc, ip, iHighLimit, maxNbAttempts, 3, matches, minMatchLen);
|
||||
default :
|
||||
case 4 : return ZSTD_BtGetAllMatches(zc, ip, iHighLimit, maxNbAttempts, 4, matches, minMatchLen);
|
||||
case 5 : return ZSTD_BtGetAllMatches(zc, ip, iHighLimit, maxNbAttempts, 5, matches, minMatchLen);
|
||||
case 7 :
|
||||
case 6 : return ZSTD_BtGetAllMatches(zc, ip, iHighLimit, maxNbAttempts, 6, matches, minMatchLen);
|
||||
}
|
||||
}
|
||||
|
||||
/** Tree updater, providing best match */
|
||||
static U32 ZSTD_BtGetAllMatches_extDict (
|
||||
ZSTD_CCtx* zc,
|
||||
const BYTE* const ip, const BYTE* const iLimit,
|
||||
const U32 maxNbAttempts, const U32 mls, ZSTD_match_t* matches, const U32 minMatchLen)
|
||||
{
|
||||
if (ip < zc->base + zc->nextToUpdate) return 0; /* skipped area */
|
||||
ZSTD_updateTree_extDict(zc, ip, iLimit, maxNbAttempts, mls);
|
||||
return ZSTD_insertBtAndGetAllMatches(zc, ip, iLimit, maxNbAttempts, mls, 1, matches, minMatchLen);
|
||||
}
|
||||
|
||||
|
||||
static U32 ZSTD_BtGetAllMatches_selectMLS_extDict (
|
||||
ZSTD_CCtx* zc, /* Index table will be updated */
|
||||
const BYTE* ip, const BYTE* const iHighLimit,
|
||||
const U32 maxNbAttempts, const U32 matchLengthSearch, ZSTD_match_t* matches, const U32 minMatchLen)
|
||||
{
|
||||
switch(matchLengthSearch)
|
||||
{
|
||||
case 3 : return ZSTD_BtGetAllMatches_extDict(zc, ip, iHighLimit, maxNbAttempts, 3, matches, minMatchLen);
|
||||
default :
|
||||
case 4 : return ZSTD_BtGetAllMatches_extDict(zc, ip, iHighLimit, maxNbAttempts, 4, matches, minMatchLen);
|
||||
case 5 : return ZSTD_BtGetAllMatches_extDict(zc, ip, iHighLimit, maxNbAttempts, 5, matches, minMatchLen);
|
||||
case 7 :
|
||||
case 6 : return ZSTD_BtGetAllMatches_extDict(zc, ip, iHighLimit, maxNbAttempts, 6, matches, minMatchLen);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*-*******************************
|
||||
* Optimal parser
|
||||
*********************************/
|
||||
FORCE_INLINE_TEMPLATE
|
||||
size_t ZSTD_compressBlock_opt_generic(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize, const int ultra)
|
||||
{
|
||||
seqStore_t* seqStorePtr = &(ctx->seqStore);
|
||||
optState_t* optStatePtr = &(ctx->optState);
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - 8;
|
||||
const BYTE* const base = ctx->base;
|
||||
const BYTE* const prefixStart = base + ctx->dictLimit;
|
||||
|
||||
const U32 maxSearches = 1U << ctx->appliedParams.cParams.searchLog;
|
||||
const U32 sufficient_len = ctx->appliedParams.cParams.targetLength;
|
||||
const U32 mls = ctx->appliedParams.cParams.searchLength;
|
||||
const U32 minMatch = (ctx->appliedParams.cParams.searchLength == 3) ? 3 : 4;
|
||||
|
||||
ZSTD_optimal_t* opt = optStatePtr->priceTable;
|
||||
ZSTD_match_t* matches = optStatePtr->matchTable;
|
||||
const BYTE* inr;
|
||||
U32 offset, rep[ZSTD_REP_NUM];
|
||||
|
||||
/* init */
|
||||
ctx->nextToUpdate3 = ctx->nextToUpdate;
|
||||
ZSTD_rescaleFreqs(optStatePtr, (const BYTE*)src, srcSize);
|
||||
ip += (ip==prefixStart);
|
||||
{ U32 i; for (i=0; i<ZSTD_REP_NUM; i++) rep[i]=seqStorePtr->rep[i]; }
|
||||
|
||||
/* Match Loop */
|
||||
while (ip < ilimit) {
|
||||
U32 cur, match_num, last_pos, litlen, price;
|
||||
U32 u, mlen, best_mlen, best_off, litLength;
|
||||
memset(opt, 0, sizeof(ZSTD_optimal_t));
|
||||
last_pos = 0;
|
||||
litlen = (U32)(ip - anchor);
|
||||
|
||||
/* check repCode */
|
||||
{ U32 i, last_i = ZSTD_REP_CHECK + (ip==anchor);
|
||||
for (i=(ip == anchor); i<last_i; i++) {
|
||||
const S32 repCur = (i==ZSTD_REP_MOVE_OPT) ? (rep[0] - 1) : rep[i];
|
||||
if ( (repCur > 0) && (repCur < (S32)(ip-prefixStart))
|
||||
&& (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(ip - repCur, minMatch))) {
|
||||
mlen = (U32)ZSTD_count(ip+minMatch, ip+minMatch-repCur, iend) + minMatch;
|
||||
if (mlen > sufficient_len || mlen >= ZSTD_OPT_NUM) {
|
||||
best_mlen = mlen; best_off = i; cur = 0; last_pos = 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
best_off = i - (ip == anchor);
|
||||
do {
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, best_off, mlen - MINMATCH, ultra);
|
||||
if (mlen > last_pos || price < opt[mlen].price)
|
||||
SET_PRICE(mlen, mlen, i, litlen, price); /* note : macro modifies last_pos */
|
||||
mlen--;
|
||||
} while (mlen >= minMatch);
|
||||
} } }
|
||||
|
||||
match_num = ZSTD_BtGetAllMatches_selectMLS(ctx, ip, iend, maxSearches, mls, matches, minMatch);
|
||||
|
||||
if (!last_pos && !match_num) { ip++; continue; }
|
||||
|
||||
if (match_num && (matches[match_num-1].len > sufficient_len || matches[match_num-1].len >= ZSTD_OPT_NUM)) {
|
||||
best_mlen = matches[match_num-1].len;
|
||||
best_off = matches[match_num-1].off;
|
||||
cur = 0;
|
||||
last_pos = 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
/* set prices using matches at position = 0 */
|
||||
best_mlen = (last_pos) ? last_pos : minMatch;
|
||||
for (u = 0; u < match_num; u++) {
|
||||
mlen = (u>0) ? matches[u-1].len+1 : best_mlen;
|
||||
best_mlen = matches[u].len;
|
||||
while (mlen <= best_mlen) {
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
if (mlen > last_pos || price < opt[mlen].price)
|
||||
SET_PRICE(mlen, mlen, matches[u].off, litlen, price); /* note : macro modifies last_pos */
|
||||
mlen++;
|
||||
} }
|
||||
|
||||
if (last_pos < minMatch) { ip++; continue; }
|
||||
|
||||
/* initialize opt[0] */
|
||||
{ U32 i ; for (i=0; i<ZSTD_REP_NUM; i++) opt[0].rep[i] = rep[i]; }
|
||||
opt[0].mlen = 1;
|
||||
opt[0].litlen = litlen;
|
||||
|
||||
/* check further positions */
|
||||
for (cur = 1; cur <= last_pos; cur++) {
|
||||
inr = ip + cur;
|
||||
|
||||
if (opt[cur-1].mlen == 1) {
|
||||
litlen = opt[cur-1].litlen + 1;
|
||||
if (cur > litlen) {
|
||||
price = opt[cur - litlen].price + ZSTD_getLiteralPrice(optStatePtr, litlen, inr-litlen);
|
||||
} else
|
||||
price = ZSTD_getLiteralPrice(optStatePtr, litlen, anchor);
|
||||
} else {
|
||||
litlen = 1;
|
||||
price = opt[cur - 1].price + ZSTD_getLiteralPrice(optStatePtr, litlen, inr-1);
|
||||
}
|
||||
|
||||
if (cur > last_pos || price <= opt[cur].price)
|
||||
SET_PRICE(cur, 1, 0, litlen, price);
|
||||
|
||||
if (cur == last_pos) break;
|
||||
|
||||
if (inr > ilimit) /* last match must start at a minimum distance of 8 from oend */
|
||||
continue;
|
||||
|
||||
mlen = opt[cur].mlen;
|
||||
if (opt[cur].off > ZSTD_REP_MOVE_OPT) {
|
||||
opt[cur].rep[2] = opt[cur-mlen].rep[1];
|
||||
opt[cur].rep[1] = opt[cur-mlen].rep[0];
|
||||
opt[cur].rep[0] = opt[cur].off - ZSTD_REP_MOVE_OPT;
|
||||
} else {
|
||||
opt[cur].rep[2] = (opt[cur].off > 1) ? opt[cur-mlen].rep[1] : opt[cur-mlen].rep[2];
|
||||
opt[cur].rep[1] = (opt[cur].off > 0) ? opt[cur-mlen].rep[0] : opt[cur-mlen].rep[1];
|
||||
/* If opt[cur].off == ZSTD_REP_MOVE_OPT, then mlen != 1.
|
||||
* offset ZSTD_REP_MOVE_OPT is used for the special case
|
||||
* litLength == 0, where offset 0 means something special.
|
||||
* mlen == 1 means the previous byte was stored as a literal,
|
||||
* so they are mutually exclusive.
|
||||
*/
|
||||
assert(!(opt[cur].off == ZSTD_REP_MOVE_OPT && mlen == 1));
|
||||
opt[cur].rep[0] = (opt[cur].off == ZSTD_REP_MOVE_OPT) ? (opt[cur-mlen].rep[0] - 1) : (opt[cur-mlen].rep[opt[cur].off]);
|
||||
}
|
||||
|
||||
best_mlen = minMatch;
|
||||
{ U32 i, last_i = ZSTD_REP_CHECK + (mlen != 1);
|
||||
for (i=(opt[cur].mlen != 1); i<last_i; i++) { /* check rep */
|
||||
const S32 repCur = (i==ZSTD_REP_MOVE_OPT) ? (opt[cur].rep[0] - 1) : opt[cur].rep[i];
|
||||
if ( (repCur > 0) && (repCur < (S32)(inr-prefixStart))
|
||||
&& (ZSTD_readMINMATCH(inr, minMatch) == ZSTD_readMINMATCH(inr - repCur, minMatch))) {
|
||||
mlen = (U32)ZSTD_count(inr+minMatch, inr+minMatch - repCur, iend) + minMatch;
|
||||
|
||||
if (mlen > sufficient_len || cur + mlen >= ZSTD_OPT_NUM) {
|
||||
best_mlen = mlen; best_off = i; last_pos = cur + 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
best_off = i - (opt[cur].mlen != 1);
|
||||
if (mlen > best_mlen) best_mlen = mlen;
|
||||
|
||||
do {
|
||||
if (opt[cur].mlen == 1) {
|
||||
litlen = opt[cur].litlen;
|
||||
if (cur > litlen) {
|
||||
price = opt[cur - litlen].price + ZSTD_getPrice(optStatePtr, litlen, inr-litlen, best_off, mlen - MINMATCH, ultra);
|
||||
} else
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, best_off, mlen - MINMATCH, ultra);
|
||||
} else {
|
||||
litlen = 0;
|
||||
price = opt[cur].price + ZSTD_getPrice(optStatePtr, 0, NULL, best_off, mlen - MINMATCH, ultra);
|
||||
}
|
||||
|
||||
if (cur + mlen > last_pos || price <= opt[cur + mlen].price)
|
||||
SET_PRICE(cur + mlen, mlen, i, litlen, price);
|
||||
mlen--;
|
||||
} while (mlen >= minMatch);
|
||||
} } }
|
||||
|
||||
match_num = ZSTD_BtGetAllMatches_selectMLS(ctx, inr, iend, maxSearches, mls, matches, best_mlen);
|
||||
|
||||
if (match_num > 0 && (matches[match_num-1].len > sufficient_len || cur + matches[match_num-1].len >= ZSTD_OPT_NUM)) {
|
||||
best_mlen = matches[match_num-1].len;
|
||||
best_off = matches[match_num-1].off;
|
||||
last_pos = cur + 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
/* set prices using matches at position = cur */
|
||||
for (u = 0; u < match_num; u++) {
|
||||
mlen = (u>0) ? matches[u-1].len+1 : best_mlen;
|
||||
best_mlen = matches[u].len;
|
||||
|
||||
while (mlen <= best_mlen) {
|
||||
if (opt[cur].mlen == 1) {
|
||||
litlen = opt[cur].litlen;
|
||||
if (cur > litlen)
|
||||
price = opt[cur - litlen].price + ZSTD_getPrice(optStatePtr, litlen, ip+cur-litlen, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
else
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
} else {
|
||||
litlen = 0;
|
||||
price = opt[cur].price + ZSTD_getPrice(optStatePtr, 0, NULL, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
}
|
||||
|
||||
if (cur + mlen > last_pos || (price < opt[cur + mlen].price))
|
||||
SET_PRICE(cur + mlen, mlen, matches[u].off, litlen, price);
|
||||
|
||||
mlen++;
|
||||
} } }
|
||||
|
||||
best_mlen = opt[last_pos].mlen;
|
||||
best_off = opt[last_pos].off;
|
||||
cur = last_pos - best_mlen;
|
||||
|
||||
/* store sequence */
|
||||
_storeSequence: /* cur, last_pos, best_mlen, best_off have to be set */
|
||||
opt[0].mlen = 1;
|
||||
|
||||
while (1) {
|
||||
mlen = opt[cur].mlen;
|
||||
offset = opt[cur].off;
|
||||
opt[cur].mlen = best_mlen;
|
||||
opt[cur].off = best_off;
|
||||
best_mlen = mlen;
|
||||
best_off = offset;
|
||||
if (mlen > cur) break;
|
||||
cur -= mlen;
|
||||
}
|
||||
|
||||
for (u = 0; u <= last_pos;) {
|
||||
u += opt[u].mlen;
|
||||
}
|
||||
|
||||
for (cur=0; cur < last_pos; ) {
|
||||
mlen = opt[cur].mlen;
|
||||
if (mlen == 1) { ip++; cur++; continue; }
|
||||
offset = opt[cur].off;
|
||||
cur += mlen;
|
||||
litLength = (U32)(ip - anchor);
|
||||
|
||||
if (offset > ZSTD_REP_MOVE_OPT) {
|
||||
rep[2] = rep[1];
|
||||
rep[1] = rep[0];
|
||||
rep[0] = offset - ZSTD_REP_MOVE_OPT;
|
||||
offset--;
|
||||
} else {
|
||||
if (offset != 0) {
|
||||
best_off = (offset==ZSTD_REP_MOVE_OPT) ? (rep[0] - 1) : (rep[offset]);
|
||||
if (offset != 1) rep[2] = rep[1];
|
||||
rep[1] = rep[0];
|
||||
rep[0] = best_off;
|
||||
}
|
||||
if (litLength==0) offset--;
|
||||
}
|
||||
|
||||
ZSTD_updatePrice(optStatePtr, litLength, anchor, offset, mlen-MINMATCH);
|
||||
ZSTD_storeSeq(seqStorePtr, litLength, anchor, offset, mlen-MINMATCH);
|
||||
anchor = ip = ip + mlen;
|
||||
} } /* for (cur=0; cur < last_pos; ) */
|
||||
|
||||
/* Save reps for next block */
|
||||
{ int i; for (i=0; i<ZSTD_REP_NUM; i++) seqStorePtr->repToConfirm[i] = rep[i]; }
|
||||
|
||||
/* Return the last literals size */
|
||||
return iend - anchor;
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTD_compressBlock_btopt(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_opt_generic(ctx, src, srcSize, 0);
|
||||
}
|
||||
|
||||
size_t ZSTD_compressBlock_btultra(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_opt_generic(ctx, src, srcSize, 1);
|
||||
}
|
||||
|
||||
|
||||
FORCE_INLINE_TEMPLATE
|
||||
size_t ZSTD_compressBlock_opt_extDict_generic(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize, const int ultra)
|
||||
{
|
||||
seqStore_t* seqStorePtr = &(ctx->seqStore);
|
||||
optState_t* optStatePtr = &(ctx->optState);
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - 8;
|
||||
const BYTE* const base = ctx->base;
|
||||
const U32 lowestIndex = ctx->lowLimit;
|
||||
const U32 dictLimit = ctx->dictLimit;
|
||||
const BYTE* const prefixStart = base + dictLimit;
|
||||
const BYTE* const dictBase = ctx->dictBase;
|
||||
const BYTE* const dictEnd = dictBase + dictLimit;
|
||||
|
||||
const U32 maxSearches = 1U << ctx->appliedParams.cParams.searchLog;
|
||||
const U32 sufficient_len = ctx->appliedParams.cParams.targetLength;
|
||||
const U32 mls = ctx->appliedParams.cParams.searchLength;
|
||||
const U32 minMatch = (ctx->appliedParams.cParams.searchLength == 3) ? 3 : 4;
|
||||
|
||||
ZSTD_optimal_t* opt = optStatePtr->priceTable;
|
||||
ZSTD_match_t* matches = optStatePtr->matchTable;
|
||||
const BYTE* inr;
|
||||
|
||||
/* init */
|
||||
U32 offset, rep[ZSTD_REP_NUM];
|
||||
{ U32 i; for (i=0; i<ZSTD_REP_NUM; i++) rep[i]=seqStorePtr->rep[i]; }
|
||||
|
||||
ctx->nextToUpdate3 = ctx->nextToUpdate;
|
||||
ZSTD_rescaleFreqs(optStatePtr, (const BYTE*)src, srcSize);
|
||||
ip += (ip==prefixStart);
|
||||
|
||||
/* Match Loop */
|
||||
while (ip < ilimit) {
|
||||
U32 cur, match_num, last_pos, litlen, price;
|
||||
U32 u, mlen, best_mlen, best_off, litLength;
|
||||
U32 current = (U32)(ip-base);
|
||||
memset(opt, 0, sizeof(ZSTD_optimal_t));
|
||||
last_pos = 0;
|
||||
opt[0].litlen = (U32)(ip - anchor);
|
||||
|
||||
/* check repCode */
|
||||
{ U32 i, last_i = ZSTD_REP_CHECK + (ip==anchor);
|
||||
for (i = (ip==anchor); i<last_i; i++) {
|
||||
const S32 repCur = (i==ZSTD_REP_MOVE_OPT) ? (rep[0] - 1) : rep[i];
|
||||
const U32 repIndex = (U32)(current - repCur);
|
||||
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* const repMatch = repBase + repIndex;
|
||||
if ( (repCur > 0 && repCur <= (S32)current)
|
||||
&& (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex>lowestIndex)) /* intentional overflow */
|
||||
&& (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch)) ) {
|
||||
/* repcode detected we should take it */
|
||||
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
|
||||
mlen = (U32)ZSTD_count_2segments(ip+minMatch, repMatch+minMatch, iend, repEnd, prefixStart) + minMatch;
|
||||
|
||||
if (mlen > sufficient_len || mlen >= ZSTD_OPT_NUM) {
|
||||
best_mlen = mlen; best_off = i; cur = 0; last_pos = 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
best_off = i - (ip==anchor);
|
||||
litlen = opt[0].litlen;
|
||||
do {
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, best_off, mlen - MINMATCH, ultra);
|
||||
if (mlen > last_pos || price < opt[mlen].price)
|
||||
SET_PRICE(mlen, mlen, i, litlen, price); /* note : macro modifies last_pos */
|
||||
mlen--;
|
||||
} while (mlen >= minMatch);
|
||||
} } }
|
||||
|
||||
match_num = ZSTD_BtGetAllMatches_selectMLS_extDict(ctx, ip, iend, maxSearches, mls, matches, minMatch); /* first search (depth 0) */
|
||||
|
||||
if (!last_pos && !match_num) { ip++; continue; }
|
||||
|
||||
{ U32 i; for (i=0; i<ZSTD_REP_NUM; i++) opt[0].rep[i] = rep[i]; }
|
||||
opt[0].mlen = 1;
|
||||
|
||||
if (match_num && (matches[match_num-1].len > sufficient_len || matches[match_num-1].len >= ZSTD_OPT_NUM)) {
|
||||
best_mlen = matches[match_num-1].len;
|
||||
best_off = matches[match_num-1].off;
|
||||
cur = 0;
|
||||
last_pos = 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
best_mlen = (last_pos) ? last_pos : minMatch;
|
||||
|
||||
/* set prices using matches at position = 0 */
|
||||
for (u = 0; u < match_num; u++) {
|
||||
mlen = (u>0) ? matches[u-1].len+1 : best_mlen;
|
||||
best_mlen = matches[u].len;
|
||||
litlen = opt[0].litlen;
|
||||
while (mlen <= best_mlen) {
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
if (mlen > last_pos || price < opt[mlen].price)
|
||||
SET_PRICE(mlen, mlen, matches[u].off, litlen, price);
|
||||
mlen++;
|
||||
} }
|
||||
|
||||
if (last_pos < minMatch) {
|
||||
ip++; continue;
|
||||
}
|
||||
|
||||
/* check further positions */
|
||||
for (cur = 1; cur <= last_pos; cur++) {
|
||||
inr = ip + cur;
|
||||
|
||||
if (opt[cur-1].mlen == 1) {
|
||||
litlen = opt[cur-1].litlen + 1;
|
||||
if (cur > litlen) {
|
||||
price = opt[cur - litlen].price + ZSTD_getLiteralPrice(optStatePtr, litlen, inr-litlen);
|
||||
} else
|
||||
price = ZSTD_getLiteralPrice(optStatePtr, litlen, anchor);
|
||||
} else {
|
||||
litlen = 1;
|
||||
price = opt[cur - 1].price + ZSTD_getLiteralPrice(optStatePtr, litlen, inr-1);
|
||||
}
|
||||
|
||||
if (cur > last_pos || price <= opt[cur].price)
|
||||
SET_PRICE(cur, 1, 0, litlen, price);
|
||||
|
||||
if (cur == last_pos) break;
|
||||
|
||||
if (inr > ilimit) /* last match must start at a minimum distance of 8 from oend */
|
||||
continue;
|
||||
|
||||
mlen = opt[cur].mlen;
|
||||
if (opt[cur].off > ZSTD_REP_MOVE_OPT) {
|
||||
opt[cur].rep[2] = opt[cur-mlen].rep[1];
|
||||
opt[cur].rep[1] = opt[cur-mlen].rep[0];
|
||||
opt[cur].rep[0] = opt[cur].off - ZSTD_REP_MOVE_OPT;
|
||||
} else {
|
||||
opt[cur].rep[2] = (opt[cur].off > 1) ? opt[cur-mlen].rep[1] : opt[cur-mlen].rep[2];
|
||||
opt[cur].rep[1] = (opt[cur].off > 0) ? opt[cur-mlen].rep[0] : opt[cur-mlen].rep[1];
|
||||
assert(!(opt[cur].off == ZSTD_REP_MOVE_OPT && mlen == 1));
|
||||
opt[cur].rep[0] = (opt[cur].off == ZSTD_REP_MOVE_OPT) ? (opt[cur-mlen].rep[0] - 1) : (opt[cur-mlen].rep[opt[cur].off]);
|
||||
}
|
||||
|
||||
best_mlen = minMatch;
|
||||
{ U32 i, last_i = ZSTD_REP_CHECK + (mlen != 1);
|
||||
for (i = (mlen != 1); i<last_i; i++) {
|
||||
const S32 repCur = (i==ZSTD_REP_MOVE_OPT) ? (opt[cur].rep[0] - 1) : opt[cur].rep[i];
|
||||
const U32 repIndex = (U32)(current+cur - repCur);
|
||||
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* const repMatch = repBase + repIndex;
|
||||
if ( (repCur > 0 && repCur <= (S32)(current+cur))
|
||||
&& (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex>lowestIndex)) /* intentional overflow */
|
||||
&& (ZSTD_readMINMATCH(inr, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch)) ) {
|
||||
/* repcode detected */
|
||||
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
|
||||
mlen = (U32)ZSTD_count_2segments(inr+minMatch, repMatch+minMatch, iend, repEnd, prefixStart) + minMatch;
|
||||
|
||||
if (mlen > sufficient_len || cur + mlen >= ZSTD_OPT_NUM) {
|
||||
best_mlen = mlen; best_off = i; last_pos = cur + 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
best_off = i - (opt[cur].mlen != 1);
|
||||
if (mlen > best_mlen) best_mlen = mlen;
|
||||
|
||||
do {
|
||||
if (opt[cur].mlen == 1) {
|
||||
litlen = opt[cur].litlen;
|
||||
if (cur > litlen) {
|
||||
price = opt[cur - litlen].price + ZSTD_getPrice(optStatePtr, litlen, inr-litlen, best_off, mlen - MINMATCH, ultra);
|
||||
} else
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, best_off, mlen - MINMATCH, ultra);
|
||||
} else {
|
||||
litlen = 0;
|
||||
price = opt[cur].price + ZSTD_getPrice(optStatePtr, 0, NULL, best_off, mlen - MINMATCH, ultra);
|
||||
}
|
||||
|
||||
if (cur + mlen > last_pos || price <= opt[cur + mlen].price)
|
||||
SET_PRICE(cur + mlen, mlen, i, litlen, price);
|
||||
mlen--;
|
||||
} while (mlen >= minMatch);
|
||||
} } }
|
||||
|
||||
match_num = ZSTD_BtGetAllMatches_selectMLS_extDict(ctx, inr, iend, maxSearches, mls, matches, minMatch);
|
||||
|
||||
if (match_num > 0 && (matches[match_num-1].len > sufficient_len || cur + matches[match_num-1].len >= ZSTD_OPT_NUM)) {
|
||||
best_mlen = matches[match_num-1].len;
|
||||
best_off = matches[match_num-1].off;
|
||||
last_pos = cur + 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
/* set prices using matches at position = cur */
|
||||
for (u = 0; u < match_num; u++) {
|
||||
mlen = (u>0) ? matches[u-1].len+1 : best_mlen;
|
||||
best_mlen = matches[u].len;
|
||||
|
||||
while (mlen <= best_mlen) {
|
||||
if (opt[cur].mlen == 1) {
|
||||
litlen = opt[cur].litlen;
|
||||
if (cur > litlen)
|
||||
price = opt[cur - litlen].price + ZSTD_getPrice(optStatePtr, litlen, ip+cur-litlen, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
else
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
} else {
|
||||
litlen = 0;
|
||||
price = opt[cur].price + ZSTD_getPrice(optStatePtr, 0, NULL, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
}
|
||||
|
||||
if (cur + mlen > last_pos || (price < opt[cur + mlen].price))
|
||||
SET_PRICE(cur + mlen, mlen, matches[u].off, litlen, price);
|
||||
|
||||
mlen++;
|
||||
} } } /* for (cur = 1; cur <= last_pos; cur++) */
|
||||
|
||||
best_mlen = opt[last_pos].mlen;
|
||||
best_off = opt[last_pos].off;
|
||||
cur = last_pos - best_mlen;
|
||||
|
||||
/* store sequence */
|
||||
_storeSequence: /* cur, last_pos, best_mlen, best_off have to be set */
|
||||
opt[0].mlen = 1;
|
||||
|
||||
while (1) {
|
||||
mlen = opt[cur].mlen;
|
||||
offset = opt[cur].off;
|
||||
opt[cur].mlen = best_mlen;
|
||||
opt[cur].off = best_off;
|
||||
best_mlen = mlen;
|
||||
best_off = offset;
|
||||
if (mlen > cur) break;
|
||||
cur -= mlen;
|
||||
}
|
||||
|
||||
for (u = 0; u <= last_pos; ) {
|
||||
u += opt[u].mlen;
|
||||
}
|
||||
|
||||
for (cur=0; cur < last_pos; ) {
|
||||
mlen = opt[cur].mlen;
|
||||
if (mlen == 1) { ip++; cur++; continue; }
|
||||
offset = opt[cur].off;
|
||||
cur += mlen;
|
||||
litLength = (U32)(ip - anchor);
|
||||
|
||||
if (offset > ZSTD_REP_MOVE_OPT) {
|
||||
rep[2] = rep[1];
|
||||
rep[1] = rep[0];
|
||||
rep[0] = offset - ZSTD_REP_MOVE_OPT;
|
||||
offset--;
|
||||
} else {
|
||||
if (offset != 0) {
|
||||
best_off = (offset==ZSTD_REP_MOVE_OPT) ? (rep[0] - 1) : (rep[offset]);
|
||||
if (offset != 1) rep[2] = rep[1];
|
||||
rep[1] = rep[0];
|
||||
rep[0] = best_off;
|
||||
}
|
||||
|
||||
if (litLength==0) offset--;
|
||||
}
|
||||
|
||||
ZSTD_updatePrice(optStatePtr, litLength, anchor, offset, mlen-MINMATCH);
|
||||
ZSTD_storeSeq(seqStorePtr, litLength, anchor, offset, mlen-MINMATCH);
|
||||
anchor = ip = ip + mlen;
|
||||
} } /* for (cur=0; cur < last_pos; ) */
|
||||
|
||||
/* Save reps for next block */
|
||||
{ int i; for (i=0; i<ZSTD_REP_NUM; i++) seqStorePtr->repToConfirm[i] = rep[i]; }
|
||||
|
||||
/* Return the last literals size */
|
||||
return iend - anchor;
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTD_compressBlock_btopt_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_opt_extDict_generic(ctx, src, srcSize, 0);
|
||||
}
|
||||
|
||||
size_t ZSTD_compressBlock_btultra_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize)
|
||||
{
|
||||
return ZSTD_compressBlock_opt_extDict_generic(ctx, src, srcSize, 1);
|
||||
}
|
@ -5,934 +5,26 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef ZSTD_OPT_H
|
||||
#define ZSTD_OPT_H
|
||||
|
||||
/* Note : this file is intended to be included within zstd_compress.c */
|
||||
#include "zstd_compress.h"
|
||||
|
||||
#if defined (__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#ifndef ZSTD_OPT_H_91842398743
|
||||
#define ZSTD_OPT_H_91842398743
|
||||
size_t ZSTD_compressBlock_btopt(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
size_t ZSTD_compressBlock_btultra(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
|
||||
size_t ZSTD_compressBlock_btopt_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
size_t ZSTD_compressBlock_btultra_extDict(ZSTD_CCtx* ctx, const void* src, size_t srcSize);
|
||||
|
||||
#define ZSTD_LITFREQ_ADD 2
|
||||
#define ZSTD_FREQ_DIV 4
|
||||
#define ZSTD_MAX_PRICE (1<<30)
|
||||
|
||||
/*-*************************************
|
||||
* Price functions for optimal parser
|
||||
***************************************/
|
||||
static void ZSTD_setLog2Prices(optState_t* optPtr)
|
||||
{
|
||||
optPtr->log2matchLengthSum = ZSTD_highbit32(optPtr->matchLengthSum+1);
|
||||
optPtr->log2litLengthSum = ZSTD_highbit32(optPtr->litLengthSum+1);
|
||||
optPtr->log2litSum = ZSTD_highbit32(optPtr->litSum+1);
|
||||
optPtr->log2offCodeSum = ZSTD_highbit32(optPtr->offCodeSum+1);
|
||||
optPtr->factor = 1 + ((optPtr->litSum>>5) / optPtr->litLengthSum) + ((optPtr->litSum<<1) / (optPtr->litSum + optPtr->matchSum));
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
|
||||
static void ZSTD_rescaleFreqs(optState_t* optPtr, const BYTE* src, size_t srcSize)
|
||||
{
|
||||
unsigned u;
|
||||
|
||||
optPtr->cachedLiterals = NULL;
|
||||
optPtr->cachedPrice = optPtr->cachedLitLength = 0;
|
||||
optPtr->staticPrices = 0;
|
||||
|
||||
if (optPtr->litLengthSum == 0) {
|
||||
if (srcSize <= 1024) optPtr->staticPrices = 1;
|
||||
|
||||
assert(optPtr->litFreq!=NULL);
|
||||
for (u=0; u<=MaxLit; u++)
|
||||
optPtr->litFreq[u] = 0;
|
||||
for (u=0; u<srcSize; u++)
|
||||
optPtr->litFreq[src[u]]++;
|
||||
|
||||
optPtr->litSum = 0;
|
||||
optPtr->litLengthSum = MaxLL+1;
|
||||
optPtr->matchLengthSum = MaxML+1;
|
||||
optPtr->offCodeSum = (MaxOff+1);
|
||||
optPtr->matchSum = (ZSTD_LITFREQ_ADD<<Litbits);
|
||||
|
||||
for (u=0; u<=MaxLit; u++) {
|
||||
optPtr->litFreq[u] = 1 + (optPtr->litFreq[u]>>ZSTD_FREQ_DIV);
|
||||
optPtr->litSum += optPtr->litFreq[u];
|
||||
}
|
||||
for (u=0; u<=MaxLL; u++)
|
||||
optPtr->litLengthFreq[u] = 1;
|
||||
for (u=0; u<=MaxML; u++)
|
||||
optPtr->matchLengthFreq[u] = 1;
|
||||
for (u=0; u<=MaxOff; u++)
|
||||
optPtr->offCodeFreq[u] = 1;
|
||||
} else {
|
||||
optPtr->matchLengthSum = 0;
|
||||
optPtr->litLengthSum = 0;
|
||||
optPtr->offCodeSum = 0;
|
||||
optPtr->matchSum = 0;
|
||||
optPtr->litSum = 0;
|
||||
|
||||
for (u=0; u<=MaxLit; u++) {
|
||||
optPtr->litFreq[u] = 1 + (optPtr->litFreq[u]>>(ZSTD_FREQ_DIV+1));
|
||||
optPtr->litSum += optPtr->litFreq[u];
|
||||
}
|
||||
for (u=0; u<=MaxLL; u++) {
|
||||
optPtr->litLengthFreq[u] = 1 + (optPtr->litLengthFreq[u]>>(ZSTD_FREQ_DIV+1));
|
||||
optPtr->litLengthSum += optPtr->litLengthFreq[u];
|
||||
}
|
||||
for (u=0; u<=MaxML; u++) {
|
||||
optPtr->matchLengthFreq[u] = 1 + (optPtr->matchLengthFreq[u]>>ZSTD_FREQ_DIV);
|
||||
optPtr->matchLengthSum += optPtr->matchLengthFreq[u];
|
||||
optPtr->matchSum += optPtr->matchLengthFreq[u] * (u + 3);
|
||||
}
|
||||
optPtr->matchSum *= ZSTD_LITFREQ_ADD;
|
||||
for (u=0; u<=MaxOff; u++) {
|
||||
optPtr->offCodeFreq[u] = 1 + (optPtr->offCodeFreq[u]>>ZSTD_FREQ_DIV);
|
||||
optPtr->offCodeSum += optPtr->offCodeFreq[u];
|
||||
}
|
||||
}
|
||||
|
||||
ZSTD_setLog2Prices(optPtr);
|
||||
}
|
||||
|
||||
|
||||
static U32 ZSTD_getLiteralPrice(optState_t* optPtr, U32 litLength, const BYTE* literals)
|
||||
{
|
||||
U32 price, u;
|
||||
|
||||
if (optPtr->staticPrices)
|
||||
return ZSTD_highbit32((U32)litLength+1) + (litLength*6);
|
||||
|
||||
if (litLength == 0)
|
||||
return optPtr->log2litLengthSum - ZSTD_highbit32(optPtr->litLengthFreq[0]+1);
|
||||
|
||||
/* literals */
|
||||
if (optPtr->cachedLiterals == literals) {
|
||||
U32 const additional = litLength - optPtr->cachedLitLength;
|
||||
const BYTE* literals2 = optPtr->cachedLiterals + optPtr->cachedLitLength;
|
||||
price = optPtr->cachedPrice + additional * optPtr->log2litSum;
|
||||
for (u=0; u < additional; u++)
|
||||
price -= ZSTD_highbit32(optPtr->litFreq[literals2[u]]+1);
|
||||
optPtr->cachedPrice = price;
|
||||
optPtr->cachedLitLength = litLength;
|
||||
} else {
|
||||
price = litLength * optPtr->log2litSum;
|
||||
for (u=0; u < litLength; u++)
|
||||
price -= ZSTD_highbit32(optPtr->litFreq[literals[u]]+1);
|
||||
|
||||
if (litLength >= 12) {
|
||||
optPtr->cachedLiterals = literals;
|
||||
optPtr->cachedPrice = price;
|
||||
optPtr->cachedLitLength = litLength;
|
||||
}
|
||||
}
|
||||
|
||||
/* literal Length */
|
||||
{ const BYTE LL_deltaCode = 19;
|
||||
const BYTE llCode = (litLength>63) ? (BYTE)ZSTD_highbit32(litLength) + LL_deltaCode : LL_Code[litLength];
|
||||
price += LL_bits[llCode] + optPtr->log2litLengthSum - ZSTD_highbit32(optPtr->litLengthFreq[llCode]+1);
|
||||
}
|
||||
|
||||
return price;
|
||||
}
|
||||
|
||||
|
||||
FORCE_INLINE_TEMPLATE U32 ZSTD_getPrice(optState_t* optPtr, U32 litLength, const BYTE* literals, U32 offset, U32 matchLength, const int ultra)
|
||||
{
|
||||
/* offset */
|
||||
U32 price;
|
||||
BYTE const offCode = (BYTE)ZSTD_highbit32(offset+1);
|
||||
|
||||
if (optPtr->staticPrices)
|
||||
return ZSTD_getLiteralPrice(optPtr, litLength, literals) + ZSTD_highbit32((U32)matchLength+1) + 16 + offCode;
|
||||
|
||||
price = offCode + optPtr->log2offCodeSum - ZSTD_highbit32(optPtr->offCodeFreq[offCode]+1);
|
||||
if (!ultra && offCode >= 20) price += (offCode-19)*2;
|
||||
|
||||
/* match Length */
|
||||
{ const BYTE ML_deltaCode = 36;
|
||||
const BYTE mlCode = (matchLength>127) ? (BYTE)ZSTD_highbit32(matchLength) + ML_deltaCode : ML_Code[matchLength];
|
||||
price += ML_bits[mlCode] + optPtr->log2matchLengthSum - ZSTD_highbit32(optPtr->matchLengthFreq[mlCode]+1);
|
||||
}
|
||||
|
||||
return price + ZSTD_getLiteralPrice(optPtr, litLength, literals) + optPtr->factor;
|
||||
}
|
||||
|
||||
|
||||
static void ZSTD_updatePrice(optState_t* optPtr, U32 litLength, const BYTE* literals, U32 offset, U32 matchLength)
|
||||
{
|
||||
U32 u;
|
||||
|
||||
/* literals */
|
||||
optPtr->litSum += litLength*ZSTD_LITFREQ_ADD;
|
||||
for (u=0; u < litLength; u++)
|
||||
optPtr->litFreq[literals[u]] += ZSTD_LITFREQ_ADD;
|
||||
|
||||
/* literal Length */
|
||||
{ const BYTE LL_deltaCode = 19;
|
||||
const BYTE llCode = (litLength>63) ? (BYTE)ZSTD_highbit32(litLength) + LL_deltaCode : LL_Code[litLength];
|
||||
optPtr->litLengthFreq[llCode]++;
|
||||
optPtr->litLengthSum++;
|
||||
}
|
||||
|
||||
/* match offset */
|
||||
{ BYTE const offCode = (BYTE)ZSTD_highbit32(offset+1);
|
||||
optPtr->offCodeSum++;
|
||||
optPtr->offCodeFreq[offCode]++;
|
||||
}
|
||||
|
||||
/* match Length */
|
||||
{ const BYTE ML_deltaCode = 36;
|
||||
const BYTE mlCode = (matchLength>127) ? (BYTE)ZSTD_highbit32(matchLength) + ML_deltaCode : ML_Code[matchLength];
|
||||
optPtr->matchLengthFreq[mlCode]++;
|
||||
optPtr->matchLengthSum++;
|
||||
}
|
||||
|
||||
ZSTD_setLog2Prices(optPtr);
|
||||
}
|
||||
|
||||
|
||||
#define SET_PRICE(pos, mlen_, offset_, litlen_, price_) \
|
||||
{ \
|
||||
while (last_pos < pos) { opt[last_pos+1].price = ZSTD_MAX_PRICE; last_pos++; } \
|
||||
opt[pos].mlen = mlen_; \
|
||||
opt[pos].off = offset_; \
|
||||
opt[pos].litlen = litlen_; \
|
||||
opt[pos].price = price_; \
|
||||
}
|
||||
|
||||
|
||||
/* function safe only for comparisons */
|
||||
static U32 ZSTD_readMINMATCH(const void* memPtr, U32 length)
|
||||
{
|
||||
switch (length)
|
||||
{
|
||||
default :
|
||||
case 4 : return MEM_read32(memPtr);
|
||||
case 3 : if (MEM_isLittleEndian())
|
||||
return MEM_read32(memPtr)<<8;
|
||||
else
|
||||
return MEM_read32(memPtr)>>8;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/* Update hashTable3 up to ip (excluded)
|
||||
Assumption : always within prefix (i.e. not within extDict) */
|
||||
static
|
||||
U32 ZSTD_insertAndFindFirstIndexHash3 (ZSTD_CCtx* zc, const BYTE* ip)
|
||||
{
|
||||
U32* const hashTable3 = zc->hashTable3;
|
||||
U32 const hashLog3 = zc->hashLog3;
|
||||
const BYTE* const base = zc->base;
|
||||
U32 idx = zc->nextToUpdate3;
|
||||
const U32 target = zc->nextToUpdate3 = (U32)(ip - base);
|
||||
const size_t hash3 = ZSTD_hash3Ptr(ip, hashLog3);
|
||||
|
||||
while(idx < target) {
|
||||
hashTable3[ZSTD_hash3Ptr(base+idx, hashLog3)] = idx;
|
||||
idx++;
|
||||
}
|
||||
|
||||
return hashTable3[hash3];
|
||||
}
|
||||
|
||||
|
||||
/*-*************************************
|
||||
* Binary Tree search
|
||||
***************************************/
|
||||
static U32 ZSTD_insertBtAndGetAllMatches (
|
||||
ZSTD_CCtx* zc,
|
||||
const BYTE* const ip, const BYTE* const iLimit,
|
||||
U32 nbCompares, const U32 mls,
|
||||
U32 extDict, ZSTD_match_t* matches, const U32 minMatchLen)
|
||||
{
|
||||
const BYTE* const base = zc->base;
|
||||
const U32 current = (U32)(ip-base);
|
||||
const U32 hashLog = zc->appliedParams.cParams.hashLog;
|
||||
const size_t h = ZSTD_hashPtr(ip, hashLog, mls);
|
||||
U32* const hashTable = zc->hashTable;
|
||||
U32 matchIndex = hashTable[h];
|
||||
U32* const bt = zc->chainTable;
|
||||
const U32 btLog = zc->appliedParams.cParams.chainLog - 1;
|
||||
const U32 btMask= (1U << btLog) - 1;
|
||||
size_t commonLengthSmaller=0, commonLengthLarger=0;
|
||||
const BYTE* const dictBase = zc->dictBase;
|
||||
const U32 dictLimit = zc->dictLimit;
|
||||
const BYTE* const dictEnd = dictBase + dictLimit;
|
||||
const BYTE* const prefixStart = base + dictLimit;
|
||||
const U32 btLow = btMask >= current ? 0 : current - btMask;
|
||||
const U32 windowLow = zc->lowLimit;
|
||||
U32* smallerPtr = bt + 2*(current&btMask);
|
||||
U32* largerPtr = bt + 2*(current&btMask) + 1;
|
||||
U32 matchEndIdx = current+8;
|
||||
U32 dummy32; /* to be nullified at the end */
|
||||
U32 mnum = 0;
|
||||
|
||||
const U32 minMatch = (mls == 3) ? 3 : 4;
|
||||
size_t bestLength = minMatchLen-1;
|
||||
|
||||
if (minMatch == 3) { /* HC3 match finder */
|
||||
U32 const matchIndex3 = ZSTD_insertAndFindFirstIndexHash3 (zc, ip);
|
||||
if (matchIndex3>windowLow && (current - matchIndex3 < (1<<18))) {
|
||||
const BYTE* match;
|
||||
size_t currentMl=0;
|
||||
if ((!extDict) || matchIndex3 >= dictLimit) {
|
||||
match = base + matchIndex3;
|
||||
if (match[bestLength] == ip[bestLength]) currentMl = ZSTD_count(ip, match, iLimit);
|
||||
} else {
|
||||
match = dictBase + matchIndex3;
|
||||
if (ZSTD_readMINMATCH(match, MINMATCH) == ZSTD_readMINMATCH(ip, MINMATCH)) /* assumption : matchIndex3 <= dictLimit-4 (by table construction) */
|
||||
currentMl = ZSTD_count_2segments(ip+MINMATCH, match+MINMATCH, iLimit, dictEnd, prefixStart) + MINMATCH;
|
||||
}
|
||||
|
||||
/* save best solution */
|
||||
if (currentMl > bestLength) {
|
||||
bestLength = currentMl;
|
||||
matches[mnum].off = ZSTD_REP_MOVE_OPT + current - matchIndex3;
|
||||
matches[mnum].len = (U32)currentMl;
|
||||
mnum++;
|
||||
if (currentMl > ZSTD_OPT_NUM) goto update;
|
||||
if (ip+currentMl == iLimit) goto update; /* best possible, and avoid read overflow*/
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
hashTable[h] = current; /* Update Hash Table */
|
||||
|
||||
while (nbCompares-- && (matchIndex > windowLow)) {
|
||||
U32* nextPtr = bt + 2*(matchIndex & btMask);
|
||||
size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
|
||||
const BYTE* match;
|
||||
|
||||
if ((!extDict) || (matchIndex+matchLength >= dictLimit)) {
|
||||
match = base + matchIndex;
|
||||
if (match[matchLength] == ip[matchLength]) {
|
||||
matchLength += ZSTD_count(ip+matchLength+1, match+matchLength+1, iLimit) +1;
|
||||
}
|
||||
} else {
|
||||
match = dictBase + matchIndex;
|
||||
matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iLimit, dictEnd, prefixStart);
|
||||
if (matchIndex+matchLength >= dictLimit)
|
||||
match = base + matchIndex; /* to prepare for next usage of match[matchLength] */
|
||||
}
|
||||
|
||||
if (matchLength > bestLength) {
|
||||
if (matchLength > matchEndIdx - matchIndex) matchEndIdx = matchIndex + (U32)matchLength;
|
||||
bestLength = matchLength;
|
||||
matches[mnum].off = ZSTD_REP_MOVE_OPT + current - matchIndex;
|
||||
matches[mnum].len = (U32)matchLength;
|
||||
mnum++;
|
||||
if (matchLength > ZSTD_OPT_NUM) break;
|
||||
if (ip+matchLength == iLimit) /* equal : no way to know if inf or sup */
|
||||
break; /* drop, to guarantee consistency (miss a little bit of compression) */
|
||||
}
|
||||
|
||||
if (match[matchLength] < ip[matchLength]) {
|
||||
/* match is smaller than current */
|
||||
*smallerPtr = matchIndex; /* update smaller idx */
|
||||
commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
|
||||
if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */
|
||||
smallerPtr = nextPtr+1; /* new "smaller" => larger of match */
|
||||
matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
|
||||
} else {
|
||||
/* match is larger than current */
|
||||
*largerPtr = matchIndex;
|
||||
commonLengthLarger = matchLength;
|
||||
if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */
|
||||
largerPtr = nextPtr;
|
||||
matchIndex = nextPtr[0];
|
||||
} }
|
||||
|
||||
*smallerPtr = *largerPtr = 0;
|
||||
|
||||
update:
|
||||
zc->nextToUpdate = (matchEndIdx > current + 8) ? matchEndIdx - 8 : current+1;
|
||||
return mnum;
|
||||
}
|
||||
|
||||
|
||||
/** Tree updater, providing best match */
|
||||
static U32 ZSTD_BtGetAllMatches (
|
||||
ZSTD_CCtx* zc,
|
||||
const BYTE* const ip, const BYTE* const iLimit,
|
||||
const U32 maxNbAttempts, const U32 mls, ZSTD_match_t* matches, const U32 minMatchLen)
|
||||
{
|
||||
if (ip < zc->base + zc->nextToUpdate) return 0; /* skipped area */
|
||||
ZSTD_updateTree(zc, ip, iLimit, maxNbAttempts, mls);
|
||||
return ZSTD_insertBtAndGetAllMatches(zc, ip, iLimit, maxNbAttempts, mls, 0, matches, minMatchLen);
|
||||
}
|
||||
|
||||
|
||||
static U32 ZSTD_BtGetAllMatches_selectMLS (
|
||||
ZSTD_CCtx* zc, /* Index table will be updated */
|
||||
const BYTE* ip, const BYTE* const iHighLimit,
|
||||
const U32 maxNbAttempts, const U32 matchLengthSearch, ZSTD_match_t* matches, const U32 minMatchLen)
|
||||
{
|
||||
switch(matchLengthSearch)
|
||||
{
|
||||
case 3 : return ZSTD_BtGetAllMatches(zc, ip, iHighLimit, maxNbAttempts, 3, matches, minMatchLen);
|
||||
default :
|
||||
case 4 : return ZSTD_BtGetAllMatches(zc, ip, iHighLimit, maxNbAttempts, 4, matches, minMatchLen);
|
||||
case 5 : return ZSTD_BtGetAllMatches(zc, ip, iHighLimit, maxNbAttempts, 5, matches, minMatchLen);
|
||||
case 7 :
|
||||
case 6 : return ZSTD_BtGetAllMatches(zc, ip, iHighLimit, maxNbAttempts, 6, matches, minMatchLen);
|
||||
}
|
||||
}
|
||||
|
||||
/** Tree updater, providing best match */
|
||||
static U32 ZSTD_BtGetAllMatches_extDict (
|
||||
ZSTD_CCtx* zc,
|
||||
const BYTE* const ip, const BYTE* const iLimit,
|
||||
const U32 maxNbAttempts, const U32 mls, ZSTD_match_t* matches, const U32 minMatchLen)
|
||||
{
|
||||
if (ip < zc->base + zc->nextToUpdate) return 0; /* skipped area */
|
||||
ZSTD_updateTree_extDict(zc, ip, iLimit, maxNbAttempts, mls);
|
||||
return ZSTD_insertBtAndGetAllMatches(zc, ip, iLimit, maxNbAttempts, mls, 1, matches, minMatchLen);
|
||||
}
|
||||
|
||||
|
||||
static U32 ZSTD_BtGetAllMatches_selectMLS_extDict (
|
||||
ZSTD_CCtx* zc, /* Index table will be updated */
|
||||
const BYTE* ip, const BYTE* const iHighLimit,
|
||||
const U32 maxNbAttempts, const U32 matchLengthSearch, ZSTD_match_t* matches, const U32 minMatchLen)
|
||||
{
|
||||
switch(matchLengthSearch)
|
||||
{
|
||||
case 3 : return ZSTD_BtGetAllMatches_extDict(zc, ip, iHighLimit, maxNbAttempts, 3, matches, minMatchLen);
|
||||
default :
|
||||
case 4 : return ZSTD_BtGetAllMatches_extDict(zc, ip, iHighLimit, maxNbAttempts, 4, matches, minMatchLen);
|
||||
case 5 : return ZSTD_BtGetAllMatches_extDict(zc, ip, iHighLimit, maxNbAttempts, 5, matches, minMatchLen);
|
||||
case 7 :
|
||||
case 6 : return ZSTD_BtGetAllMatches_extDict(zc, ip, iHighLimit, maxNbAttempts, 6, matches, minMatchLen);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*-*******************************
|
||||
* Optimal parser
|
||||
*********************************/
|
||||
FORCE_INLINE_TEMPLATE
|
||||
void ZSTD_compressBlock_opt_generic(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize, const int ultra)
|
||||
{
|
||||
seqStore_t* seqStorePtr = &(ctx->seqStore);
|
||||
optState_t* optStatePtr = &(ctx->optState);
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - 8;
|
||||
const BYTE* const base = ctx->base;
|
||||
const BYTE* const prefixStart = base + ctx->dictLimit;
|
||||
|
||||
const U32 maxSearches = 1U << ctx->appliedParams.cParams.searchLog;
|
||||
const U32 sufficient_len = ctx->appliedParams.cParams.targetLength;
|
||||
const U32 mls = ctx->appliedParams.cParams.searchLength;
|
||||
const U32 minMatch = (ctx->appliedParams.cParams.searchLength == 3) ? 3 : 4;
|
||||
|
||||
ZSTD_optimal_t* opt = optStatePtr->priceTable;
|
||||
ZSTD_match_t* matches = optStatePtr->matchTable;
|
||||
const BYTE* inr;
|
||||
U32 offset, rep[ZSTD_REP_NUM];
|
||||
|
||||
/* init */
|
||||
ctx->nextToUpdate3 = ctx->nextToUpdate;
|
||||
ZSTD_rescaleFreqs(optStatePtr, (const BYTE*)src, srcSize);
|
||||
ip += (ip==prefixStart);
|
||||
{ U32 i; for (i=0; i<ZSTD_REP_NUM; i++) rep[i]=seqStorePtr->rep[i]; }
|
||||
|
||||
/* Match Loop */
|
||||
while (ip < ilimit) {
|
||||
U32 cur, match_num, last_pos, litlen, price;
|
||||
U32 u, mlen, best_mlen, best_off, litLength;
|
||||
memset(opt, 0, sizeof(ZSTD_optimal_t));
|
||||
last_pos = 0;
|
||||
litlen = (U32)(ip - anchor);
|
||||
|
||||
/* check repCode */
|
||||
{ U32 i, last_i = ZSTD_REP_CHECK + (ip==anchor);
|
||||
for (i=(ip == anchor); i<last_i; i++) {
|
||||
const S32 repCur = (i==ZSTD_REP_MOVE_OPT) ? (rep[0] - 1) : rep[i];
|
||||
if ( (repCur > 0) && (repCur < (S32)(ip-prefixStart))
|
||||
&& (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(ip - repCur, minMatch))) {
|
||||
mlen = (U32)ZSTD_count(ip+minMatch, ip+minMatch-repCur, iend) + minMatch;
|
||||
if (mlen > sufficient_len || mlen >= ZSTD_OPT_NUM) {
|
||||
best_mlen = mlen; best_off = i; cur = 0; last_pos = 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
best_off = i - (ip == anchor);
|
||||
do {
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, best_off, mlen - MINMATCH, ultra);
|
||||
if (mlen > last_pos || price < opt[mlen].price)
|
||||
SET_PRICE(mlen, mlen, i, litlen, price); /* note : macro modifies last_pos */
|
||||
mlen--;
|
||||
} while (mlen >= minMatch);
|
||||
} } }
|
||||
|
||||
match_num = ZSTD_BtGetAllMatches_selectMLS(ctx, ip, iend, maxSearches, mls, matches, minMatch);
|
||||
|
||||
if (!last_pos && !match_num) { ip++; continue; }
|
||||
|
||||
if (match_num && (matches[match_num-1].len > sufficient_len || matches[match_num-1].len >= ZSTD_OPT_NUM)) {
|
||||
best_mlen = matches[match_num-1].len;
|
||||
best_off = matches[match_num-1].off;
|
||||
cur = 0;
|
||||
last_pos = 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
/* set prices using matches at position = 0 */
|
||||
best_mlen = (last_pos) ? last_pos : minMatch;
|
||||
for (u = 0; u < match_num; u++) {
|
||||
mlen = (u>0) ? matches[u-1].len+1 : best_mlen;
|
||||
best_mlen = matches[u].len;
|
||||
while (mlen <= best_mlen) {
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
if (mlen > last_pos || price < opt[mlen].price)
|
||||
SET_PRICE(mlen, mlen, matches[u].off, litlen, price); /* note : macro modifies last_pos */
|
||||
mlen++;
|
||||
} }
|
||||
|
||||
if (last_pos < minMatch) { ip++; continue; }
|
||||
|
||||
/* initialize opt[0] */
|
||||
{ U32 i ; for (i=0; i<ZSTD_REP_NUM; i++) opt[0].rep[i] = rep[i]; }
|
||||
opt[0].mlen = 1;
|
||||
opt[0].litlen = litlen;
|
||||
|
||||
/* check further positions */
|
||||
for (cur = 1; cur <= last_pos; cur++) {
|
||||
inr = ip + cur;
|
||||
|
||||
if (opt[cur-1].mlen == 1) {
|
||||
litlen = opt[cur-1].litlen + 1;
|
||||
if (cur > litlen) {
|
||||
price = opt[cur - litlen].price + ZSTD_getLiteralPrice(optStatePtr, litlen, inr-litlen);
|
||||
} else
|
||||
price = ZSTD_getLiteralPrice(optStatePtr, litlen, anchor);
|
||||
} else {
|
||||
litlen = 1;
|
||||
price = opt[cur - 1].price + ZSTD_getLiteralPrice(optStatePtr, litlen, inr-1);
|
||||
}
|
||||
|
||||
if (cur > last_pos || price <= opt[cur].price)
|
||||
SET_PRICE(cur, 1, 0, litlen, price);
|
||||
|
||||
if (cur == last_pos) break;
|
||||
|
||||
if (inr > ilimit) /* last match must start at a minimum distance of 8 from oend */
|
||||
continue;
|
||||
|
||||
mlen = opt[cur].mlen;
|
||||
if (opt[cur].off > ZSTD_REP_MOVE_OPT) {
|
||||
opt[cur].rep[2] = opt[cur-mlen].rep[1];
|
||||
opt[cur].rep[1] = opt[cur-mlen].rep[0];
|
||||
opt[cur].rep[0] = opt[cur].off - ZSTD_REP_MOVE_OPT;
|
||||
} else {
|
||||
opt[cur].rep[2] = (opt[cur].off > 1) ? opt[cur-mlen].rep[1] : opt[cur-mlen].rep[2];
|
||||
opt[cur].rep[1] = (opt[cur].off > 0) ? opt[cur-mlen].rep[0] : opt[cur-mlen].rep[1];
|
||||
opt[cur].rep[0] = ((opt[cur].off==ZSTD_REP_MOVE_OPT) && (mlen != 1)) ? (opt[cur-mlen].rep[0] - 1) : (opt[cur-mlen].rep[opt[cur].off]);
|
||||
}
|
||||
|
||||
best_mlen = minMatch;
|
||||
{ U32 i, last_i = ZSTD_REP_CHECK + (mlen != 1);
|
||||
for (i=(opt[cur].mlen != 1); i<last_i; i++) { /* check rep */
|
||||
const S32 repCur = (i==ZSTD_REP_MOVE_OPT) ? (opt[cur].rep[0] - 1) : opt[cur].rep[i];
|
||||
if ( (repCur > 0) && (repCur < (S32)(inr-prefixStart))
|
||||
&& (ZSTD_readMINMATCH(inr, minMatch) == ZSTD_readMINMATCH(inr - repCur, minMatch))) {
|
||||
mlen = (U32)ZSTD_count(inr+minMatch, inr+minMatch - repCur, iend) + minMatch;
|
||||
|
||||
if (mlen > sufficient_len || cur + mlen >= ZSTD_OPT_NUM) {
|
||||
best_mlen = mlen; best_off = i; last_pos = cur + 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
best_off = i - (opt[cur].mlen != 1);
|
||||
if (mlen > best_mlen) best_mlen = mlen;
|
||||
|
||||
do {
|
||||
if (opt[cur].mlen == 1) {
|
||||
litlen = opt[cur].litlen;
|
||||
if (cur > litlen) {
|
||||
price = opt[cur - litlen].price + ZSTD_getPrice(optStatePtr, litlen, inr-litlen, best_off, mlen - MINMATCH, ultra);
|
||||
} else
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, best_off, mlen - MINMATCH, ultra);
|
||||
} else {
|
||||
litlen = 0;
|
||||
price = opt[cur].price + ZSTD_getPrice(optStatePtr, 0, NULL, best_off, mlen - MINMATCH, ultra);
|
||||
}
|
||||
|
||||
if (cur + mlen > last_pos || price <= opt[cur + mlen].price)
|
||||
SET_PRICE(cur + mlen, mlen, i, litlen, price);
|
||||
mlen--;
|
||||
} while (mlen >= minMatch);
|
||||
} } }
|
||||
|
||||
match_num = ZSTD_BtGetAllMatches_selectMLS(ctx, inr, iend, maxSearches, mls, matches, best_mlen);
|
||||
|
||||
if (match_num > 0 && (matches[match_num-1].len > sufficient_len || cur + matches[match_num-1].len >= ZSTD_OPT_NUM)) {
|
||||
best_mlen = matches[match_num-1].len;
|
||||
best_off = matches[match_num-1].off;
|
||||
last_pos = cur + 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
/* set prices using matches at position = cur */
|
||||
for (u = 0; u < match_num; u++) {
|
||||
mlen = (u>0) ? matches[u-1].len+1 : best_mlen;
|
||||
best_mlen = matches[u].len;
|
||||
|
||||
while (mlen <= best_mlen) {
|
||||
if (opt[cur].mlen == 1) {
|
||||
litlen = opt[cur].litlen;
|
||||
if (cur > litlen)
|
||||
price = opt[cur - litlen].price + ZSTD_getPrice(optStatePtr, litlen, ip+cur-litlen, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
else
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
} else {
|
||||
litlen = 0;
|
||||
price = opt[cur].price + ZSTD_getPrice(optStatePtr, 0, NULL, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
}
|
||||
|
||||
if (cur + mlen > last_pos || (price < opt[cur + mlen].price))
|
||||
SET_PRICE(cur + mlen, mlen, matches[u].off, litlen, price);
|
||||
|
||||
mlen++;
|
||||
} } }
|
||||
|
||||
best_mlen = opt[last_pos].mlen;
|
||||
best_off = opt[last_pos].off;
|
||||
cur = last_pos - best_mlen;
|
||||
|
||||
/* store sequence */
|
||||
_storeSequence: /* cur, last_pos, best_mlen, best_off have to be set */
|
||||
opt[0].mlen = 1;
|
||||
|
||||
while (1) {
|
||||
mlen = opt[cur].mlen;
|
||||
offset = opt[cur].off;
|
||||
opt[cur].mlen = best_mlen;
|
||||
opt[cur].off = best_off;
|
||||
best_mlen = mlen;
|
||||
best_off = offset;
|
||||
if (mlen > cur) break;
|
||||
cur -= mlen;
|
||||
}
|
||||
|
||||
for (u = 0; u <= last_pos;) {
|
||||
u += opt[u].mlen;
|
||||
}
|
||||
|
||||
for (cur=0; cur < last_pos; ) {
|
||||
mlen = opt[cur].mlen;
|
||||
if (mlen == 1) { ip++; cur++; continue; }
|
||||
offset = opt[cur].off;
|
||||
cur += mlen;
|
||||
litLength = (U32)(ip - anchor);
|
||||
|
||||
if (offset > ZSTD_REP_MOVE_OPT) {
|
||||
rep[2] = rep[1];
|
||||
rep[1] = rep[0];
|
||||
rep[0] = offset - ZSTD_REP_MOVE_OPT;
|
||||
offset--;
|
||||
} else {
|
||||
if (offset != 0) {
|
||||
best_off = (offset==ZSTD_REP_MOVE_OPT) ? (rep[0] - 1) : (rep[offset]);
|
||||
if (offset != 1) rep[2] = rep[1];
|
||||
rep[1] = rep[0];
|
||||
rep[0] = best_off;
|
||||
}
|
||||
if (litLength==0) offset--;
|
||||
}
|
||||
|
||||
ZSTD_updatePrice(optStatePtr, litLength, anchor, offset, mlen-MINMATCH);
|
||||
ZSTD_storeSeq(seqStorePtr, litLength, anchor, offset, mlen-MINMATCH);
|
||||
anchor = ip = ip + mlen;
|
||||
} } /* for (cur=0; cur < last_pos; ) */
|
||||
|
||||
/* Save reps for next block */
|
||||
{ int i; for (i=0; i<ZSTD_REP_NUM; i++) seqStorePtr->repToConfirm[i] = rep[i]; }
|
||||
|
||||
/* Last Literals */
|
||||
{ size_t const lastLLSize = iend - anchor;
|
||||
memcpy(seqStorePtr->lit, anchor, lastLLSize);
|
||||
seqStorePtr->lit += lastLLSize;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
FORCE_INLINE_TEMPLATE
|
||||
void ZSTD_compressBlock_opt_extDict_generic(ZSTD_CCtx* ctx,
|
||||
const void* src, size_t srcSize, const int ultra)
|
||||
{
|
||||
seqStore_t* seqStorePtr = &(ctx->seqStore);
|
||||
optState_t* optStatePtr = &(ctx->optState);
|
||||
const BYTE* const istart = (const BYTE*)src;
|
||||
const BYTE* ip = istart;
|
||||
const BYTE* anchor = istart;
|
||||
const BYTE* const iend = istart + srcSize;
|
||||
const BYTE* const ilimit = iend - 8;
|
||||
const BYTE* const base = ctx->base;
|
||||
const U32 lowestIndex = ctx->lowLimit;
|
||||
const U32 dictLimit = ctx->dictLimit;
|
||||
const BYTE* const prefixStart = base + dictLimit;
|
||||
const BYTE* const dictBase = ctx->dictBase;
|
||||
const BYTE* const dictEnd = dictBase + dictLimit;
|
||||
|
||||
const U32 maxSearches = 1U << ctx->appliedParams.cParams.searchLog;
|
||||
const U32 sufficient_len = ctx->appliedParams.cParams.targetLength;
|
||||
const U32 mls = ctx->appliedParams.cParams.searchLength;
|
||||
const U32 minMatch = (ctx->appliedParams.cParams.searchLength == 3) ? 3 : 4;
|
||||
|
||||
ZSTD_optimal_t* opt = optStatePtr->priceTable;
|
||||
ZSTD_match_t* matches = optStatePtr->matchTable;
|
||||
const BYTE* inr;
|
||||
|
||||
/* init */
|
||||
U32 offset, rep[ZSTD_REP_NUM];
|
||||
{ U32 i; for (i=0; i<ZSTD_REP_NUM; i++) rep[i]=seqStorePtr->rep[i]; }
|
||||
|
||||
ctx->nextToUpdate3 = ctx->nextToUpdate;
|
||||
ZSTD_rescaleFreqs(optStatePtr, (const BYTE*)src, srcSize);
|
||||
ip += (ip==prefixStart);
|
||||
|
||||
/* Match Loop */
|
||||
while (ip < ilimit) {
|
||||
U32 cur, match_num, last_pos, litlen, price;
|
||||
U32 u, mlen, best_mlen, best_off, litLength;
|
||||
U32 current = (U32)(ip-base);
|
||||
memset(opt, 0, sizeof(ZSTD_optimal_t));
|
||||
last_pos = 0;
|
||||
opt[0].litlen = (U32)(ip - anchor);
|
||||
|
||||
/* check repCode */
|
||||
{ U32 i, last_i = ZSTD_REP_CHECK + (ip==anchor);
|
||||
for (i = (ip==anchor); i<last_i; i++) {
|
||||
const S32 repCur = (i==ZSTD_REP_MOVE_OPT) ? (rep[0] - 1) : rep[i];
|
||||
const U32 repIndex = (U32)(current - repCur);
|
||||
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* const repMatch = repBase + repIndex;
|
||||
if ( (repCur > 0 && repCur <= (S32)current)
|
||||
&& (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex>lowestIndex)) /* intentional overflow */
|
||||
&& (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch)) ) {
|
||||
/* repcode detected we should take it */
|
||||
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
|
||||
mlen = (U32)ZSTD_count_2segments(ip+minMatch, repMatch+minMatch, iend, repEnd, prefixStart) + minMatch;
|
||||
|
||||
if (mlen > sufficient_len || mlen >= ZSTD_OPT_NUM) {
|
||||
best_mlen = mlen; best_off = i; cur = 0; last_pos = 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
best_off = i - (ip==anchor);
|
||||
litlen = opt[0].litlen;
|
||||
do {
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, best_off, mlen - MINMATCH, ultra);
|
||||
if (mlen > last_pos || price < opt[mlen].price)
|
||||
SET_PRICE(mlen, mlen, i, litlen, price); /* note : macro modifies last_pos */
|
||||
mlen--;
|
||||
} while (mlen >= minMatch);
|
||||
} } }
|
||||
|
||||
match_num = ZSTD_BtGetAllMatches_selectMLS_extDict(ctx, ip, iend, maxSearches, mls, matches, minMatch); /* first search (depth 0) */
|
||||
|
||||
if (!last_pos && !match_num) { ip++; continue; }
|
||||
|
||||
{ U32 i; for (i=0; i<ZSTD_REP_NUM; i++) opt[0].rep[i] = rep[i]; }
|
||||
opt[0].mlen = 1;
|
||||
|
||||
if (match_num && (matches[match_num-1].len > sufficient_len || matches[match_num-1].len >= ZSTD_OPT_NUM)) {
|
||||
best_mlen = matches[match_num-1].len;
|
||||
best_off = matches[match_num-1].off;
|
||||
cur = 0;
|
||||
last_pos = 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
best_mlen = (last_pos) ? last_pos : minMatch;
|
||||
|
||||
/* set prices using matches at position = 0 */
|
||||
for (u = 0; u < match_num; u++) {
|
||||
mlen = (u>0) ? matches[u-1].len+1 : best_mlen;
|
||||
best_mlen = matches[u].len;
|
||||
litlen = opt[0].litlen;
|
||||
while (mlen <= best_mlen) {
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
if (mlen > last_pos || price < opt[mlen].price)
|
||||
SET_PRICE(mlen, mlen, matches[u].off, litlen, price);
|
||||
mlen++;
|
||||
} }
|
||||
|
||||
if (last_pos < minMatch) {
|
||||
ip++; continue;
|
||||
}
|
||||
|
||||
/* check further positions */
|
||||
for (cur = 1; cur <= last_pos; cur++) {
|
||||
inr = ip + cur;
|
||||
|
||||
if (opt[cur-1].mlen == 1) {
|
||||
litlen = opt[cur-1].litlen + 1;
|
||||
if (cur > litlen) {
|
||||
price = opt[cur - litlen].price + ZSTD_getLiteralPrice(optStatePtr, litlen, inr-litlen);
|
||||
} else
|
||||
price = ZSTD_getLiteralPrice(optStatePtr, litlen, anchor);
|
||||
} else {
|
||||
litlen = 1;
|
||||
price = opt[cur - 1].price + ZSTD_getLiteralPrice(optStatePtr, litlen, inr-1);
|
||||
}
|
||||
|
||||
if (cur > last_pos || price <= opt[cur].price)
|
||||
SET_PRICE(cur, 1, 0, litlen, price);
|
||||
|
||||
if (cur == last_pos) break;
|
||||
|
||||
if (inr > ilimit) /* last match must start at a minimum distance of 8 from oend */
|
||||
continue;
|
||||
|
||||
mlen = opt[cur].mlen;
|
||||
if (opt[cur].off > ZSTD_REP_MOVE_OPT) {
|
||||
opt[cur].rep[2] = opt[cur-mlen].rep[1];
|
||||
opt[cur].rep[1] = opt[cur-mlen].rep[0];
|
||||
opt[cur].rep[0] = opt[cur].off - ZSTD_REP_MOVE_OPT;
|
||||
} else {
|
||||
opt[cur].rep[2] = (opt[cur].off > 1) ? opt[cur-mlen].rep[1] : opt[cur-mlen].rep[2];
|
||||
opt[cur].rep[1] = (opt[cur].off > 0) ? opt[cur-mlen].rep[0] : opt[cur-mlen].rep[1];
|
||||
opt[cur].rep[0] = ((opt[cur].off==ZSTD_REP_MOVE_OPT) && (mlen != 1)) ? (opt[cur-mlen].rep[0] - 1) : (opt[cur-mlen].rep[opt[cur].off]);
|
||||
}
|
||||
|
||||
best_mlen = minMatch;
|
||||
{ U32 i, last_i = ZSTD_REP_CHECK + (mlen != 1);
|
||||
for (i = (mlen != 1); i<last_i; i++) {
|
||||
const S32 repCur = (i==ZSTD_REP_MOVE_OPT) ? (opt[cur].rep[0] - 1) : opt[cur].rep[i];
|
||||
const U32 repIndex = (U32)(current+cur - repCur);
|
||||
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
|
||||
const BYTE* const repMatch = repBase + repIndex;
|
||||
if ( (repCur > 0 && repCur <= (S32)(current+cur))
|
||||
&& (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex>lowestIndex)) /* intentional overflow */
|
||||
&& (ZSTD_readMINMATCH(inr, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch)) ) {
|
||||
/* repcode detected */
|
||||
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
|
||||
mlen = (U32)ZSTD_count_2segments(inr+minMatch, repMatch+minMatch, iend, repEnd, prefixStart) + minMatch;
|
||||
|
||||
if (mlen > sufficient_len || cur + mlen >= ZSTD_OPT_NUM) {
|
||||
best_mlen = mlen; best_off = i; last_pos = cur + 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
best_off = i - (opt[cur].mlen != 1);
|
||||
if (mlen > best_mlen) best_mlen = mlen;
|
||||
|
||||
do {
|
||||
if (opt[cur].mlen == 1) {
|
||||
litlen = opt[cur].litlen;
|
||||
if (cur > litlen) {
|
||||
price = opt[cur - litlen].price + ZSTD_getPrice(optStatePtr, litlen, inr-litlen, best_off, mlen - MINMATCH, ultra);
|
||||
} else
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, best_off, mlen - MINMATCH, ultra);
|
||||
} else {
|
||||
litlen = 0;
|
||||
price = opt[cur].price + ZSTD_getPrice(optStatePtr, 0, NULL, best_off, mlen - MINMATCH, ultra);
|
||||
}
|
||||
|
||||
if (cur + mlen > last_pos || price <= opt[cur + mlen].price)
|
||||
SET_PRICE(cur + mlen, mlen, i, litlen, price);
|
||||
mlen--;
|
||||
} while (mlen >= minMatch);
|
||||
} } }
|
||||
|
||||
match_num = ZSTD_BtGetAllMatches_selectMLS_extDict(ctx, inr, iend, maxSearches, mls, matches, minMatch);
|
||||
|
||||
if (match_num > 0 && (matches[match_num-1].len > sufficient_len || cur + matches[match_num-1].len >= ZSTD_OPT_NUM)) {
|
||||
best_mlen = matches[match_num-1].len;
|
||||
best_off = matches[match_num-1].off;
|
||||
last_pos = cur + 1;
|
||||
goto _storeSequence;
|
||||
}
|
||||
|
||||
/* set prices using matches at position = cur */
|
||||
for (u = 0; u < match_num; u++) {
|
||||
mlen = (u>0) ? matches[u-1].len+1 : best_mlen;
|
||||
best_mlen = matches[u].len;
|
||||
|
||||
while (mlen <= best_mlen) {
|
||||
if (opt[cur].mlen == 1) {
|
||||
litlen = opt[cur].litlen;
|
||||
if (cur > litlen)
|
||||
price = opt[cur - litlen].price + ZSTD_getPrice(optStatePtr, litlen, ip+cur-litlen, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
else
|
||||
price = ZSTD_getPrice(optStatePtr, litlen, anchor, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
} else {
|
||||
litlen = 0;
|
||||
price = opt[cur].price + ZSTD_getPrice(optStatePtr, 0, NULL, matches[u].off-1, mlen - MINMATCH, ultra);
|
||||
}
|
||||
|
||||
if (cur + mlen > last_pos || (price < opt[cur + mlen].price))
|
||||
SET_PRICE(cur + mlen, mlen, matches[u].off, litlen, price);
|
||||
|
||||
mlen++;
|
||||
} } } /* for (cur = 1; cur <= last_pos; cur++) */
|
||||
|
||||
best_mlen = opt[last_pos].mlen;
|
||||
best_off = opt[last_pos].off;
|
||||
cur = last_pos - best_mlen;
|
||||
|
||||
/* store sequence */
|
||||
_storeSequence: /* cur, last_pos, best_mlen, best_off have to be set */
|
||||
opt[0].mlen = 1;
|
||||
|
||||
while (1) {
|
||||
mlen = opt[cur].mlen;
|
||||
offset = opt[cur].off;
|
||||
opt[cur].mlen = best_mlen;
|
||||
opt[cur].off = best_off;
|
||||
best_mlen = mlen;
|
||||
best_off = offset;
|
||||
if (mlen > cur) break;
|
||||
cur -= mlen;
|
||||
}
|
||||
|
||||
for (u = 0; u <= last_pos; ) {
|
||||
u += opt[u].mlen;
|
||||
}
|
||||
|
||||
for (cur=0; cur < last_pos; ) {
|
||||
mlen = opt[cur].mlen;
|
||||
if (mlen == 1) { ip++; cur++; continue; }
|
||||
offset = opt[cur].off;
|
||||
cur += mlen;
|
||||
litLength = (U32)(ip - anchor);
|
||||
|
||||
if (offset > ZSTD_REP_MOVE_OPT) {
|
||||
rep[2] = rep[1];
|
||||
rep[1] = rep[0];
|
||||
rep[0] = offset - ZSTD_REP_MOVE_OPT;
|
||||
offset--;
|
||||
} else {
|
||||
if (offset != 0) {
|
||||
best_off = (offset==ZSTD_REP_MOVE_OPT) ? (rep[0] - 1) : (rep[offset]);
|
||||
if (offset != 1) rep[2] = rep[1];
|
||||
rep[1] = rep[0];
|
||||
rep[0] = best_off;
|
||||
}
|
||||
|
||||
if (litLength==0) offset--;
|
||||
}
|
||||
|
||||
ZSTD_updatePrice(optStatePtr, litLength, anchor, offset, mlen-MINMATCH);
|
||||
ZSTD_storeSeq(seqStorePtr, litLength, anchor, offset, mlen-MINMATCH);
|
||||
anchor = ip = ip + mlen;
|
||||
} } /* for (cur=0; cur < last_pos; ) */
|
||||
|
||||
/* Save reps for next block */
|
||||
{ int i; for (i=0; i<ZSTD_REP_NUM; i++) seqStorePtr->repToConfirm[i] = rep[i]; }
|
||||
|
||||
/* Last Literals */
|
||||
{ size_t lastLLSize = iend - anchor;
|
||||
memcpy(seqStorePtr->lit, anchor, lastLLSize);
|
||||
seqStorePtr->lit += lastLLSize;
|
||||
}
|
||||
}
|
||||
|
||||
#endif /* ZSTD_OPT_H_91842398743 */
|
||||
#endif /* ZSTD_OPT_H */
|
||||
|
@ -5,11 +5,12 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
|
||||
/* ====== Tuning parameters ====== */
|
||||
#define ZSTDMT_NBTHREADS_MAX 256
|
||||
#define ZSTDMT_NBTHREADS_MAX 200
|
||||
#define ZSTDMT_OVERLAPLOG_DEFAULT 6
|
||||
|
||||
|
||||
@ -52,22 +53,24 @@ static unsigned long long GetCurrentClockTimeMicroseconds(void)
|
||||
}
|
||||
|
||||
#define MUTEX_WAIT_TIME_DLEVEL 6
|
||||
#define PTHREAD_MUTEX_LOCK(mutex) { \
|
||||
if (ZSTD_DEBUG>=MUTEX_WAIT_TIME_DLEVEL) { \
|
||||
#define ZSTD_PTHREAD_MUTEX_LOCK(mutex) { \
|
||||
if (ZSTD_DEBUG >= MUTEX_WAIT_TIME_DLEVEL) { \
|
||||
unsigned long long const beforeTime = GetCurrentClockTimeMicroseconds(); \
|
||||
pthread_mutex_lock(mutex); \
|
||||
ZSTD_pthread_mutex_lock(mutex); \
|
||||
{ unsigned long long const afterTime = GetCurrentClockTimeMicroseconds(); \
|
||||
unsigned long long const elapsedTime = (afterTime-beforeTime); \
|
||||
if (elapsedTime > 1000) { /* or whatever threshold you like; I'm using 1 millisecond here */ \
|
||||
DEBUGLOG(MUTEX_WAIT_TIME_DLEVEL, "Thread took %llu microseconds to acquire mutex %s \n", \
|
||||
elapsedTime, #mutex); \
|
||||
} } \
|
||||
} else pthread_mutex_lock(mutex); \
|
||||
} else { \
|
||||
ZSTD_pthread_mutex_lock(mutex); \
|
||||
} \
|
||||
}
|
||||
|
||||
#else
|
||||
|
||||
# define PTHREAD_MUTEX_LOCK(m) pthread_mutex_lock(m)
|
||||
# define ZSTD_PTHREAD_MUTEX_LOCK(m) ZSTD_pthread_mutex_lock(m)
|
||||
# define DEBUG_PRINTHEX(l,p,n) {}
|
||||
|
||||
#endif
|
||||
@ -84,7 +87,7 @@ typedef struct buffer_s {
|
||||
static const buffer_t g_nullBuffer = { NULL, 0 };
|
||||
|
||||
typedef struct ZSTDMT_bufferPool_s {
|
||||
pthread_mutex_t poolMutex;
|
||||
ZSTD_pthread_mutex_t poolMutex;
|
||||
size_t bufferSize;
|
||||
unsigned totalBuffers;
|
||||
unsigned nbBuffers;
|
||||
@ -98,7 +101,7 @@ static ZSTDMT_bufferPool* ZSTDMT_createBufferPool(unsigned nbThreads, ZSTD_custo
|
||||
ZSTDMT_bufferPool* const bufPool = (ZSTDMT_bufferPool*)ZSTD_calloc(
|
||||
sizeof(ZSTDMT_bufferPool) + (maxNbBuffers-1) * sizeof(buffer_t), cMem);
|
||||
if (bufPool==NULL) return NULL;
|
||||
if (pthread_mutex_init(&bufPool->poolMutex, NULL)) {
|
||||
if (ZSTD_pthread_mutex_init(&bufPool->poolMutex, NULL)) {
|
||||
ZSTD_free(bufPool, cMem);
|
||||
return NULL;
|
||||
}
|
||||
@ -112,10 +115,13 @@ static ZSTDMT_bufferPool* ZSTDMT_createBufferPool(unsigned nbThreads, ZSTD_custo
|
||||
static void ZSTDMT_freeBufferPool(ZSTDMT_bufferPool* bufPool)
|
||||
{
|
||||
unsigned u;
|
||||
DEBUGLOG(3, "ZSTDMT_freeBufferPool (address:%08X)", (U32)(size_t)bufPool);
|
||||
if (!bufPool) return; /* compatibility with free on NULL */
|
||||
for (u=0; u<bufPool->totalBuffers; u++)
|
||||
for (u=0; u<bufPool->totalBuffers; u++) {
|
||||
DEBUGLOG(4, "free buffer %2u (address:%08X)", u, (U32)(size_t)bufPool->bTable[u].start);
|
||||
ZSTD_free(bufPool->bTable[u].start, bufPool->cMem);
|
||||
pthread_mutex_destroy(&bufPool->poolMutex);
|
||||
}
|
||||
ZSTD_pthread_mutex_destroy(&bufPool->poolMutex);
|
||||
ZSTD_free(bufPool, bufPool->cMem);
|
||||
}
|
||||
|
||||
@ -126,10 +132,10 @@ static size_t ZSTDMT_sizeof_bufferPool(ZSTDMT_bufferPool* bufPool)
|
||||
+ (bufPool->totalBuffers - 1) * sizeof(buffer_t);
|
||||
unsigned u;
|
||||
size_t totalBufferSize = 0;
|
||||
pthread_mutex_lock(&bufPool->poolMutex);
|
||||
ZSTD_pthread_mutex_lock(&bufPool->poolMutex);
|
||||
for (u=0; u<bufPool->totalBuffers; u++)
|
||||
totalBufferSize += bufPool->bTable[u].size;
|
||||
pthread_mutex_unlock(&bufPool->poolMutex);
|
||||
ZSTD_pthread_mutex_unlock(&bufPool->poolMutex);
|
||||
|
||||
return poolSize + totalBufferSize;
|
||||
}
|
||||
@ -145,20 +151,21 @@ static buffer_t ZSTDMT_getBuffer(ZSTDMT_bufferPool* bufPool)
|
||||
{
|
||||
size_t const bSize = bufPool->bufferSize;
|
||||
DEBUGLOG(5, "ZSTDMT_getBuffer");
|
||||
pthread_mutex_lock(&bufPool->poolMutex);
|
||||
ZSTD_pthread_mutex_lock(&bufPool->poolMutex);
|
||||
if (bufPool->nbBuffers) { /* try to use an existing buffer */
|
||||
buffer_t const buf = bufPool->bTable[--(bufPool->nbBuffers)];
|
||||
size_t const availBufferSize = buf.size;
|
||||
bufPool->bTable[bufPool->nbBuffers] = g_nullBuffer;
|
||||
if ((availBufferSize >= bSize) & (availBufferSize <= 10*bSize)) {
|
||||
/* large enough, but not too much */
|
||||
pthread_mutex_unlock(&bufPool->poolMutex);
|
||||
ZSTD_pthread_mutex_unlock(&bufPool->poolMutex);
|
||||
return buf;
|
||||
}
|
||||
/* size conditions not respected : scratch this buffer, create new one */
|
||||
DEBUGLOG(5, "existing buffer does not meet size conditions => freeing");
|
||||
ZSTD_free(buf.start, bufPool->cMem);
|
||||
}
|
||||
pthread_mutex_unlock(&bufPool->poolMutex);
|
||||
ZSTD_pthread_mutex_unlock(&bufPool->poolMutex);
|
||||
/* create new buffer */
|
||||
DEBUGLOG(5, "create a new buffer");
|
||||
{ buffer_t buffer;
|
||||
@ -174,24 +181,38 @@ static void ZSTDMT_releaseBuffer(ZSTDMT_bufferPool* bufPool, buffer_t buf)
|
||||
{
|
||||
if (buf.start == NULL) return; /* compatible with release on NULL */
|
||||
DEBUGLOG(5, "ZSTDMT_releaseBuffer");
|
||||
pthread_mutex_lock(&bufPool->poolMutex);
|
||||
ZSTD_pthread_mutex_lock(&bufPool->poolMutex);
|
||||
if (bufPool->nbBuffers < bufPool->totalBuffers) {
|
||||
bufPool->bTable[bufPool->nbBuffers++] = buf; /* stored for later use */
|
||||
pthread_mutex_unlock(&bufPool->poolMutex);
|
||||
ZSTD_pthread_mutex_unlock(&bufPool->poolMutex);
|
||||
return;
|
||||
}
|
||||
pthread_mutex_unlock(&bufPool->poolMutex);
|
||||
ZSTD_pthread_mutex_unlock(&bufPool->poolMutex);
|
||||
/* Reached bufferPool capacity (should not happen) */
|
||||
DEBUGLOG(5, "buffer pool capacity reached => freeing ");
|
||||
ZSTD_free(buf.start, bufPool->cMem);
|
||||
}
|
||||
|
||||
/* Sets parameters relevant to the compression job, initializing others to
|
||||
* default values. Notably, nbThreads should probably be zero. */
|
||||
static ZSTD_CCtx_params ZSTDMT_makeJobCCtxParams(ZSTD_CCtx_params const params)
|
||||
{
|
||||
ZSTD_CCtx_params jobParams;
|
||||
memset(&jobParams, 0, sizeof(jobParams));
|
||||
|
||||
jobParams.cParams = params.cParams;
|
||||
jobParams.fParams = params.fParams;
|
||||
jobParams.compressionLevel = params.compressionLevel;
|
||||
|
||||
jobParams.ldmParams = params.ldmParams;
|
||||
return jobParams;
|
||||
}
|
||||
|
||||
/* ===== CCtx Pool ===== */
|
||||
/* a single CCtx Pool can be invoked from multiple threads in parallel */
|
||||
|
||||
typedef struct {
|
||||
pthread_mutex_t poolMutex;
|
||||
ZSTD_pthread_mutex_t poolMutex;
|
||||
unsigned totalCCtx;
|
||||
unsigned availCCtx;
|
||||
ZSTD_customMem cMem;
|
||||
@ -204,7 +225,7 @@ static void ZSTDMT_freeCCtxPool(ZSTDMT_CCtxPool* pool)
|
||||
unsigned u;
|
||||
for (u=0; u<pool->totalCCtx; u++)
|
||||
ZSTD_freeCCtx(pool->cctx[u]); /* note : compatible with free on NULL */
|
||||
pthread_mutex_destroy(&pool->poolMutex);
|
||||
ZSTD_pthread_mutex_destroy(&pool->poolMutex);
|
||||
ZSTD_free(pool, pool->cMem);
|
||||
}
|
||||
|
||||
@ -216,7 +237,7 @@ static ZSTDMT_CCtxPool* ZSTDMT_createCCtxPool(unsigned nbThreads,
|
||||
ZSTDMT_CCtxPool* const cctxPool = (ZSTDMT_CCtxPool*) ZSTD_calloc(
|
||||
sizeof(ZSTDMT_CCtxPool) + (nbThreads-1)*sizeof(ZSTD_CCtx*), cMem);
|
||||
if (!cctxPool) return NULL;
|
||||
if (pthread_mutex_init(&cctxPool->poolMutex, NULL)) {
|
||||
if (ZSTD_pthread_mutex_init(&cctxPool->poolMutex, NULL)) {
|
||||
ZSTD_free(cctxPool, cMem);
|
||||
return NULL;
|
||||
}
|
||||
@ -232,7 +253,7 @@ static ZSTDMT_CCtxPool* ZSTDMT_createCCtxPool(unsigned nbThreads,
|
||||
/* only works during initialization phase, not during compression */
|
||||
static size_t ZSTDMT_sizeof_CCtxPool(ZSTDMT_CCtxPool* cctxPool)
|
||||
{
|
||||
pthread_mutex_lock(&cctxPool->poolMutex);
|
||||
ZSTD_pthread_mutex_lock(&cctxPool->poolMutex);
|
||||
{ unsigned const nbThreads = cctxPool->totalCCtx;
|
||||
size_t const poolSize = sizeof(*cctxPool)
|
||||
+ (nbThreads-1)*sizeof(ZSTD_CCtx*);
|
||||
@ -241,7 +262,7 @@ static size_t ZSTDMT_sizeof_CCtxPool(ZSTDMT_CCtxPool* cctxPool)
|
||||
for (u=0; u<nbThreads; u++) {
|
||||
totalCCtxSize += ZSTD_sizeof_CCtx(cctxPool->cctx[u]);
|
||||
}
|
||||
pthread_mutex_unlock(&cctxPool->poolMutex);
|
||||
ZSTD_pthread_mutex_unlock(&cctxPool->poolMutex);
|
||||
return poolSize + totalCCtxSize;
|
||||
}
|
||||
}
|
||||
@ -249,14 +270,14 @@ static size_t ZSTDMT_sizeof_CCtxPool(ZSTDMT_CCtxPool* cctxPool)
|
||||
static ZSTD_CCtx* ZSTDMT_getCCtx(ZSTDMT_CCtxPool* cctxPool)
|
||||
{
|
||||
DEBUGLOG(5, "ZSTDMT_getCCtx");
|
||||
pthread_mutex_lock(&cctxPool->poolMutex);
|
||||
ZSTD_pthread_mutex_lock(&cctxPool->poolMutex);
|
||||
if (cctxPool->availCCtx) {
|
||||
cctxPool->availCCtx--;
|
||||
{ ZSTD_CCtx* const cctx = cctxPool->cctx[cctxPool->availCCtx];
|
||||
pthread_mutex_unlock(&cctxPool->poolMutex);
|
||||
ZSTD_pthread_mutex_unlock(&cctxPool->poolMutex);
|
||||
return cctx;
|
||||
} }
|
||||
pthread_mutex_unlock(&cctxPool->poolMutex);
|
||||
ZSTD_pthread_mutex_unlock(&cctxPool->poolMutex);
|
||||
DEBUGLOG(5, "create one more CCtx");
|
||||
return ZSTD_createCCtx_advanced(cctxPool->cMem); /* note : can be NULL, when creation fails ! */
|
||||
}
|
||||
@ -264,7 +285,7 @@ static ZSTD_CCtx* ZSTDMT_getCCtx(ZSTDMT_CCtxPool* cctxPool)
|
||||
static void ZSTDMT_releaseCCtx(ZSTDMT_CCtxPool* pool, ZSTD_CCtx* cctx)
|
||||
{
|
||||
if (cctx==NULL) return; /* compatibility with release on NULL */
|
||||
pthread_mutex_lock(&pool->poolMutex);
|
||||
ZSTD_pthread_mutex_lock(&pool->poolMutex);
|
||||
if (pool->availCCtx < pool->totalCCtx)
|
||||
pool->cctx[pool->availCCtx++] = cctx;
|
||||
else {
|
||||
@ -272,7 +293,7 @@ static void ZSTDMT_releaseCCtx(ZSTDMT_CCtxPool* pool, ZSTD_CCtx* cctx)
|
||||
DEBUGLOG(5, "CCtx pool overflow : free cctx");
|
||||
ZSTD_freeCCtx(cctx);
|
||||
}
|
||||
pthread_mutex_unlock(&pool->poolMutex);
|
||||
ZSTD_pthread_mutex_unlock(&pool->poolMutex);
|
||||
}
|
||||
|
||||
|
||||
@ -290,9 +311,9 @@ typedef struct {
|
||||
unsigned lastChunk;
|
||||
unsigned jobCompleted;
|
||||
unsigned jobScanned;
|
||||
pthread_mutex_t* jobCompleted_mutex;
|
||||
pthread_cond_t* jobCompleted_cond;
|
||||
ZSTD_parameters params;
|
||||
ZSTD_pthread_mutex_t* jobCompleted_mutex;
|
||||
ZSTD_pthread_cond_t* jobCompleted_cond;
|
||||
ZSTD_CCtx_params params;
|
||||
const ZSTD_CDict* cdict;
|
||||
ZSTDMT_CCtxPool* cctxPool;
|
||||
ZSTDMT_bufferPool* bufPool;
|
||||
@ -329,10 +350,15 @@ void ZSTDMT_compressChunk(void* jobDescription)
|
||||
if (ZSTD_isError(initError)) { job->cSize = initError; goto _endJob; }
|
||||
} else { /* srcStart points at reloaded section */
|
||||
if (!job->firstChunk) job->params.fParams.contentSizeFlag = 0; /* ensure no srcSize control */
|
||||
{ size_t const dictModeError = ZSTD_setCCtxParameter(cctx, ZSTD_p_forceRawDict, 1); /* Force loading dictionary in "content-only" mode (no header analysis) */
|
||||
size_t const initError = ZSTD_compressBegin_advanced(cctx, job->srcStart, job->dictSize, job->params, job->fullFrameSize);
|
||||
if (ZSTD_isError(initError) || ZSTD_isError(dictModeError)) { job->cSize = initError; goto _endJob; }
|
||||
ZSTD_setCCtxParameter(cctx, ZSTD_p_forceWindow, 1);
|
||||
{ ZSTD_CCtx_params jobParams = job->params;
|
||||
size_t const forceWindowError =
|
||||
ZSTD_CCtxParam_setParameter(&jobParams, ZSTD_p_forceMaxWindow, !job->firstChunk);
|
||||
/* Force loading dictionary in "content-only" mode (no header analysis) */
|
||||
size_t const initError = ZSTD_compressBegin_advanced_internal(cctx, job->srcStart, job->dictSize, ZSTD_dm_rawContent, jobParams, job->fullFrameSize);
|
||||
if (ZSTD_isError(initError) || ZSTD_isError(forceWindowError)) {
|
||||
job->cSize = initError;
|
||||
goto _endJob;
|
||||
}
|
||||
} }
|
||||
if (!job->firstChunk) { /* flush and overwrite frame header when it's not first segment */
|
||||
size_t const hSize = ZSTD_compressContinue(cctx, dstBuff.start, dstBuff.size, src, 0);
|
||||
@ -353,11 +379,11 @@ void ZSTDMT_compressChunk(void* jobDescription)
|
||||
ZSTDMT_releaseCCtx(job->cctxPool, cctx);
|
||||
ZSTDMT_releaseBuffer(job->bufPool, job->src);
|
||||
job->src = g_nullBuffer; job->srcStart = NULL;
|
||||
PTHREAD_MUTEX_LOCK(job->jobCompleted_mutex);
|
||||
ZSTD_PTHREAD_MUTEX_LOCK(job->jobCompleted_mutex);
|
||||
job->jobCompleted = 1;
|
||||
job->jobScanned = 0;
|
||||
pthread_cond_signal(job->jobCompleted_cond);
|
||||
pthread_mutex_unlock(job->jobCompleted_mutex);
|
||||
ZSTD_pthread_cond_signal(job->jobCompleted_cond);
|
||||
ZSTD_pthread_mutex_unlock(job->jobCompleted_mutex);
|
||||
}
|
||||
|
||||
|
||||
@ -375,24 +401,21 @@ struct ZSTDMT_CCtx_s {
|
||||
ZSTDMT_jobDescription* jobs;
|
||||
ZSTDMT_bufferPool* bufPool;
|
||||
ZSTDMT_CCtxPool* cctxPool;
|
||||
pthread_mutex_t jobCompleted_mutex;
|
||||
pthread_cond_t jobCompleted_cond;
|
||||
ZSTD_pthread_mutex_t jobCompleted_mutex;
|
||||
ZSTD_pthread_cond_t jobCompleted_cond;
|
||||
size_t targetSectionSize;
|
||||
size_t inBuffSize;
|
||||
size_t dictSize;
|
||||
size_t targetDictSize;
|
||||
inBuff_t inBuff;
|
||||
ZSTD_parameters params;
|
||||
ZSTD_CCtx_params params;
|
||||
XXH64_state_t xxhState;
|
||||
unsigned nbThreads;
|
||||
unsigned jobIDMask;
|
||||
unsigned doneJobID;
|
||||
unsigned nextJobID;
|
||||
unsigned frameEnded;
|
||||
unsigned allJobsCompleted;
|
||||
unsigned overlapLog;
|
||||
unsigned long long frameContentSize;
|
||||
size_t sectionSize;
|
||||
ZSTD_customMem cMem;
|
||||
ZSTD_CDict* cdictLocal;
|
||||
const ZSTD_CDict* cdict;
|
||||
@ -407,6 +430,15 @@ static ZSTDMT_jobDescription* ZSTDMT_allocJobsTable(U32* nbJobsPtr, ZSTD_customM
|
||||
nbJobs * sizeof(ZSTDMT_jobDescription), cMem);
|
||||
}
|
||||
|
||||
/* Internal only */
|
||||
size_t ZSTDMT_initializeCCtxParameters(ZSTD_CCtx_params* params, unsigned nbThreads)
|
||||
{
|
||||
params->nbThreads = nbThreads;
|
||||
params->overlapSizeLog = ZSTDMT_OVERLAPLOG_DEFAULT;
|
||||
params->jobSize = 0;
|
||||
return 0;
|
||||
}
|
||||
|
||||
ZSTDMT_CCtx* ZSTDMT_createCCtx_advanced(unsigned nbThreads, ZSTD_customMem cMem)
|
||||
{
|
||||
ZSTDMT_CCtx* mtctx;
|
||||
@ -421,12 +453,10 @@ ZSTDMT_CCtx* ZSTDMT_createCCtx_advanced(unsigned nbThreads, ZSTD_customMem cMem)
|
||||
|
||||
mtctx = (ZSTDMT_CCtx*) ZSTD_calloc(sizeof(ZSTDMT_CCtx), cMem);
|
||||
if (!mtctx) return NULL;
|
||||
ZSTDMT_initializeCCtxParameters(&mtctx->params, nbThreads);
|
||||
mtctx->cMem = cMem;
|
||||
mtctx->nbThreads = nbThreads;
|
||||
mtctx->allJobsCompleted = 1;
|
||||
mtctx->sectionSize = 0;
|
||||
mtctx->overlapLog = ZSTDMT_OVERLAPLOG_DEFAULT;
|
||||
mtctx->factory = POOL_create(nbThreads, 0);
|
||||
mtctx->factory = POOL_create_advanced(nbThreads, 0, cMem);
|
||||
mtctx->jobs = ZSTDMT_allocJobsTable(&nbJobs, cMem);
|
||||
mtctx->jobIDMask = nbJobs - 1;
|
||||
mtctx->bufPool = ZSTDMT_createBufferPool(nbThreads, cMem);
|
||||
@ -435,11 +465,11 @@ ZSTDMT_CCtx* ZSTDMT_createCCtx_advanced(unsigned nbThreads, ZSTD_customMem cMem)
|
||||
ZSTDMT_freeCCtx(mtctx);
|
||||
return NULL;
|
||||
}
|
||||
if (pthread_mutex_init(&mtctx->jobCompleted_mutex, NULL)) {
|
||||
if (ZSTD_pthread_mutex_init(&mtctx->jobCompleted_mutex, NULL)) {
|
||||
ZSTDMT_freeCCtx(mtctx);
|
||||
return NULL;
|
||||
}
|
||||
if (pthread_cond_init(&mtctx->jobCompleted_cond, NULL)) {
|
||||
if (ZSTD_pthread_cond_init(&mtctx->jobCompleted_cond, NULL)) {
|
||||
ZSTDMT_freeCCtx(mtctx);
|
||||
return NULL;
|
||||
}
|
||||
@ -459,28 +489,46 @@ static void ZSTDMT_releaseAllJobResources(ZSTDMT_CCtx* mtctx)
|
||||
unsigned jobID;
|
||||
DEBUGLOG(3, "ZSTDMT_releaseAllJobResources");
|
||||
for (jobID=0; jobID <= mtctx->jobIDMask; jobID++) {
|
||||
DEBUGLOG(4, "job%02u: release dst address %08X", jobID, (U32)(size_t)mtctx->jobs[jobID].dstBuff.start);
|
||||
ZSTDMT_releaseBuffer(mtctx->bufPool, mtctx->jobs[jobID].dstBuff);
|
||||
mtctx->jobs[jobID].dstBuff = g_nullBuffer;
|
||||
DEBUGLOG(4, "job%02u: release src address %08X", jobID, (U32)(size_t)mtctx->jobs[jobID].src.start);
|
||||
ZSTDMT_releaseBuffer(mtctx->bufPool, mtctx->jobs[jobID].src);
|
||||
mtctx->jobs[jobID].src = g_nullBuffer;
|
||||
}
|
||||
memset(mtctx->jobs, 0, (mtctx->jobIDMask+1)*sizeof(ZSTDMT_jobDescription));
|
||||
DEBUGLOG(4, "input: release address %08X", (U32)(size_t)mtctx->inBuff.buffer.start);
|
||||
ZSTDMT_releaseBuffer(mtctx->bufPool, mtctx->inBuff.buffer);
|
||||
mtctx->inBuff.buffer = g_nullBuffer;
|
||||
mtctx->allJobsCompleted = 1;
|
||||
}
|
||||
|
||||
static void ZSTDMT_waitForAllJobsCompleted(ZSTDMT_CCtx* zcs)
|
||||
{
|
||||
DEBUGLOG(4, "ZSTDMT_waitForAllJobsCompleted");
|
||||
while (zcs->doneJobID < zcs->nextJobID) {
|
||||
unsigned const jobID = zcs->doneJobID & zcs->jobIDMask;
|
||||
ZSTD_PTHREAD_MUTEX_LOCK(&zcs->jobCompleted_mutex);
|
||||
while (zcs->jobs[jobID].jobCompleted==0) {
|
||||
DEBUGLOG(5, "waiting for jobCompleted signal from chunk %u", zcs->doneJobID); /* we want to block when waiting for data to flush */
|
||||
ZSTD_pthread_cond_wait(&zcs->jobCompleted_cond, &zcs->jobCompleted_mutex);
|
||||
}
|
||||
ZSTD_pthread_mutex_unlock(&zcs->jobCompleted_mutex);
|
||||
zcs->doneJobID++;
|
||||
}
|
||||
}
|
||||
|
||||
size_t ZSTDMT_freeCCtx(ZSTDMT_CCtx* mtctx)
|
||||
{
|
||||
if (mtctx==NULL) return 0; /* compatible with free on NULL */
|
||||
POOL_free(mtctx->factory);
|
||||
if (!mtctx->allJobsCompleted) ZSTDMT_releaseAllJobResources(mtctx); /* stop workers first */
|
||||
ZSTDMT_freeBufferPool(mtctx->bufPool); /* release job resources into pools first */
|
||||
POOL_free(mtctx->factory); /* stop and free worker threads */
|
||||
ZSTDMT_releaseAllJobResources(mtctx); /* release job resources into pools first */
|
||||
ZSTD_free(mtctx->jobs, mtctx->cMem);
|
||||
ZSTDMT_freeBufferPool(mtctx->bufPool);
|
||||
ZSTDMT_freeCCtxPool(mtctx->cctxPool);
|
||||
ZSTD_freeCDict(mtctx->cdictLocal);
|
||||
pthread_mutex_destroy(&mtctx->jobCompleted_mutex);
|
||||
pthread_cond_destroy(&mtctx->jobCompleted_cond);
|
||||
ZSTD_pthread_mutex_destroy(&mtctx->jobCompleted_mutex);
|
||||
ZSTD_pthread_cond_destroy(&mtctx->jobCompleted_cond);
|
||||
ZSTD_free(mtctx, mtctx->cMem);
|
||||
return 0;
|
||||
}
|
||||
@ -496,22 +544,35 @@ size_t ZSTDMT_sizeof_CCtx(ZSTDMT_CCtx* mtctx)
|
||||
+ ZSTD_sizeof_CDict(mtctx->cdictLocal);
|
||||
}
|
||||
|
||||
size_t ZSTDMT_setMTCtxParameter(ZSTDMT_CCtx* mtctx, ZSDTMT_parameter parameter, unsigned value)
|
||||
{
|
||||
/* Internal only */
|
||||
size_t ZSTDMT_CCtxParam_setMTCtxParameter(
|
||||
ZSTD_CCtx_params* params, ZSTDMT_parameter parameter, unsigned value) {
|
||||
switch(parameter)
|
||||
{
|
||||
case ZSTDMT_p_sectionSize :
|
||||
mtctx->sectionSize = value;
|
||||
params->jobSize = value;
|
||||
return 0;
|
||||
case ZSTDMT_p_overlapSectionLog :
|
||||
DEBUGLOG(5, "ZSTDMT_p_overlapSectionLog : %u", value);
|
||||
mtctx->overlapLog = (value >= 9) ? 9 : value;
|
||||
DEBUGLOG(4, "ZSTDMT_p_overlapSectionLog : %u", value);
|
||||
params->overlapSizeLog = (value >= 9) ? 9 : value;
|
||||
return 0;
|
||||
default :
|
||||
return ERROR(parameter_unsupported);
|
||||
}
|
||||
}
|
||||
|
||||
size_t ZSTDMT_setMTCtxParameter(ZSTDMT_CCtx* mtctx, ZSTDMT_parameter parameter, unsigned value)
|
||||
{
|
||||
switch(parameter)
|
||||
{
|
||||
case ZSTDMT_p_sectionSize :
|
||||
return ZSTDMT_CCtxParam_setMTCtxParameter(&mtctx->params, parameter, value);
|
||||
case ZSTDMT_p_overlapSectionLog :
|
||||
return ZSTDMT_CCtxParam_setMTCtxParameter(&mtctx->params, parameter, value);
|
||||
default :
|
||||
return ERROR(parameter_unsupported);
|
||||
}
|
||||
}
|
||||
|
||||
/* ------------------------------------------ */
|
||||
/* ===== Multi-threaded compression ===== */
|
||||
@ -528,17 +589,17 @@ static unsigned computeNbChunks(size_t srcSize, unsigned windowLog, unsigned nbT
|
||||
return (multiplier>1) ? nbChunksLarge : nbChunksSmall;
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTDMT_compress_advanced(ZSTDMT_CCtx* mtctx,
|
||||
void* dst, size_t dstCapacity,
|
||||
const void* src, size_t srcSize,
|
||||
const ZSTD_CDict* cdict,
|
||||
ZSTD_parameters const params,
|
||||
unsigned overlapLog)
|
||||
static size_t ZSTDMT_compress_advanced_internal(
|
||||
ZSTDMT_CCtx* mtctx,
|
||||
void* dst, size_t dstCapacity,
|
||||
const void* src, size_t srcSize,
|
||||
const ZSTD_CDict* cdict,
|
||||
ZSTD_CCtx_params const params)
|
||||
{
|
||||
unsigned const overlapRLog = (overlapLog>9) ? 0 : 9-overlapLog;
|
||||
ZSTD_CCtx_params const jobParams = ZSTDMT_makeJobCCtxParams(params);
|
||||
unsigned const overlapRLog = (params.overlapSizeLog>9) ? 0 : 9-params.overlapSizeLog;
|
||||
size_t const overlapSize = (overlapRLog>=9) ? 0 : (size_t)1 << (params.cParams.windowLog - overlapRLog);
|
||||
unsigned nbChunks = computeNbChunks(srcSize, params.cParams.windowLog, mtctx->nbThreads);
|
||||
unsigned nbChunks = computeNbChunks(srcSize, params.cParams.windowLog, params.nbThreads);
|
||||
size_t const proposedChunkSize = (srcSize + (nbChunks-1)) / nbChunks;
|
||||
size_t const avgChunkSize = ((proposedChunkSize & 0x1FFFF) < 0x7FFF) ? proposedChunkSize + 0xFFFF : proposedChunkSize; /* avoid too small last block */
|
||||
const char* const srcStart = (const char*)src;
|
||||
@ -546,12 +607,14 @@ size_t ZSTDMT_compress_advanced(ZSTDMT_CCtx* mtctx,
|
||||
unsigned const compressWithinDst = (dstCapacity >= ZSTD_compressBound(srcSize)) ? nbChunks : (unsigned)(dstCapacity / ZSTD_compressBound(avgChunkSize)); /* presumes avgChunkSize >= 256 KB, which should be the case */
|
||||
size_t frameStartPos = 0, dstBufferPos = 0;
|
||||
XXH64_state_t xxh64;
|
||||
assert(jobParams.nbThreads == 0);
|
||||
assert(mtctx->cctxPool->totalCCtx == params.nbThreads);
|
||||
|
||||
DEBUGLOG(4, "nbChunks : %2u (chunkSize : %u bytes) ", nbChunks, (U32)avgChunkSize);
|
||||
if (nbChunks==1) { /* fallback to single-thread mode */
|
||||
ZSTD_CCtx* const cctx = mtctx->cctxPool->cctx[0];
|
||||
if (cdict) return ZSTD_compress_usingCDict_advanced(cctx, dst, dstCapacity, src, srcSize, cdict, params.fParams);
|
||||
return ZSTD_compress_advanced(cctx, dst, dstCapacity, src, srcSize, NULL, 0, params);
|
||||
if (cdict) return ZSTD_compress_usingCDict_advanced(cctx, dst, dstCapacity, src, srcSize, cdict, jobParams.fParams);
|
||||
return ZSTD_compress_advanced_internal(cctx, dst, dstCapacity, src, srcSize, NULL, 0, jobParams);
|
||||
}
|
||||
assert(avgChunkSize >= 256 KB); /* condition for ZSTD_compressBound(A) + ZSTD_compressBound(B) <= ZSTD_compressBound(A+B), which is required for compressWithinDst */
|
||||
ZSTDMT_setBufferSize(mtctx->bufPool, ZSTD_compressBound(avgChunkSize) );
|
||||
@ -580,7 +643,7 @@ size_t ZSTDMT_compress_advanced(ZSTDMT_CCtx* mtctx,
|
||||
mtctx->jobs[u].srcSize = chunkSize;
|
||||
mtctx->jobs[u].cdict = mtctx->nextJobID==0 ? cdict : NULL;
|
||||
mtctx->jobs[u].fullFrameSize = srcSize;
|
||||
mtctx->jobs[u].params = params;
|
||||
mtctx->jobs[u].params = jobParams;
|
||||
/* do not calculate checksum within sections, but write it in header for first section */
|
||||
if (u!=0) mtctx->jobs[u].params.fParams.checksumFlag = 0;
|
||||
mtctx->jobs[u].dstBuff = dstBuffer;
|
||||
@ -610,12 +673,12 @@ size_t ZSTDMT_compress_advanced(ZSTDMT_CCtx* mtctx,
|
||||
unsigned chunkID;
|
||||
for (chunkID=0; chunkID<nbChunks; chunkID++) {
|
||||
DEBUGLOG(5, "waiting for chunk %u ", chunkID);
|
||||
PTHREAD_MUTEX_LOCK(&mtctx->jobCompleted_mutex);
|
||||
ZSTD_PTHREAD_MUTEX_LOCK(&mtctx->jobCompleted_mutex);
|
||||
while (mtctx->jobs[chunkID].jobCompleted==0) {
|
||||
DEBUGLOG(5, "waiting for jobCompleted signal from chunk %u", chunkID);
|
||||
pthread_cond_wait(&mtctx->jobCompleted_cond, &mtctx->jobCompleted_mutex);
|
||||
ZSTD_pthread_cond_wait(&mtctx->jobCompleted_cond, &mtctx->jobCompleted_mutex);
|
||||
}
|
||||
pthread_mutex_unlock(&mtctx->jobCompleted_mutex);
|
||||
ZSTD_pthread_mutex_unlock(&mtctx->jobCompleted_mutex);
|
||||
DEBUGLOG(5, "ready to write chunk %u ", chunkID);
|
||||
|
||||
mtctx->jobs[chunkID].srcStart = NULL;
|
||||
@ -628,9 +691,8 @@ size_t ZSTDMT_compress_advanced(ZSTDMT_CCtx* mtctx,
|
||||
if (chunkID >= compressWithinDst) { /* chunk compressed into its own buffer, which must be released */
|
||||
DEBUGLOG(5, "releasing buffer %u>=%u", chunkID, compressWithinDst);
|
||||
ZSTDMT_releaseBuffer(mtctx->bufPool, mtctx->jobs[chunkID].dstBuff);
|
||||
}
|
||||
mtctx->jobs[chunkID].dstBuff = g_nullBuffer;
|
||||
}
|
||||
} }
|
||||
mtctx->jobs[chunkID].dstBuff = g_nullBuffer;
|
||||
dstPos += cSize ;
|
||||
}
|
||||
} /* for (chunkID=0; chunkID<nbChunks; chunkID++) */
|
||||
@ -651,6 +713,23 @@ size_t ZSTDMT_compress_advanced(ZSTDMT_CCtx* mtctx,
|
||||
}
|
||||
}
|
||||
|
||||
size_t ZSTDMT_compress_advanced(ZSTDMT_CCtx* mtctx,
|
||||
void* dst, size_t dstCapacity,
|
||||
const void* src, size_t srcSize,
|
||||
const ZSTD_CDict* cdict,
|
||||
ZSTD_parameters const params,
|
||||
unsigned overlapLog)
|
||||
{
|
||||
ZSTD_CCtx_params cctxParams = mtctx->params;
|
||||
cctxParams.cParams = params.cParams;
|
||||
cctxParams.fParams = params.fParams;
|
||||
cctxParams.overlapSizeLog = overlapLog;
|
||||
return ZSTDMT_compress_advanced_internal(mtctx,
|
||||
dst, dstCapacity,
|
||||
src, srcSize,
|
||||
cdict, cctxParams);
|
||||
}
|
||||
|
||||
|
||||
size_t ZSTDMT_compressCCtx(ZSTDMT_CCtx* mtctx,
|
||||
void* dst, size_t dstCapacity,
|
||||
@ -668,38 +747,25 @@ size_t ZSTDMT_compressCCtx(ZSTDMT_CCtx* mtctx,
|
||||
/* ======= Streaming API ======= */
|
||||
/* ====================================== */
|
||||
|
||||
static void ZSTDMT_waitForAllJobsCompleted(ZSTDMT_CCtx* zcs)
|
||||
{
|
||||
DEBUGLOG(4, "ZSTDMT_waitForAllJobsCompleted");
|
||||
while (zcs->doneJobID < zcs->nextJobID) {
|
||||
unsigned const jobID = zcs->doneJobID & zcs->jobIDMask;
|
||||
PTHREAD_MUTEX_LOCK(&zcs->jobCompleted_mutex);
|
||||
while (zcs->jobs[jobID].jobCompleted==0) {
|
||||
DEBUGLOG(5, "waiting for jobCompleted signal from chunk %u", zcs->doneJobID); /* we want to block when waiting for data to flush */
|
||||
pthread_cond_wait(&zcs->jobCompleted_cond, &zcs->jobCompleted_mutex);
|
||||
}
|
||||
pthread_mutex_unlock(&zcs->jobCompleted_mutex);
|
||||
zcs->doneJobID++;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/** ZSTDMT_initCStream_internal() :
|
||||
* internal usage only */
|
||||
size_t ZSTDMT_initCStream_internal(ZSTDMT_CCtx* zcs,
|
||||
const void* dict, size_t dictSize, const ZSTD_CDict* cdict,
|
||||
ZSTD_parameters params, unsigned long long pledgedSrcSize)
|
||||
size_t ZSTDMT_initCStream_internal(
|
||||
ZSTDMT_CCtx* zcs,
|
||||
const void* dict, size_t dictSize, ZSTD_dictMode_e dictMode,
|
||||
const ZSTD_CDict* cdict, ZSTD_CCtx_params params,
|
||||
unsigned long long pledgedSrcSize)
|
||||
{
|
||||
DEBUGLOG(4, "ZSTDMT_initCStream_internal");
|
||||
/* params are supposed to be fully validated at this point */
|
||||
assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams)));
|
||||
assert(!((dict) && (cdict))); /* either dict or cdict, not both */
|
||||
assert(zcs->cctxPool->totalCCtx == params.nbThreads);
|
||||
|
||||
if (zcs->nbThreads==1) {
|
||||
if (params.nbThreads==1) {
|
||||
ZSTD_CCtx_params const singleThreadParams = ZSTDMT_makeJobCCtxParams(params);
|
||||
DEBUGLOG(4, "single thread mode");
|
||||
assert(singleThreadParams.nbThreads == 0);
|
||||
return ZSTD_initCStream_internal(zcs->cctxPool->cctx[0],
|
||||
dict, dictSize, cdict,
|
||||
params, pledgedSrcSize);
|
||||
dict, dictSize, cdict,
|
||||
singleThreadParams, pledgedSrcSize);
|
||||
}
|
||||
|
||||
if (zcs->allJobsCompleted == 0) { /* previous compression not correctly finished */
|
||||
@ -714,7 +780,7 @@ size_t ZSTDMT_initCStream_internal(ZSTDMT_CCtx* zcs,
|
||||
DEBUGLOG(4,"cdictLocal: %08X", (U32)(size_t)zcs->cdictLocal);
|
||||
ZSTD_freeCDict(zcs->cdictLocal);
|
||||
zcs->cdictLocal = ZSTD_createCDict_advanced(dict, dictSize,
|
||||
0 /* byRef */, ZSTD_dm_auto, /* note : a loadPrefix becomes an internal CDict */
|
||||
ZSTD_dlm_byCopy, dictMode, /* note : a loadPrefix becomes an internal CDict */
|
||||
params.cParams, zcs->cMem);
|
||||
zcs->cdict = zcs->cdictLocal;
|
||||
if (zcs->cdictLocal == NULL) return ERROR(memory_allocation);
|
||||
@ -725,10 +791,10 @@ size_t ZSTDMT_initCStream_internal(ZSTDMT_CCtx* zcs,
|
||||
zcs->cdict = cdict;
|
||||
}
|
||||
|
||||
zcs->targetDictSize = (zcs->overlapLog==0) ? 0 : (size_t)1 << (zcs->params.cParams.windowLog - (9 - zcs->overlapLog));
|
||||
DEBUGLOG(4, "overlapLog : %u ", zcs->overlapLog);
|
||||
zcs->targetDictSize = (params.overlapSizeLog==0) ? 0 : (size_t)1 << (params.cParams.windowLog - (9 - params.overlapSizeLog));
|
||||
DEBUGLOG(4, "overlapLog : %u ", params.overlapSizeLog);
|
||||
DEBUGLOG(4, "overlap Size : %u KB", (U32)(zcs->targetDictSize>>10));
|
||||
zcs->targetSectionSize = zcs->sectionSize ? zcs->sectionSize : (size_t)1 << (zcs->params.cParams.windowLog + 2);
|
||||
zcs->targetSectionSize = params.jobSize ? params.jobSize : (size_t)1 << (params.cParams.windowLog + 2);
|
||||
zcs->targetSectionSize = MAX(ZSTDMT_SECTION_SIZE_MIN, zcs->targetSectionSize);
|
||||
zcs->targetSectionSize = MAX(zcs->targetDictSize, zcs->targetSectionSize);
|
||||
DEBUGLOG(4, "Section Size : %u KB", (U32)(zcs->targetSectionSize>>10));
|
||||
@ -749,8 +815,12 @@ size_t ZSTDMT_initCStream_advanced(ZSTDMT_CCtx* mtctx,
|
||||
ZSTD_parameters params,
|
||||
unsigned long long pledgedSrcSize)
|
||||
{
|
||||
ZSTD_CCtx_params cctxParams = mtctx->params;
|
||||
DEBUGLOG(5, "ZSTDMT_initCStream_advanced");
|
||||
return ZSTDMT_initCStream_internal(mtctx, dict, dictSize, NULL, params, pledgedSrcSize);
|
||||
cctxParams.cParams = params.cParams;
|
||||
cctxParams.fParams = params.fParams;
|
||||
return ZSTDMT_initCStream_internal(mtctx, dict, dictSize, ZSTD_dm_auto, NULL,
|
||||
cctxParams, pledgedSrcSize);
|
||||
}
|
||||
|
||||
size_t ZSTDMT_initCStream_usingCDict(ZSTDMT_CCtx* mtctx,
|
||||
@ -758,11 +828,12 @@ size_t ZSTDMT_initCStream_usingCDict(ZSTDMT_CCtx* mtctx,
|
||||
ZSTD_frameParameters fParams,
|
||||
unsigned long long pledgedSrcSize)
|
||||
{
|
||||
ZSTD_parameters params = ZSTD_getParamsFromCDict(cdict);
|
||||
ZSTD_CCtx_params cctxParams = mtctx->params;
|
||||
cctxParams.cParams = ZSTD_getCParamsFromCDict(cdict);
|
||||
cctxParams.fParams = fParams;
|
||||
if (cdict==NULL) return ERROR(dictionary_wrong); /* method incompatible with NULL cdict */
|
||||
params.fParams = fParams;
|
||||
return ZSTDMT_initCStream_internal(mtctx, NULL, 0 /*dictSize*/, cdict,
|
||||
params, pledgedSrcSize);
|
||||
return ZSTDMT_initCStream_internal(mtctx, NULL, 0 /*dictSize*/, ZSTD_dm_auto, cdict,
|
||||
cctxParams, pledgedSrcSize);
|
||||
}
|
||||
|
||||
|
||||
@ -770,14 +841,18 @@ size_t ZSTDMT_initCStream_usingCDict(ZSTDMT_CCtx* mtctx,
|
||||
* pledgedSrcSize is optional and can be zero == unknown */
|
||||
size_t ZSTDMT_resetCStream(ZSTDMT_CCtx* zcs, unsigned long long pledgedSrcSize)
|
||||
{
|
||||
if (zcs->nbThreads==1)
|
||||
if (zcs->params.nbThreads==1)
|
||||
return ZSTD_resetCStream(zcs->cctxPool->cctx[0], pledgedSrcSize);
|
||||
return ZSTDMT_initCStream_internal(zcs, NULL, 0, 0, zcs->params, pledgedSrcSize);
|
||||
return ZSTDMT_initCStream_internal(zcs, NULL, 0, ZSTD_dm_auto, 0, zcs->params,
|
||||
pledgedSrcSize);
|
||||
}
|
||||
|
||||
size_t ZSTDMT_initCStream(ZSTDMT_CCtx* zcs, int compressionLevel) {
|
||||
ZSTD_parameters const params = ZSTD_getParams(compressionLevel, 0, 0);
|
||||
return ZSTDMT_initCStream_internal(zcs, NULL, 0, NULL, params, 0);
|
||||
ZSTD_CCtx_params cctxParams = zcs->params;
|
||||
cctxParams.cParams = params.cParams;
|
||||
cctxParams.fParams = params.fParams;
|
||||
return ZSTDMT_initCStream_internal(zcs, NULL, 0, ZSTD_dm_auto, NULL, cctxParams, 0);
|
||||
}
|
||||
|
||||
|
||||
@ -856,13 +931,13 @@ static size_t ZSTDMT_flushNextJob(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output, unsi
|
||||
{
|
||||
unsigned const wJobID = zcs->doneJobID & zcs->jobIDMask;
|
||||
if (zcs->doneJobID == zcs->nextJobID) return 0; /* all flushed ! */
|
||||
PTHREAD_MUTEX_LOCK(&zcs->jobCompleted_mutex);
|
||||
ZSTD_PTHREAD_MUTEX_LOCK(&zcs->jobCompleted_mutex);
|
||||
while (zcs->jobs[wJobID].jobCompleted==0) {
|
||||
DEBUGLOG(5, "waiting for jobCompleted signal from job %u", zcs->doneJobID);
|
||||
if (!blockToFlush) { pthread_mutex_unlock(&zcs->jobCompleted_mutex); return 0; } /* nothing ready to be flushed => skip */
|
||||
pthread_cond_wait(&zcs->jobCompleted_cond, &zcs->jobCompleted_mutex); /* block when nothing available to flush */
|
||||
if (!blockToFlush) { ZSTD_pthread_mutex_unlock(&zcs->jobCompleted_mutex); return 0; } /* nothing ready to be flushed => skip */
|
||||
ZSTD_pthread_cond_wait(&zcs->jobCompleted_cond, &zcs->jobCompleted_mutex); /* block when nothing available to flush */
|
||||
}
|
||||
pthread_mutex_unlock(&zcs->jobCompleted_mutex);
|
||||
ZSTD_pthread_mutex_unlock(&zcs->jobCompleted_mutex);
|
||||
/* compression job completed : output can be flushed */
|
||||
{ ZSTDMT_jobDescription job = zcs->jobs[wJobID];
|
||||
if (!job.jobScanned) {
|
||||
@ -906,7 +981,7 @@ static size_t ZSTDMT_flushNextJob(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output, unsi
|
||||
|
||||
|
||||
/** ZSTDMT_compressStream_generic() :
|
||||
* internal use only
|
||||
* internal use only - exposed to be invoked from zstd_compress.c
|
||||
* assumption : output and input are valid (pos <= size)
|
||||
* @return : minimum amount of data remaining to flush, 0 if none */
|
||||
size_t ZSTDMT_compressStream_generic(ZSTDMT_CCtx* mtctx,
|
||||
@ -915,25 +990,26 @@ size_t ZSTDMT_compressStream_generic(ZSTDMT_CCtx* mtctx,
|
||||
ZSTD_EndDirective endOp)
|
||||
{
|
||||
size_t const newJobThreshold = mtctx->dictSize + mtctx->targetSectionSize;
|
||||
unsigned forwardInputProgress = 0;
|
||||
assert(output->pos <= output->size);
|
||||
assert(input->pos <= input->size);
|
||||
if ((mtctx->frameEnded) && (endOp==ZSTD_e_continue)) {
|
||||
/* current frame being ended. Only flush/end are allowed. Or start new frame with init */
|
||||
/* current frame being ended. Only flush/end are allowed */
|
||||
return ERROR(stage_wrong);
|
||||
}
|
||||
if (mtctx->nbThreads==1) { /* delegate to single-thread (synchronous) */
|
||||
if (mtctx->params.nbThreads==1) { /* delegate to single-thread (synchronous) */
|
||||
return ZSTD_compressStream_generic(mtctx->cctxPool->cctx[0], output, input, endOp);
|
||||
}
|
||||
|
||||
/* single-pass shortcut (note : this is synchronous-mode) */
|
||||
if ( (mtctx->nextJobID==0) /* just started */
|
||||
&& (mtctx->inBuff.filled==0) /* nothing buffered */
|
||||
&& (endOp==ZSTD_e_end) /* end order */
|
||||
/* single-pass shortcut (note : synchronous-mode) */
|
||||
if ( (mtctx->nextJobID == 0) /* just started */
|
||||
&& (mtctx->inBuff.filled == 0) /* nothing buffered */
|
||||
&& (endOp == ZSTD_e_end) /* end order */
|
||||
&& (output->size - output->pos >= ZSTD_compressBound(input->size - input->pos)) ) { /* enough room */
|
||||
size_t const cSize = ZSTDMT_compress_advanced(mtctx,
|
||||
size_t const cSize = ZSTDMT_compress_advanced_internal(mtctx,
|
||||
(char*)output->dst + output->pos, output->size - output->pos,
|
||||
(const char*)input->src + input->pos, input->size - input->pos,
|
||||
mtctx->cdict, mtctx->params, mtctx->overlapLog);
|
||||
mtctx->cdict, mtctx->params);
|
||||
if (ZSTD_isError(cSize)) return cSize;
|
||||
input->pos = input->size;
|
||||
output->pos += cSize;
|
||||
@ -946,15 +1022,16 @@ size_t ZSTDMT_compressStream_generic(ZSTDMT_CCtx* mtctx,
|
||||
/* fill input buffer */
|
||||
if (input->size > input->pos) { /* support NULL input */
|
||||
if (mtctx->inBuff.buffer.start == NULL) {
|
||||
mtctx->inBuff.buffer = ZSTDMT_getBuffer(mtctx->bufPool);
|
||||
if (mtctx->inBuff.buffer.start == NULL) return ERROR(memory_allocation);
|
||||
mtctx->inBuff.buffer = ZSTDMT_getBuffer(mtctx->bufPool); /* note : may fail, in which case, no forward input progress */
|
||||
mtctx->inBuff.filled = 0;
|
||||
}
|
||||
{ size_t const toLoad = MIN(input->size - input->pos, mtctx->inBuffSize - mtctx->inBuff.filled);
|
||||
if (mtctx->inBuff.buffer.start) {
|
||||
size_t const toLoad = MIN(input->size - input->pos, mtctx->inBuffSize - mtctx->inBuff.filled);
|
||||
DEBUGLOG(5, "inBuff:%08X; inBuffSize=%u; ToCopy=%u", (U32)(size_t)mtctx->inBuff.buffer.start, (U32)mtctx->inBuffSize, (U32)toLoad);
|
||||
memcpy((char*)mtctx->inBuff.buffer.start + mtctx->inBuff.filled, (const char*)input->src + input->pos, toLoad);
|
||||
input->pos += toLoad;
|
||||
mtctx->inBuff.filled += toLoad;
|
||||
forwardInputProgress = toLoad>0;
|
||||
} }
|
||||
|
||||
if ( (mtctx->inBuff.filled >= newJobThreshold) /* filled enough : let's compress */
|
||||
@ -963,7 +1040,7 @@ size_t ZSTDMT_compressStream_generic(ZSTDMT_CCtx* mtctx,
|
||||
}
|
||||
|
||||
/* check for potential compressed data ready to be flushed */
|
||||
CHECK_F( ZSTDMT_flushNextJob(mtctx, output, (mtctx->inBuff.filled == mtctx->inBuffSize) /* blockToFlush */) ); /* block if it wasn't possible to create new job due to saturation */
|
||||
CHECK_F( ZSTDMT_flushNextJob(mtctx, output, !forwardInputProgress /* blockToFlush */) ); /* block if there was no forward input progress */
|
||||
|
||||
if (input->pos < input->size) /* input not consumed : do not flush yet */
|
||||
endOp = ZSTD_e_continue;
|
||||
@ -1008,7 +1085,7 @@ static size_t ZSTDMT_flushStream_internal(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* outp
|
||||
size_t ZSTDMT_flushStream(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output)
|
||||
{
|
||||
DEBUGLOG(5, "ZSTDMT_flushStream");
|
||||
if (zcs->nbThreads==1)
|
||||
if (zcs->params.nbThreads==1)
|
||||
return ZSTD_flushStream(zcs->cctxPool->cctx[0], output);
|
||||
return ZSTDMT_flushStream_internal(zcs, output, 0 /* endFrame */);
|
||||
}
|
||||
@ -1016,7 +1093,7 @@ size_t ZSTDMT_flushStream(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output)
|
||||
size_t ZSTDMT_endStream(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output)
|
||||
{
|
||||
DEBUGLOG(4, "ZSTDMT_endStream");
|
||||
if (zcs->nbThreads==1)
|
||||
if (zcs->params.nbThreads==1)
|
||||
return ZSTD_endStream(zcs->cctxPool->cctx[0], output);
|
||||
return ZSTDMT_flushStream_internal(zcs, output, 1 /* endFrame */);
|
||||
}
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef ZSTDMT_COMPRESS_H
|
||||
@ -80,19 +81,19 @@ ZSTDLIB_API size_t ZSTDMT_initCStream_usingCDict(ZSTDMT_CCtx* mtctx,
|
||||
ZSTD_frameParameters fparams,
|
||||
unsigned long long pledgedSrcSize); /* note : zero means empty */
|
||||
|
||||
/* ZSDTMT_parameter :
|
||||
/* ZSTDMT_parameter :
|
||||
* List of parameters that can be set using ZSTDMT_setMTCtxParameter() */
|
||||
typedef enum {
|
||||
ZSTDMT_p_sectionSize, /* size of input "section". Each section is compressed in parallel. 0 means default, which is dynamically determined within compression functions */
|
||||
ZSTDMT_p_overlapSectionLog /* Log of overlapped section; 0 == no overlap, 6(default) == use 1/8th of window, >=9 == use full window */
|
||||
} ZSDTMT_parameter;
|
||||
} ZSTDMT_parameter;
|
||||
|
||||
/* ZSTDMT_setMTCtxParameter() :
|
||||
* allow setting individual parameters, one at a time, among a list of enums defined in ZSTDMT_parameter.
|
||||
* The function must be called typically after ZSTD_createCCtx().
|
||||
* Parameters not explicitly reset by ZSTDMT_init*() remain the same in consecutive compression sessions.
|
||||
* @return : 0, or an error code (which can be tested using ZSTD_isError()) */
|
||||
ZSTDLIB_API size_t ZSTDMT_setMTCtxParameter(ZSTDMT_CCtx* mtctx, ZSDTMT_parameter parameter, unsigned value);
|
||||
ZSTDLIB_API size_t ZSTDMT_setMTCtxParameter(ZSTDMT_CCtx* mtctx, ZSTDMT_parameter parameter, unsigned value);
|
||||
|
||||
|
||||
/*! ZSTDMT_compressStream_generic() :
|
||||
@ -107,6 +108,22 @@ ZSTDLIB_API size_t ZSTDMT_compressStream_generic(ZSTDMT_CCtx* mtctx,
|
||||
ZSTD_EndDirective endOp);
|
||||
|
||||
|
||||
/* === Private definitions; never ever use directly === */
|
||||
|
||||
size_t ZSTDMT_CCtxParam_setMTCtxParameter(ZSTD_CCtx_params* params, ZSTDMT_parameter parameter, unsigned value);
|
||||
|
||||
size_t ZSTDMT_initializeCCtxParameters(ZSTD_CCtx_params* params, unsigned nbThreads);
|
||||
|
||||
/*! ZSTDMT_initCStream_internal() :
|
||||
* Private use only. Init streaming operation.
|
||||
* expects params to be valid.
|
||||
* must receive dict, or cdict, or none, but not both.
|
||||
* @return : 0, or an error code */
|
||||
size_t ZSTDMT_initCStream_internal(ZSTDMT_CCtx* zcs,
|
||||
const void* dict, size_t dictSize, ZSTD_dictMode_e dictMode,
|
||||
const ZSTD_CDict* cdict,
|
||||
ZSTD_CCtx_params params, unsigned long long pledgedSrcSize);
|
||||
|
||||
|
||||
#if defined (__cplusplus)
|
||||
}
|
||||
|
File diff suppressed because it is too large
Load Diff
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
/* ***************************************************************
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
/*-*************************************
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
/* *****************************************************************************
|
||||
@ -382,7 +383,7 @@ static void COVER_group(COVER_ctx_t *ctx, const void *group,
|
||||
typedef struct {
|
||||
U32 begin;
|
||||
U32 end;
|
||||
double score;
|
||||
U32 score;
|
||||
} COVER_segment_t;
|
||||
|
||||
/**
|
||||
@ -479,11 +480,16 @@ static COVER_segment_t COVER_selectSegment(const COVER_ctx_t *ctx, U32 *freqs,
|
||||
* Check the validity of the parameters.
|
||||
* Returns non-zero if the parameters are valid and 0 otherwise.
|
||||
*/
|
||||
static int COVER_checkParameters(ZDICT_cover_params_t parameters) {
|
||||
static int COVER_checkParameters(ZDICT_cover_params_t parameters,
|
||||
size_t maxDictSize) {
|
||||
/* k and d are required parameters */
|
||||
if (parameters.d == 0 || parameters.k == 0) {
|
||||
return 0;
|
||||
}
|
||||
/* k <= maxDictSize */
|
||||
if (parameters.k > maxDictSize) {
|
||||
return 0;
|
||||
}
|
||||
/* d <= k */
|
||||
if (parameters.d > parameters.k) {
|
||||
return 0;
|
||||
@ -622,9 +628,13 @@ static size_t COVER_buildDictionary(const COVER_ctx_t *ctx, U32 *freqs,
|
||||
/* Select a segment */
|
||||
COVER_segment_t segment = COVER_selectSegment(
|
||||
ctx, freqs, activeDmers, epochBegin, epochEnd, parameters);
|
||||
/* Trim the segment if necessary and if it is empty then we are done */
|
||||
/* If the segment covers no dmers, then we are out of content */
|
||||
if (segment.score == 0) {
|
||||
break;
|
||||
}
|
||||
/* Trim the segment if necessary and if it is too small then we are done */
|
||||
segmentSize = MIN(segment.end - segment.begin + parameters.d - 1, tail);
|
||||
if (segmentSize == 0) {
|
||||
if (segmentSize < parameters.d) {
|
||||
break;
|
||||
}
|
||||
/* We fill the dictionary from the back to allow the best segments to be
|
||||
@ -648,7 +658,7 @@ ZDICTLIB_API size_t ZDICT_trainFromBuffer_cover(
|
||||
COVER_ctx_t ctx;
|
||||
COVER_map_t activeDmers;
|
||||
/* Checks */
|
||||
if (!COVER_checkParameters(parameters)) {
|
||||
if (!COVER_checkParameters(parameters, dictBufferCapacity)) {
|
||||
DISPLAYLEVEL(1, "Cover parameters incorrect\n");
|
||||
return ERROR(GENERIC);
|
||||
}
|
||||
@ -701,8 +711,8 @@ ZDICTLIB_API size_t ZDICT_trainFromBuffer_cover(
|
||||
* compiled with multithreaded support.
|
||||
*/
|
||||
typedef struct COVER_best_s {
|
||||
pthread_mutex_t mutex;
|
||||
pthread_cond_t cond;
|
||||
ZSTD_pthread_mutex_t mutex;
|
||||
ZSTD_pthread_cond_t cond;
|
||||
size_t liveJobs;
|
||||
void *dict;
|
||||
size_t dictSize;
|
||||
@ -715,8 +725,8 @@ typedef struct COVER_best_s {
|
||||
*/
|
||||
static void COVER_best_init(COVER_best_t *best) {
|
||||
if (best==NULL) return; /* compatible with init on NULL */
|
||||
(void)pthread_mutex_init(&best->mutex, NULL);
|
||||
(void)pthread_cond_init(&best->cond, NULL);
|
||||
(void)ZSTD_pthread_mutex_init(&best->mutex, NULL);
|
||||
(void)ZSTD_pthread_cond_init(&best->cond, NULL);
|
||||
best->liveJobs = 0;
|
||||
best->dict = NULL;
|
||||
best->dictSize = 0;
|
||||
@ -731,11 +741,11 @@ static void COVER_best_wait(COVER_best_t *best) {
|
||||
if (!best) {
|
||||
return;
|
||||
}
|
||||
pthread_mutex_lock(&best->mutex);
|
||||
ZSTD_pthread_mutex_lock(&best->mutex);
|
||||
while (best->liveJobs != 0) {
|
||||
pthread_cond_wait(&best->cond, &best->mutex);
|
||||
ZSTD_pthread_cond_wait(&best->cond, &best->mutex);
|
||||
}
|
||||
pthread_mutex_unlock(&best->mutex);
|
||||
ZSTD_pthread_mutex_unlock(&best->mutex);
|
||||
}
|
||||
|
||||
/**
|
||||
@ -749,8 +759,8 @@ static void COVER_best_destroy(COVER_best_t *best) {
|
||||
if (best->dict) {
|
||||
free(best->dict);
|
||||
}
|
||||
pthread_mutex_destroy(&best->mutex);
|
||||
pthread_cond_destroy(&best->cond);
|
||||
ZSTD_pthread_mutex_destroy(&best->mutex);
|
||||
ZSTD_pthread_cond_destroy(&best->cond);
|
||||
}
|
||||
|
||||
/**
|
||||
@ -761,9 +771,9 @@ static void COVER_best_start(COVER_best_t *best) {
|
||||
if (!best) {
|
||||
return;
|
||||
}
|
||||
pthread_mutex_lock(&best->mutex);
|
||||
ZSTD_pthread_mutex_lock(&best->mutex);
|
||||
++best->liveJobs;
|
||||
pthread_mutex_unlock(&best->mutex);
|
||||
ZSTD_pthread_mutex_unlock(&best->mutex);
|
||||
}
|
||||
|
||||
/**
|
||||
@ -779,7 +789,7 @@ static void COVER_best_finish(COVER_best_t *best, size_t compressedSize,
|
||||
}
|
||||
{
|
||||
size_t liveJobs;
|
||||
pthread_mutex_lock(&best->mutex);
|
||||
ZSTD_pthread_mutex_lock(&best->mutex);
|
||||
--best->liveJobs;
|
||||
liveJobs = best->liveJobs;
|
||||
/* If the new dictionary is better */
|
||||
@ -802,9 +812,9 @@ static void COVER_best_finish(COVER_best_t *best, size_t compressedSize,
|
||||
best->parameters = parameters;
|
||||
best->compressedSize = compressedSize;
|
||||
}
|
||||
pthread_mutex_unlock(&best->mutex);
|
||||
ZSTD_pthread_mutex_unlock(&best->mutex);
|
||||
if (liveJobs == 0) {
|
||||
pthread_cond_broadcast(&best->cond);
|
||||
ZSTD_pthread_cond_broadcast(&best->cond);
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -884,7 +894,7 @@ static void COVER_tryParameters(void *opaque) {
|
||||
goto _compressCleanup;
|
||||
}
|
||||
/* Compress each sample and sum their sizes (or error) */
|
||||
totalCompressedSize = 0;
|
||||
totalCompressedSize = dictBufferCapacity;
|
||||
for (i = 0; i < ctx->nbSamples; ++i) {
|
||||
const size_t size = ZSTD_compress_usingCDict(
|
||||
cctx, dst, dstCapacity, ctx->samples + ctx->offsets[i],
|
||||
@ -960,7 +970,7 @@ ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_cover(
|
||||
/* Initialization */
|
||||
COVER_best_init(&best);
|
||||
/* Turn down global display level to clean up display at level 2 and below */
|
||||
g_displayLevel = parameters->zParams.notificationLevel - 1;
|
||||
g_displayLevel = displayLevel == 0 ? 0 : displayLevel - 1;
|
||||
/* Loop through d first because each new value needs a new context */
|
||||
LOCALDISPLAYLEVEL(displayLevel, 2, "Trying %u different sets of parameters\n",
|
||||
kIterations);
|
||||
@ -994,8 +1004,9 @@ ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_cover(
|
||||
data->parameters.k = k;
|
||||
data->parameters.d = d;
|
||||
data->parameters.steps = kSteps;
|
||||
data->parameters.zParams.notificationLevel = g_displayLevel;
|
||||
/* Check the parameters */
|
||||
if (!COVER_checkParameters(data->parameters)) {
|
||||
if (!COVER_checkParameters(data->parameters, dictBufferCapacity)) {
|
||||
DISPLAYLEVEL(1, "Cover parameters incorrect\n");
|
||||
free(data);
|
||||
continue;
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
|
||||
@ -60,7 +61,7 @@
|
||||
|
||||
#define NOISELENGTH 32
|
||||
|
||||
static const int g_compressionLevel_default = 6;
|
||||
static const int g_compressionLevel_default = 3;
|
||||
static const U32 g_selectivity_default = 9;
|
||||
|
||||
|
||||
@ -374,7 +375,7 @@ static int isIncluded(const void* in, const void* container, size_t length)
|
||||
return u==length;
|
||||
}
|
||||
|
||||
/*! ZDICT_checkMerge
|
||||
/*! ZDICT_tryMerge() :
|
||||
check if dictItem can be merged, do it if possible
|
||||
@return : id of destination elt, 0 if not merged
|
||||
*/
|
||||
@ -439,8 +440,8 @@ static U32 ZDICT_tryMerge(dictItem* table, dictItem elt, U32 eltNbToSkip, const
|
||||
|
||||
static void ZDICT_removeDictItem(dictItem* table, U32 id)
|
||||
{
|
||||
/* convention : first element is nb of elts */
|
||||
U32 const max = table->pos;
|
||||
/* convention : table[0].pos stores nb of elts */
|
||||
U32 const max = table[0].pos;
|
||||
U32 u;
|
||||
if (!id) return; /* protection, should never happen */
|
||||
for (u=id; u<max-1; u++)
|
||||
@ -703,7 +704,7 @@ static size_t ZDICT_analyzeEntropy(void* dstBuffer, size_t maxDstSize,
|
||||
memset(repOffset, 0, sizeof(repOffset));
|
||||
repOffset[1] = repOffset[4] = repOffset[8] = 1;
|
||||
memset(bestRepOffset, 0, sizeof(bestRepOffset));
|
||||
if (compressionLevel==0) compressionLevel = g_compressionLevel_default;
|
||||
if (compressionLevel<=0) compressionLevel = g_compressionLevel_default;
|
||||
params = ZSTD_getParams(compressionLevel, averageSampleSize, dictBufferSize);
|
||||
{ size_t const beginResult = ZSTD_compressBegin_advanced(esr.ref, dictBuffer, dictBufferSize, params, 0);
|
||||
if (ZSTD_isError(beginResult)) {
|
||||
@ -1056,6 +1057,8 @@ size_t ZDICT_trainFromBuffer(void* dictBuffer, size_t dictBufferCapacity,
|
||||
memset(¶ms, 0, sizeof(params));
|
||||
params.d = 8;
|
||||
params.steps = 4;
|
||||
/* Default to level 6 since no compression level information is avaialble */
|
||||
params.zParams.compressionLevel = 6;
|
||||
return ZDICT_optimizeTrainFromBuffer_cover(dictBuffer, dictBufferCapacity,
|
||||
samplesBuffer, samplesSizes,
|
||||
nbSamples, ¶ms);
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef DICTBUILDER_H_001
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef ZSTD_LEGACY_H
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef ZSTD_V01_H_28739879432
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef ZSTD_V02_H_4174539423
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
#ifndef ZSTD_V03_H_298734209782
|
||||
|
@ -5,6 +5,7 @@
|
||||
* This source code is licensed under both the BSD-style license (found in the
|
||||
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
|
||||
* in the COPYING file in the root directory of this source tree).
|
||||
* You may select, at your option, one of the above-listed licenses.
|
||||
*/
|
||||
|
||||
|
||||
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user