and geom_uncompress(4):
1. mkuzip(8):
- Proper support for eliminating all-zero blocks when compressing an
image. This feature is already supported by the geom_uzip(4) module
and CLOOP format in general, so it's just a matter of making mkuzip(8)
match. It should be noted, however that this feature while it sounds
great, results in very slight improvement in the overall compression
ratio, since compressing default 16k all-zero block produces only 39
bytes compressed output block, which is 99.8% compression ratio. With
typical average compression ratio of amd64 binaries and data being
around 60-70% the difference between 99.8% and 100.0% is not that
great further diluted by the ratio of number of zero blocks in the
uncompressed image to the overall number of blocks being less than
0.5 (typically). However, this may be important from performance
standpoint, so that kernel are not spinning its wheels decompressing
those empty blocks every time this zero region is read. It could also
be important when you create huge image mostly filled with zero
blocks for testing purposes.
- New feature allowing to de-duplicate output image. It turns out that
if you twist CLOOP format a bit you can do that as well. And unlike
zero-blocks elimination, this gives a noticeable improvement in the
overall compression ratio, reducing output image by something like
3-4% on my test UFS2 3GB image consisting of full FreeBSD base system
plus some of the packages (openjdk, apache etc), about 2.3GB worth of
file data (800+MB compressed). The only caveat is that images created
with this feature "on" would not work on older versions of FeeBSDxi
kernel, hence it's turned off by default.
- provide options to control both features and document them in manual
page.
- merge in all relevant LZMA compression support from the mkulzma(8),
add new option to select between both.
- switch license from ad-hoc beerware into standard 2-clause BSD.
2. geom_uzip(4):
- implement support for de-duplicated images;
- optimize some code paths to handle "all-zero" blocks without reading
any compressed data;
- beef up manual page to explain that geom_uzip(4) is not limited only
to md(4) images. The compressed data can be written to the block
device and accessed directly via magic of GEOM(4) and devfs(4),
including to mount root fs from a compressed drive.
- convert debug log code from being compiled in conditionally into
being present all the time and provide two sysctls to turn it on or
off. Due to intended use of the module, it can be used in
environments where there may not be a luxury to put new kernel with
debug code enabled. Having those options handy allows debug issues
without as much problem by just having access to serial console or
network shell access to a box/appliance. The resulting additional
CPU cycles are just few int comparisons and branches, and those are
minuscule when compared to data decompression which is the main
feature of the module.
- hopefully improve robustness and resiliency of the geom_uzip(4) by
performing some of the data validation / range checking on the TOC
entries and rejecting to attach to an image if those checks fail.
- merge in all relevant LZMA decompression support from the
geom_uncompress(4), enable automatically when appropriate format is
indicated in the header.
- move compilation work into its own worker thread so that it does not
clog g_up. This allows multiple instances work in parallel utilizing
smp cores.
- document new knobs in the manual page.
Reviewed by: adrian
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D5333
Add two new functions, sysdecode_abi_to_freebsd_errno() and
sysdecode_freebsd_to_abi_errno(), which convert errno values between
the native FreeBSD ABI and other supported ABIs. Note that the
mappings are not necessarily perfect meaning in some cases multiple
errors in one ABI might map to a single error in another ABI. In that
case, the reverse mapping will return one of the errors that maps, but
which error is non-deterministic.
Change truss to always report the raw error value to the user but
use libsysdecode to map it to a native errno value that can be used
with strerror() to generate a description. Previously truss reported
the "converted" error value. Now the user will always see the exact
error value that the application sees.
Change kdump to report the truly raw error value to the user. Previously
kdump would report the absolute value of the raw error value (so for
Linux binaries it didn't output the FreeBSD error value, but the positive
value of the Linux error). Now it reports the real (i.e. negative) error
value for Linux binaries. Also, use libsysdecode to convert the native
FreeBSD error reported in the ktrace record to the raw error used by the
ABI. This means that the Linux ABI can now be handled directly in
ktrsysret() and removes the need for linux_ktrsysret().
Reviewed by: bdrewery, kib
Helpful notes: wblock (manpage)
Differential Revision: https://reviews.freebsd.org/D5314
- Consolidate duplicate code for printing the metadata at the start of
each line into a shared function.
- Add an -H option which will log the thread ID of the relevant thread
for each event.
While here, remove some extraneous calls to clock_gettime() in
print_syscall() and print_syscall_ret(). The caller of print_syscall_ret()
always updates the current thread's "after" time before it is called.
Reviewed by: kib
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D5363
The output file is created as a temporary file that is moved over the
existing file after completion. Thus there is no need to immediately
flush all created db records to the temporary file.
This speeds up creation of the termcap db by a factor of 40 on my
ZFS based /etc filesytem (from 25 seconds to 0.6 seconds).
I have compared multiple output files created with and without O_SYNC
and they came out identical each time. Nonetheless it might be best
to MFC this change and the similar one for services_mkdb (r295465) at
the same time when the changes to hash.c in review D5186 are merged.
MFC: 1 week
The wrapper script has moved to libexec/makewhatis.local since it is not
directlry related to the older makewhatis(1) utility that has been replaced
by the usr.bin/mandoc version.
Reported by: vangyzen
instead of passing some of that state as arguments to print_syscall() and
print_syscallret(). This just makes the calls of these functions shorter
and easier to read.
For scripts using dialog(1) several times, it can be visually distracting
running dpv(1) several times amidst other dialogs. The `-k' option, similar
to dialog(1) `--keep-tite', enables the same functionality to smooth ti/te.
A new sysdecode_syscallname() function accepts a system call code and
returns a string of the corresponding name (or NULL if the code is
unknown). To support different process ABIs, the new function accepts a
value from a new sysdecode_abi enum as its first argument to select the
ABI in use. Current ABIs supported include FREEBSD (native binaries),
FREEBSD32, LINUX, LINUX32, and CLOUDABI64. Note that not all ABIs are
supported by all platforms. In general, a given ABI is only supported
if a platform can execute binaries for that ABI.
To simplify the implementation, libsysdecode's build reuses the
existing pre-generated files from the kernel source tree rather than
duplicating new copies of said files during the build.
kdump(1) and truss(1) now use these functions to map system call
identifiers to names. For kdump(1), a new 'syscallname()' function
consolidates duplicated code from ktrsyscall() and ktrsyscallret().
The Linux ABI no longer requires custom handling for ktrsyscall() and
linux_ktrsyscall() has been removed as a result.
Reviewed by: bdrewery
Differential Revision: https://reviews.freebsd.org/D4823
STT_SPARC_REGISTER is a SPARC-specific symbol type specified by the
Sparcv9 ABI to provide some information on register use by the object.
Also rework st_info type lookup to avoid out-of-bounds array access.
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Ensure the special cases trigger whether we come via a referral
or via the -c option. Match host names case-insensitively.
Use the default character set supported by .de (UTF-8) since that
is more compatible with the modern world than ISO 8859-1. Persuade
them to give us a useful answer whether an internationalized
domain name is given in UTF-8 or in punycode.
The IANA whois server has the right referral information for domain
names, IP addresses, and AS numbers, so whois does not need to be
able to choose servers itself (except for a few cases where referrals
do not work). We can delete a chunk of code, which is always fun.
This change improves the referral handling to be less sensitive to
all the various formats, and to allow multi-hop referral chains,
such as IANA -> registry -> registrar.
ARIN queries have the "+" flag added if no flags are present, so we
get full details if the query matches multiple objects. The Verisign
anti-spam logic is also now suppressed if the user provided a non-
trivial query string.
Uninformative rubric is now trimmed by default. The -S option
turns off trimming, and disables query fettling.
The -i option is back to its traditional pre-1999 hostname, since
whois.internic.net is more useful than whois.networksolutions.com.
Note that the old fallback/default server whois.crsnic.net is an
alias for whois.internic.net.
The manual is more informative about query syntax.
Look up AS numbers at ARIN.
Handle more referral formats.
Suppress spammy nameserver objects when querying the .com and .net
whois servers by explicitly querying for domain names by default.
The bad_truncate test sets the uimmutable flag to produce an error in
truncate, but that flag isn't supported by ZFS. If /tmp is on a ZFS
filesystem, the test will fail. Change it to use readonly permissions and
an unpriveleged user instead.
Reviewed by: jilles
MFC after: 1 week
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D4862
linking. These are too large for a branch instruction to branch from an
earlier point in the code to somewhere later.
This will also allow these to be build with Thumb-2 when we get this
infrastructure.
Reviewed by: dim
Differential Revision: https://reviews.freebsd.org/D4855
numactl was only modifying its own CPU affinity, which is fine
when creating a new process, but not very helpful when modifying
an existing processes.
Reviewed by: adrian
Sponsored by: Dell Inc.
Differential Revision: https://reviews.freebsd.org/D4927
up to now.
The new sendfile is the code that Netflix uses to send their multiple tens
of gigabits of data per second. The new implementation features asynchronous
I/O, when I/O operations are launched, but not awaited to be complete. An
explanation of why such behavior is beneficial compared to old one is
going to be too long for a commit message, so we will skip it here.
Additional features of new syscall are extra flags, which provide an
application more control over data sent. The SF_NOCACHE flag tells
kernel that data shouldn't be cached after it was sent. The SF_READAHEAD()
macro allows to specify readahead size in pages.
The new syscalls is a drop in replacement. No modifications are required
to applications. One can take nginx binary for stable/10 and run it
successfully on head. Although SF_NODISKIO lost its original sense, as now
sendfile doesn't block, and now means something completely different (tm),
using the new sendfile the old way is absolutely safe.
Celebrates: Netflix global launch!
Sponsored by: Nginx, Inc.
Sponsored by: Netflix
Relnotes: yes
Similar fix was done for passwd and group operations in r285050. When a
temporary file is created and then renamed to replace official file there
are no checks to make sure data was written to disk and if a power cycle
happens at this time, system can end up with a 0 length file
Approved by: bapt
MFC after: 1 week
Sponsored by: Netgate
Differential Revision: https://reviews.freebsd.org/D2982
