* If write block size is zero, don't block at all.
This supports the unusual requirement of applications
that need "no-delay" writes.
* Expose _write_finish_entry() to give such applications more
control over write boundaries. (Normal applications do not
need this, as entries are completed automatically.)
* Correct the type of write callbacks; this is a minor API
change that does not affect the ABI.
* Correct the error handling in _write_next_header() around
completing the previous entry.
* Correct the documentation for block-size markers: Remove
docs for the long-defunct _read_set_block_size(); document
all of the write block size manipulators.
MFC after: 14 days
traditional shortcut of defining on-disk layouts using structures of
character arrays. Unfortunately, as recently discussed on cvs-all@,
this usage is not actually sanctioned by the standards and
specifically fails on GCC/arm (unless your data structures happen to
be "naturally aligned").
The new code defines offsets/sizes for data fields and accesses
them using explicit pointer arithmetic, instead of casting to
a structure and accessing structure fields. In particular,
the new code is now clean with WARNS=6 on arm.
MFC after: 14 days
and correct the use of unary minus with an unsigned value. (The unary
minus here is actually being used as a bitwise operation, which is
unusual enough to deserve a clarifying cast.)
archive_{read,write}_open_filename():
* Update Makefile to build the files using the new name.
* Update docs to document the new names, mentioning the
old ones as "deprecated synonyms."
* The old filenames will be reconnected to the build soon;
I'll soon recyce those files for a slightly different purpose.
internal format-specific functions return the same as the public
function, so that the public API layer doesn't have to guess the
correct return value. This addresses an obscure problem that occurs
when someone tries to write more data than the size of the entry (as
indicated in the entry header). In this case, the return value from
archive_write_data() was incorrect, reflecting the requested write
rather than the amount actually written.
MFC after: 15 days
* Use public API, don't access struct archive directly. (People should be able to copy these into their applications as a template for custom I/O callbacks.)
* Set "skip" only for regular files. ("skip" allows the low-level library to catch attempts to add an archive to itself or extract over itself.)
* Simplify the write_open functions by just calling stat() at the beginning. Somehow, these functions had acquired some complex logic that tried to avoid the stat() call but never succeeded.
MFC after: 10 days
file. This doesn't happen in normal use, because the file I/O and
decompression layers only pass through smaller blocks. It can happen
with custom read functions that block I/O in larger blocks.
* Actually use the HAVE_<header>_H macros to conditionally include
system headers. They've been defined for a long time, but only
used in a few places. Now they're used pretty consistently
throughout.
* Fill in a lot of missing casts for conversions from void*.
Although Standard C doesn't require this, some people have been
trying to use C++ compilers with this code, and they do require it.
Bit-for-bit, the compiled object files are identical, except for
one assert() whose line number changed, so I'm pretty confident I
didn't break anything. ;-)
restore it directly and skip chmod() during the post-extract fixup.
In particular, bsdtar -xm now completely skips the post-extract fixup
for directories, which produces a noticable speedup in that case.
* Expose functions for setting the "skip file" dev/ino information
* Expose functions for setting/querying the block size on reads
* Correctly propagate errors out of archive_read_close/archive_write_close
* Update manpage with information about new functions
increases performance when extracting a single entry from a large
uncompressed archive, especially on slow devices such as USB hard
drives.
Requires a number of changes:
* New archive_read_open2() supports a 'skip' client function
* Old archive_read_open() is implemented as a wrapper now, to
continue supporting the old API/ABI.
* _read_open_fd and _read_open_file sprout new 'skip' functions.
* compression layer gets a new 'skip' operation.
* compression_none passes skip requests through to client.
* compression_{gzip,bzip2,compress} simply ignore skip requests.
Thanks to: Benjamin Lutz, who designed and implemented the whole thing.
I'm just committing it. ;-)
TODO: Need to update the documentation a little bit.
pax wasn't introduced until the 1993 (?) revision.
(I need to double-check when pax was introduced and
clarify some of the history here. In particular,
I should explain that the 'pax' standard now owns the
'ustar' format spec.)
in part by OpenBSD's not-quite-standard-compliant
standard libraries. (No loss of functionality,
just minor recoding to not rely on certain "standard"
facilities that weren't actually needed.)
it's only a failure if there were actually attributes to be restored.
In particular, this fixes the problem where tar -xp always returned
a failure code on FreeBSD (which doesn't yet have all of the extended
attribute support).
Thanks to: Diego "Flameeyes" Petteno
This commit implements storing/reading POSIX.1e-style extended
attribute information in "pax" format archives. An outline of the
storage format is in the tar.5 manpage. The archive_read_extract()
function has code to restore those archives to disk for Linux; FreeBSD
implementation is forthcoming.
Many thanks to Jaakko Heinonen for finding flaws in earlier
proposals and doing the bulk of the coding in this work.
routine fails or the first read fails), invoke the client close
routine immediately so the client can clean up. Also, don't store the
client pointers in this case, so that the client close routine can't
accidentally get called more than once.
A minor style fix to archive_read_open_fd.c while I'm here.
PR: 86453
Thanks to: Andrew Turner for reporting this and suggesting a fix.
archiver for Fourth Edition through Sixth Edition Unix; it was
replaced by tar in Seventh Edition. (First Edition through
Third Edition used "tap.")
Unfortunately, tp was not so very standard; there were a
few different variants. The code here attempts to support
what I believe were the most common variants.
tp support is not yet enabled by archive_read_support_format_all(),
as I'm not yet entirely comfortable with the detection
heuristics. People interested in experimenting can
add archive_read_support_format_tp() just after any calls
to archive_read_support_format_all() in bsdtar to see how
well this works.
TODO: tp format is roughly similar in structure to dump/restore
archive formats used by many systems. It should be possible
to generalize this code to handle many dump/restore variants.
Format detection heuristics are going to be rough, though.
Thanks to: Warren Toomey, whose very basic tp extraction programs
and documentation made this possible.
libarchive doesn't make malloc(0) requests, so the autoconf
checks aren't needed and the autoconf workarounds for
broken malloc(0) just create problems.
Thanks to: Dan Nelson, who reports that this fixes libarchive on AIX 5.2
expr and printf are not available during installworld, so
use /bin/sh arithmetic expansion instead of expr and simply
give up on vanity formatting. ;-)
systems (or on FreeBSD systems when using ports).
2) Overhaul the versioning logic. In particular,
SHLIB_MAJOR number is now computed as "major+minor",
which ensures library versions are the same for
the FreeBSD build system and the portable
libtool/autoconf/automake build system.
link names, usernames, or group names that contain
non-ASCII characters.
In particular, this corrects an inconsistency reported
by Ed Maste when archiving symlinks with odd characters:
long symlinks would get preserved, short ones would
be changed.
"HEADER" unless the open is successful. Instead, leave the state as
"NEW." In particular, if archive_read_open() fails, a subsequent call
to archive_read_next_header() will now cause an explicit assertion
failure instead of a silent segmentation fault.
This may need a little more work to fully realize the intention: If
archive_read_open() fails, you should be able to call it again on the
same archive handle to open a different archive (or the same archive
using a different mechanism).
(wchar_t is defined in stddef.h, and only two files need more than that.)
Portability: Since the wchar requirements are really quite modest,
it's easy to define basic replacements for wcslen, wcscmp, wcscpy,
etc, for use on systems that lack <wchar.h>. In particular, this allows
libarchive to be used on older OpenBSD systems.
extracted from tar archives. Otherwise, converting tar archives to
cpio format (with "bsdtar -cf out.cpio @in.tar") convert every entry
into a hard link to a single file. This simple logic breaks hard
links, but that's better than the alternative.
MFC after: 7 days
header of the pax extension entry, clip them to ustar limits. In particular,
this prevents an internal panic for very old files.
Thanks to: Chris Spiegel
MFC after: 7 days
GNU tar sparse files, people have extended cpio) and clarify an
important detail about pax format (that ustar-compliant archivers
can mostly read pax archives correctly).
MFC after: 7 days
compiling on IRIX and Solaris. Remove the "archive_check_magic" macro
that existed only to provide __func__ to the underlying __archive_check_magic
function.
Thanks to: Darin Broady
MFC after: 14 days
from mode before using mode for extended attributes entry, copy
mtime/atime/ctime to extended attributes entry so it's a little more
clear that it corresponds to the like-named regular entry.
MFC after: 14 days
and restoring the metadata. In particular, the metadata-restore
functions now all accept a file descriptor and a pathname. If the
file descriptor is set and the platform supports the appropriate
syscall, restore the metadata through the file descriptor. Otherwise,
restore it through the pathname. This is complicated by varying
syscall support (FreeBSD has an fchmod(2) but no fchflags(2), for
example) and because non-file entries don't have an fd to use in
restoring attributes (for example, mknod(2) doesn't return a file
handle).
MFC after: 14 days
(symlink or hardlink) is already set. Instead, it was always setting
the hardlink field. In particular, this caused GNU tar format long
symlinks to be interpreted as hardlinks.
Thanks to: Brooks Davis
MFC after: 7 days
internal error if pax extended attributes were being generated. Being
< 255 characters, the first-pass path editing (to generate a
ustar-compatible name for the main entry) wouldn't occur, and the
second-pass path editing (to generate a ustar name for the pax
attributes entry) assumed the input was already < 245 chars.
The core problem here was using an abbreviated algorithm for the
second pass that relied on the first pass having already run. The
rewritten code is much simpler: It just uses the full path-shortening
algorithm for building both ustar pathnames. This way, the second
ustar pathname will always be short enough.
Thanks to: Mark Cammidge
Related to: bin/74385
* Handles entries with compressed size >2GB (signed/unsigned cleanup)
* Handles entries with compressed size >4GB ("ZIP64" extension)
* Handles Unix extensions (ctime, atime, mtime, mode, uid, etc)
* Format-specific "skip data" override allows ZIP reader to skip
entries without decompressing them, which makes "tar -t"
a lot faster.
* Handles "length-at-end" entries generated by, e.g., "zip -r - foo"
Many thanks to: Dan Nelson, who contributed the code and test files for
the first three items above and suggested the fourth.
return a generic text message instead.
(Someday, I'll track down all the places that
are generating errors but not recording messages. ;-/
Thanks to: Jaakko Heinonen
occurred with large read-ahead requests. This only affected
formats that incorrectly make large requests (ZIP did this until
recently) or with block sizes over 32k.
When reading the bodies of Zip archive entries, request a minimum of 1
byte, rather than a minimum of the full entry size. This is faster
(since it does not force the decompression layer to combine reads) and
works around a bug in the "none" decompression handler (which I'm
testing a separate fix for now). I've also renamed "bytes_read" to
"bytes_avail" in several places to more accurately reflect that the
value returned from (a->compression_read_ahead) is the number of bytes
available, not necessarily the number of bytes requested.
(padding) entries, extract inode value from PX entry, recognize SP and
ST (start/end of SUSP extensions).
I don't enforce SP yet, as I've seen CDROMs which use Rockridge
extensions but don't have the SP record (which is officially
required).
The ISO9660 support is now mature enough to extract FreeBSD
distribution CDROMs created with mkisofs.
Only supports "deflate" and "none" compression for now.
Also, add a few clarifications to the archive_read.3 manpage as
requested by William Dean DeVries.
for libarchive error messages. Mostly, this
avoids a portability headache related to
copying va_list arguments (some FreeBSD 5
platforms require va_copy; FreeBSD 4 doesn't
support va_copy at all). It also dramatically reduces the
size of libarchive for embedded applications:
a minimal "untar" program using libarchive can now be
under 64k statically linked (as opposed to ~100k
using library *printf() functions).
MFC after: 14 days
* Reference-count the directory data so that
we don't leak memory.
* Correctly step through the directory records
(skipping unrecognized extensions)
* Use better defaults for file modes
* Sort directory entries by offset of the end of the file
rather than the beginning of the file. This fixes a
lot of "out-of-order" problems with zero-length files,
in particular.
* Style fixes, remove some debug code, add some error messages.
This seems to be able to extract a TOC and extract files from
the couple of ISO images I've tested it with.
Treat this as experimental proof-of-concept code for the
moment. There are still a bunch of debug messages (there
are a few oddities in ISO9660 that I haven't yet figured
out how to handle), a lot of bugs to be addressed (this
code leaks memory very badly), and a lot of missing features (no
Rockridge support, in particular). I'd appreciate
feedback from anyone who understands ISO9660 format
better than I do. ;-)
Suggested by: Robert Watson
the regular ustar entry. The old code sometimes created
a too-long name that overflowed the ustar fields and triggered
an internal assertion failure. This version should be more
robust.
Thanks to: Michal Listos
Fixes: bin/74385
MFC after: 15 days
ustar fields. Later, we're going to permit numeric extensions
for these fields, so we can support large values here. In particular,
this allows GNU tar to correctly extract such entries even
though it doesn't support the pax extended attributes.
Note: r1.18 and r1.17.2.1 of this file allowed similar treatment
of the uid/gid fields.
Thanks to: Ben Mesander
improves the recognition of hardlink entries
with/without bodies (which is implemented through
a look-ahead that uses the bid function).
MFC after: 7 days
device numbers. In particular, this should fix
a bug where archiving a device node with a very
large minor number would sometimes overflow and
corrupt the major number.
Thanks to: Ben Mesander
MFC after: 7 days
regular 'ustar' entry, use narrow-character version,
not wide-character version, as the ustar entry always
uses the narrow-character filename.
Thanks to: Michal Listos
Inspired by, but doesn't fix: bin/74385
the the pax attributes, I shouldn't try using the public
API for finishing out the attribute entry, either.
This also removes some old dubious state manipulations.
because the code was using the external API
(archive_write_data) and assuming internal
error-return conventions. Use the internal
API for writing data.
Thanks to: Joe Marcus Clarke
file. In particular, this allows bsdtar to append (-r) to
an empty file.
Thanks to: Ryan Sommers
While I'm here, straighten out a misleading comment about GNU-compatible
sparse file handling.
probably means it also requires a .so version
bump. Defer it until I finish some related
work on cleaning up error returns throughout
the library.
Thanks to: Conrad J. Sabatier
1. The correct cutoff for large uid/gid handling is 1<<18, not 1<<20.
2. Limit the uid/gid in the 'x' extension header (where numeric extensions
are not permitted) to 1<<18, but use the correct value in the regular
header (where numeric extensions are permitted).
Thanks to: Dan Nelson
MFC after: 3 days
denote a directory. Unfortunately, in the presence of GNU or POSIX
extensions, this code was checking the truncated filename stored in the
regular header rather than the full filename stored in the extended
attribute. As a result, long filenames with '/' in just the right
position would trigger this check and be erroneously marked as
directories. Move the check so it only considers the full filename.
Note: the check can't simply be disabled for archives that contain
these extensions because there are some very broken archivers out
there.
Thanks to: Will Froning
MFC after: 3 days
.h files. This simplifies the Makefile here a bit and makes it behave
better in a couple of situations. While I'm here, clean up some comments
and try to improve the organization a bit.
Thanks to: Ruslan Ermilov (The Marvelous Makefile Guru)
This should provide a big performance boost for folks using NIS or LDAP.
MFC after: 3 days
Thanks to: Jun Kuriyama (for reminding me that this was still on my TODO list)
This closes a security hole. Otherwise, libarchive will happily
extract into directories to which it lacks write permissions by
resetting the permissions during the extract.
Thanks to: Kris Kennaway
simple errx() function.
Improve behavior when bzlib/zlib are missing by detecting and
issuing an error message on attempts to read gzip/bzip2 compressed
archives.
and close it) and "finish" (destroy the object) functions. For backwards
compat and simplicity, have "finish" invoke "close" transparently if needed.
This allows clients to close the archive and check end-of-operation
statistics before destroying the object.
is present for FreeBSD. If you "make distfile" on FreeBSD, you will
soon have a tar.gz file suitable for deploying to other systems
(complete with the expected "configure" script, etc). This latter
relies (at least for now) on the GNU auto??? tools. (I like autoconf
okay, but someday I hope to write a custom Makefile.in and dispense
with automake, which is somewhat odious.)
As part of this, I've cleaned up some of the conditional
compilation options, added make-foo to construct archive.h dynamically
(it now contains some version constants), and added some useful
informational files.
mode bits when setting permissions from ACL data.
Thanks to: David Gilbert for first reporting this and
Jimmy Olgeni for noticing that it only occurred on
ACL-enabled filesystems.
code on the existence of the relevant libraries (actually,
the existence of the include files).
This will allow the library to be easily ported to systems
that don't have these libraries. (Of course, it also means
that clients using the library on such systems will not be
able to take advantage of the automatic compression format
detection.)
in the regular ustar header that are overridden by the pax
extended attributes. As a result, it makes perfect sense to
use numeric extensions in the regular ustar header so that readers
that don't understand pax extensions but do understand some other
extensions can still get useful information out of it.
This is especially important for filesizes, as the failure to
read a file size correctly can get the reader out of sync.
This commit introduces a "non-strict" option into the internal
function to format a ustar header. In non-strict mode, the formatter
will use longer octal values (overwriting terminators) or binary
("base-256") values as needed to ensure that large file sizes,
negative mtimes, etc, have the correct values stored in the regular
ustar header.
archive_version: Returns a text string, e.g., "libarchive 1.00.000"
archive_api_version: Returns the SHLIB major version
archive_api_feature: Returns a feature number useful for answering
questions such as "Is this recent enough to do XXX?"
The last two also have macros defined in archive.h, so you can compare
the compile-time and run-time environments. (In particular, you can
compare ARCHIVE_API_VERSION to archive_api_version() to detect library
version mismatches.)
With these in hand, it will soon be time to turn on the
shared-library version of libarchive... stay tuned.
Thanks to: David O'Brien for pointing this out.
Also, add in a few additional portability tweaks and make a few
more things conditional on features (HAVE_XXXX macros) rather
than platform.
In particular, this means we can now correctly read gtar archives that
contain timestamps prior to the start of the Epoch.
Also, make the code in this area more portable. ANSI C99 headers are
not yet ubiquitous (for example, FreeBSD 4 still lacks them), so be
prepared for systems that don't have the INT64_MAX, INT64_MIN, and
UINT64_MAX macros. This version still requires int64_t and uint64_t be
defined (which can be done in archive_platform.h if necessary), but
doesn't require them to be exactly 64 bits.
code. <whew!> This version handles all of the following edge cases:
* Restoring explicit dirs with 000 permissions (star fails this test)
* Restore of implicit or explicit dirs when umask=777
(gtar and star both fail this test)
* Restoring dir paths containing "." and ".." components
This version initially creates all dirs with permission 700 (ignoring
umask), then does a post-extract "fixup" pass to set the correct
permissions (which may or may not depend on umask, depending on the
restore flags and whether it's an explicit or implicit dir).
Permissions are restored depth-first so that permissions within
non-writable dirs can be correctly restored. (The depth-sorting does
correctly account for dirs with ".." components.)
applied to their permissions. Just calculate the
default dir mode once and use it consistently, rather than
trying to remember to calculate it everywhere it's needed.
* Rename some variables/functions/etc to try to make things clearer.
* Add separate flags to control fflag/acl restore
* Collect metadata restore into a single function for clarity
* Propagate errors in metadata restore back out to the client
* Fix some places where errors were being returned when they
shouldn't and vice-versa
* Modes are now always restored; ARCHIVE_EXTRACT_PERM just controls
whether or not umask is obeyed.
* Restore suid/sgid bits only if user/group matches archive
* Cache the last stat results to try to reduce the number of stat calls
archive_entry.
Update the Makefile MLINKS and manpage to bring it up-to-date with
the current status of archive_entry. At least the manpage actually
lists all of the functions now, even if it doesn't really yet explain
them all.
Mostly, these were being used correctly even though a lot of
variables and function names were mis-named.
In the process, I found and fixed a couple of latent bugs and
added a guard against adding an archive to itself.
a/././b/../b/../c/./../d/e/f now work correctly. And yes, a/b and a/c
both get created in this example; if you want, you can create an
entire dir heirarchy from a tar archive with only one entry.
More tweaks to umask support: umasks are now obeyed for all objects,
not just directories; the umask used is now the one in effect at the
corresponding call to archive_read_extract(), so clients that want to
tinker with umask during extract should get the expected behavior.
umask in effect when the archive is closed
* Correct a typo that broke implicit dir creation for non-directories.
Thanks to: Garret A Wollman for pointing out my umask oversight
read_extract_dir (which creates directories in the archive). This
brings a number of advantages:
* FINALLY fix the problems creating dirs ending in "/." <sigh>
* Missing parent dirs now get created securely, just like explicit dirs.
(Created 0700 initially, then edited to 0755 at end of extraction.)
* Eliminate some duplicate code and some weird special cases.
While I'm cleaning, inline the regular-file creation code as well.
This change also pointed out one API deficiency: the
archive_read_data_into_XXX functions were originally defined to return
the total bytes read. This is, of course, ambiguous when dealing with
non-contiguous files. Change it to just return a status value.
file already exists on disk.
Pointed out by: www/resin3 port (whose distfile contains the same file
twice with different permissions and relies on the permissions associated
with the second instance)
Thanks again to: Kris Kennaway
* Restore directories with 0700 permissions initially,
then use the fixup pass to correct the permissions
* Trim trailing "/" and "/." in mkdirpath()
Suggested by: Garrett Wollman
distinguish files from dirs (trailing '/' indicated a dir). Since
POSIX.1-1987, this convention is no longer necessary. However, there
are current tar programs that pretend to write POSIX-compliant
archives, yet store directories as "regular files", relying on this
old filename convention to save them. <sigh> So, move the check for
this old convention so it applies to all tar archives, not just those
identified as "old."
Pointed out by: Broken distfile for audio/faad port
it sees a truncated input the first time it gets called.
(In particular, files shorter than 512 bytes cannot be tar archives.)
This allows the top-level archive_read_next_header code to
generate a proper error message for unrecognized file types.
Pointed out by: numerous ports that expect tar to extract non-tar files ;-(
Thanks to: Kris Kennaway
version called the higher-level archive_read_data and
archive_read_data_skip functions, which screwed up state management of
those functions. This bit of mis-design has existed for a long time,
but became a serious issue with the recent changes to the
archive_read_data APIs, which added more internal state to the
high-level archive_read_data function. Most common symptom was a
failure to correctly read 'L' entries (long filename) from GNU-style
archives, causing the message ": Can't open: No such file or
directory" with an empty filename.
Pointed out by: Numerous port build failures
Thanks to: Kris Kennaway
that as end-of-archive. Otherwise, a short read at this point
generates an error. This accomodates broken tar writers (such as the
one apparently in use at AT&T Labs) that don't even write a single
end-of-archive block.
Note that both star and pdtar behave this way as well.
In contrast, gtar doesn't complain in either case, and as a
result, will generate no warning for a lot of trashed archives.
Pointed out by: shells/ksh93 port (Thanks to Kris Kennaway)
push extract data down into archive_read_extract.c and out
of the library-global archive_private.h; push dir-specific
mode/time fixup down into dir restore function; now that the
fixup list is file-local, I can use somewhat more natural
naming.
Oh, yeah, update a bunch of comments to match current reality.
* New read_data_block is both sparse-file aware and uses zero-copy semantics
* Push read_data_block down into specific formats (opens door to
various encoded entry bodies, such as zip or gtar -S)
* Reimplement read_data, read_data_skip, read_data_into_fd in terms
of new read_data_block.
* Update documentation
It's unfortunate that I couldn't just call the new interface
archive_read_data, but didn't want to upset the API that much.
there's no need to enable support for it separately
from 'tar.' (The call to enable gnutar support is
now just an alias for the tar support, left in to
avoid API breakage.)
certain flags set (e.g., schg or uappend) would fail because the flags
were restored before the hardlink was created.
To address this, I've generalized the existing machinery for deferring
directory timestamp/mode restoration and used it to defer the
restoration of highly-restrictive flags to the end of the extraction,
after any links have been created.
Pointed out by: Pawel Jakub Dawidek (pjd@)
character, as some tar implementations incorrectly include a '/' with
the prefix.
Thanks to: Divacky Roman for the UnixWare 7 tarfile that
demonstrated this issue.
code available at tuhs.org, and found out that my chronology
is a bit off. In particular, Seventh Edition already used
the "linkflag" and "linkname" fields. Also, it appears that
there was no tar in Sixth Edition, contrary to what an earlier
tar.1 manpage claimed.
A few mdoc fixes also crept in here.
the size field for a hardlink entry. Specifically, ensure that
we do obey the size field for archives that we know are pax interchange
format archives, as required by POSIX.
Also, clarify the comment explaining why this is necessary and explain
the (very unusual) conditions under which it might fail.
a non-zero size but no body, some write a non-zero size and include
a body. To distinguish these cases, look for a valid tar header immediately
following a hardlink header with non-zero size.
-U flag to bsdtar. Essentially, this option breaks existing hard
links. According to SUSv2, tar is supposed to overwrite existing
files on extract by default which, in particular, preserves
existing hard links. Note that this is yet another bug in gtar; it
appears to always break existing links. (Maybe gtar's -U is broken?)
I'm unsure about how to handle this for other file types; the current
code always unlinks first unless the NO_OVERWRITE flag is specified.
I've commented this issue liberally and will come back to it later.
archive formats supported by libarchive, with some information about
the relative strengths and weaknesses of each format and notes about
issues with libarchive's support for those formats.
This page should make it unnecessary to list all of the libarchive
formats in the manpage of each program that uses libarchive.
Such programs can simply refer to libarchive-formats(5).
* little-endian old-style binary cpio archives
* big-endian old-style binary cpio archives
* SVR4 portable archives without CRC
* SVR4 portable archives with CRC
Note that I don't yet verify the CRC for the last one, and I'm
not quite certain I'm correctly parsing device numbers.
The new fflags support in archive_entry supports Linux and FreeBSD
file flags and is a bit more gracious about unrecognized flag names
than strtofflags(3). This involves some minor API breakage.
The default tar format ("restricted pax") now enables pax extensions
when archiving files that have flags. In particular, copying dir
heirarchies with 'bsdtar cf - -C src . | bsdtar xpf - -C dest' now
preserves file flags. (Note the "p" on extract!)
While I'm here, fill in some additional explanation in the
archive_entry.3 manpage, fill in some missing MLINKS, mark some
overlooked internal functions 'static', and make a few minor style
fixes.
The original might have pointers to user-specified strings;
copying the string (instead of just the pointer) protects against
the client re-using their own buffers.
I'm trying hard to avoid dumping all of the 'set' string functions
in favor of slower, but more predictable 'copy' semantics.
High-resolution mtime/ctime/atime is not POSIX-standard, so hide
set/get of high-resolution time fields behind easily-mutable macros.
That makes it easier to change how those fields are accessed.
Earlier versions of FreeBSD don't support ACLs.
Note that the ACL support code in archive_entry is standalone code and
unaffected by this. (In particular, it should be possible to
manipulate archives containing ACLs even if the ACLs cannot be
restored on the current system.)
* Re-use a single buffer for shar output formatting rather
than hammering the heap. (archive_write_set_format_shar.c)
* Fix a handful of minor memory leaks and clean up some of the
memory-management code.
try to set ACLs even if fflag restore fails, first cut at reading
Solaris tar ACLs
Code improvement: merge gnu tar read support into main tar reader;
this eliminates a lot of duplicate code and generalizes the tar
reader to handle formats with GNU-like extensions.
Style: Makefile cleanup, eliminate 'dmalloc' references, remove 'tartype'
from archive_entry (this makes archive_entry more format-agnostic)
Thanks to: David Magda for providing Solaris tar test files
* ACL storage is no longer erased before a group of entries are added.
* ACL text creation no longer tries to skip over non-existent text.
* UTF8 encoder no longer blows up on invalid wide characters.
* Fixed ACL state management for default ACLs.
Also, publicize function for obtaining text-format ACL in various
formats. The interface is now extensible through a "flags" argument
that allows you to select a variant format.
with 'star' ACL handling, though there's still a
bit more work needed in this area.
Added 'write_open_fd' and 'read_open_fd' to simplify, e.g.,
tar's u and r modes. Eliminated old 'write_open_file_position'
as a bad idea. (It required closing/reopening files to
do updates, which led to unpleasant implications.)
Various other minor fixes, API tweaks, etc.
* Disabled shared-library building, as some API breakage is
still likely. (I didn't realize it was turned on by default.) If
you have an existing /usr/lib/libarchive.so.2, I recommend deleting it.
* Pax interchange format now correctly stores and reads UTF8
for extended attributes. In particular, pax format can portably
handle arbitrarily long pathnames containing arbitrary characters.
* Library compiles cleanly at -O2, -O3, and WARNS=6 on all
FreeBSD-CURRENT platforms.
* Minor portability improvements inspired by Juergen Lock
and Greg Lewis. (Less reliance on stdint.h, isolating of
various portability-challenged constructs.)
* archive_entry transparently converts multi-byte <-> wide character
strings, allowing clients and format handlers to deal with either
one, as appropriate.
* Support for reading 'L' and 'K' entries in standard tar archives
for star compatibility.
* Recognize (but don't yet handle) ACL entries from Solaris tar.
* Pushed format-specific data for format readers down into
format-specific storage and out of library-global storage. This
should make it easier to maintain individual formats without mucking
with the core library management.
* Documentation updates to track the above changes.
* Updates to tar.5 to correct a few mistakes and add some additional
information about GNU tar and Solaris tar formats.
Notes:
* The basic 'tar' reader is getting more general; there's not much
point in keeping the 'gnutar' reader separate. Merging the two
would lose a bunch of duplicate code.
* The libc ACL support is looking increasingly inadequate for my needs
here. I might need to assemble some fairly significant code for
parsing and building ACLs. <sigh>
Portability: Thanks to Juergen Lock, libarchive now compiles cleanly
on Linux. Along the way, I cleaned up a lot of error return codes and
reorganized some code to simplify conditional compilation of certain
sections.
Bug fixes:
* pax format now actually stores filenames that are 101-154
characters long.
* pax format now allows newline characters in extended attributes
(this fixes a long-standing bug in ACL handling)
* mtime/atime are now restored for directories
* directory list is now sorted prior to fix-up to permit
correct restore of non-writable dir heirarchies
This doesn't yet address the issue of selective restore
of hardlinked files. With cpio format, it's possible to correctly
restore any linked file; the API doesn't yet fully support this.
(There's no way for the library to inform a client whether or not
there's a file body associated with this entry. The assumption
right now is that "hardlink" entries have no file body.)
the size in the archive_entry object to zero if that format doesn't
store a body for that file type. This allows the client to determine
whether or not it should feed the file body to the archive. In
particular, cpio stores the file body for hardlinks, tar and shar
don't. With this change, bsdtar now correctly archives hardlinks in all
supported formats.
While I'm here, make shar output be more aggressive about creating directories.
Before this, commands such as:
bsdtar -cv -F shar some/explicit/path/to/a/file
wouldn't create the directory. Some simple logic to remember the last
directory creation helps reduce unnecessary mkdirs here.
At this point, I think the only flaw in libarchive's cpio support is
the failure to recognize hardlinks when reading.
While I'm here, fix a bug in reading filenames from
cpio files. (Copy should count the length of the name,
not the number of bytes available for input.)
Unfortunately, the stock zlib.h is not:
line 885: 'err' parameter shadows global 'err' definition from <err.h>
Back the WARNS level down to 3 to accomodate borked zlib.h.
What it is:
A library for reading and writing various streaming archive
formats, especially tar and cpio. Being a library, it should
be easy to incorporate into pkg_* tools, sysinstall, and any
other place that needs to read or write such archives.
Features:
* Full automatic detection of both compression and archive format.
* Extensible internal architecture to make it easy to add new formats.
* Support for "pax interchange format," a new POSIX-standard tar format
that eliminates essentially all of the restrictions of historic formats.
* BSD license
Thanks to: jkh for pushing me to start this work, gordon for
encouraging me to commit it, bde for answering endless style
questions, and many others for feedback and encouragement.
Status: Pretty good overall, though there are still a few rough edges and
the library could always use more testing. Feedback eagerly solicited.
