3.8.8 ("make ARCHIVE_FATAL sticky in data-reading entry points",
commit e1f890dc) made archive_seek_data() set
a->archive.state = ARCHIVE_STATE_FATAL whenever the call returned
ARCHIVE_FATAL. That is correct for genuine I/O / parse failures
returned by a format's seek_data() implementation, but it also
poisoned the archive in two cases where seeking is simply
unsupported for the current format:
1. The seek_data == NULL branch in archive_seek_data() itself
(no format_seek_data_block registered: ustar/tar, the
streaming ZIP reader, ...).
2. rar5_seek_data() in archive_read_support_format_rar5.c, which
unconditionally returns ARCHIVE_FATAL because RAR5 is a
streaming unpacker. This was latent before 3.8.8 because
the read core did not make FATAL sticky; the stickiness
change exposed it.
Both are capability gaps, not stream corruption. ARCHIVE_FATAL
must always be sticky (it means the archive is irrecoverably
damaged), so both sites now return ARCHIVE_FAILED with an error
string instead. ARCHIVE_FAILED is recoverable: capability probes
such as
if (archive_seek_data(a, 0, SEEK_CUR) >= 0) ...
learn that seeking is unavailable while the archive remains
usable for subsequent reads. The sticky-on-genuine-FATAL part of
the original 3.8.8 change is preserved for actual ARCHIVE_FATAL
returns from a format's seek_data() implementation.
The seek_data == NULL branch also reports the condition better: a
format that never registers a seek hook is not illegal use of the
library, so the error is now ARCHIVE_ERRNO_MISC / "Cannot seek
data with this format" instead of ARCHIVE_ERRNO_PROGRAMMER /
"Internal error: No format_seek_data_block function registered".
The rar5 change also matches how the RAR4 reader already reports
the same situation (archive_read_support_format_rar.c:1339
returns ARCHIVE_FAILED for compressed RAR files).
This restores 3.8.7 behaviour for clients that probe seekability
before reading. VLC's skins2 stream extractor
(modules/stream_extractor/archive.c, archive_seek_subentry())
probes with archive_seek_data(a, 0, SEEK_CUR); on a non-seekable
format the poisoned state killed all later archive_read_data()
calls, breaking ZIP-packed .vlt skins on distributions shipping
libarchive 3.8.8.
Tests:
- test_archive_seek_data_unsupported: covers the seek_data==NULL
branch using a ustar archive.
- test_read_format_rar5_seek_data_unsupported: covers the
rar5_seek_data() branch using an existing RAR5 fixture, and
verifies the entry content with verify_data() after the probe.
Both fail on master and pass with this change.
Files which cannot fit into a stream should never return success, which
can happen due to truncation on 32 bit systems.
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
Add a test archive that reuses one read buffer for every 512-byte block.
Reading the sparse extension overwrites the main GNU header. Check that
mode, uid, gid, and mtime still come from the original header.
Add a per-format "align" option to the pax writer that pads each
regular file's pax extended header (creating one if necessary) so the
file data that follows begins on a multiple of the requested number of
bytes within the uncompressed archive stream. The value must be a
power of two and a multiple of the 512-byte tar block.
The alignment is applied to the stream feeding into any compression
filter, so the decompressed archive is aligned regardless of whether
the archive is compressed. This lets tools share file contents out of
an (uncompressed copy of an) archive using reflinks / copy_file_range(2)
instead of doing a full data copy on extraction.
The padding is carried in an ignorable "LIBARCHIVE.pad" pax record, so
existing readers extract the archive unchanged. Files shorter than the
alignment are left unpadded since they cannot share a whole block.
Add a test covering uncompressed and gzip-compressed archives and
document the option.
Signed-off-by: Daan De Meyer <daan@amutable.com>
A symlink entry named TRAILER!!! with a body large enough to grow the
read filter's copy buffer used to leave the cached name pointer stale
before the end-of-archive comparison. Read a static newc archive in
small blocks and assert the trailer is still detected (ARCHIVE_EOF).
The fix landed in 23b54da; this only adds coverage.
If a pre-tree is larger than 10 bit, libarchive fails to read the data.
According to spec, up to 16 bit are valid.
Example file provided by Duncan Horn. Thanks a lot!
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
Drives the best-effort converter (charset iconv cannot open) through
archive_strncpy_l with a length-delimited buffer: once with trailing
non-NUL bytes after the declared length and once with an exact-length
heap allocation with no trailing NUL. Fails under ASan before the fix.
Read a Solaris pax entry whose SUN.holesdata value ends right after its
final decimal offset (no trailing space), which is the shape of every
valid map. The 1536-byte file has data at [0,512) and [1024,1536) with a
hole between them, so archive_entry_sparse must report exactly those two
data blocks. This pins the parser's loop-termination behavior for the
common case where the run consumes all value_length bytes.
While investigating improvements to Zip's header reading,
I found an overlooked case in the read core: When a format
returned FAILED for a header read (indicating that the entry
could not be read but that the archive as a whole is still valid),
the core silently converted that to FATAL, ending the archive
at that point.
Two formats already returned FAILED for certain header issues:
Zip and LHA both indicated unreadable symlink entries in this way,
which should allow the rest of the archive to continue being read.
New tests here verify that we can now continue reading past
such entries, and that clients who ignore the FAILED result
get the same FATAL handling as other API misuse.
Drives archive_acl_from_text_nl() with exact-length, non-NUL-terminated
buffers whose first field is only whitespace, covering the empty-entry
and trailing-whitespace cases. Allow the test to include
archive_acl_private.h via __LIBARCHIVE_TEST.
Add a test for the PKWARE Unix extra field 0x000d.
The fixture stores a regular file with the 12-byte fixed metadata
prefix in both the local header and central directory. The test checks
that the ZIP reader exposes mtime, atime, uid, and gid metadata for
seekable and streaming reads.
This is the test for....
Update the common archive_read_data_into_fd() to truncate
output to the actual entry size from the most recent
archive_read_header result.
This addresses a hole in handling of certain subtly-malformed archives
that can have more data than declared in the header. Before this,
bsdtar explicitly truncates the entry data to the actual declared size
when extracting to a file, but did not do so when extracting to
stdout.
Test: If the uncompressed size stored in the archive differs
from the actual uncompressed size by a multiple of 2^32,
we should report some kind of error.
This currently goes undiagnosed because of an old attempt
to support larger-than-4GiB entries in plain 32-bit Zip
archives without necessarily requiring Zip64 extensions.
While this is technically feasible, it requires treating
the 32-bit size field ambiguously, which can lead to size
confusion from readers.
Tests for...
If an entry declares an uncompressed size, we should not return any
data beyond that point. This doesn't impact bsdtar and friends since
they independently truncate to the declared entry size, but other
clients may not be so careful.
Apparently discovered (but not reported) by @bikini
Reported by @dkgkdfg65
Add a 7z test archive with a non-empty ArchiveProperties payload.
The archive has one empty file and an ArchiveProperties entry with a
1-byte payload. It is accepted by 7-Zip/7zz.
This covers parsing when ArchiveProperties has payload bytes.
Covers basic E8 translations, yet it lacks the handling of negative
values because this would require large archives.
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
Add PE and ELF SFX fixtures for the SFX scanner boundary cases.
The PE fixture places an empty 7z archive at the final exact
32-byte scan window. The ELF fixture places ".data\0" at the final
valid section string-table position.
Register the fixtures for autotools distcheck.
Several ISO tests used a bz2 suffix even though they read .Z files
and verify ARCHIVE_FILTER_COMPRESS. Rename them to use a compress
suffix so the test names match the filter.
The write option setter checks used several tiny source files with the
same basic shape.
Keep the same DEFINE_TEST cases. Put the write-side option setter
checks in test_archive_write_set_options.c. Remove the extra source
files from the build lists.
The read option setter checks used several tiny source files with the
same basic shape.
Keep the same DEFINE_TEST cases. Put the read-side option setter
checks in test_archive_read_set_options.c. Remove the extra source
files from the build lists.
The close-twice read checks used three tiny source files.
Keep the same DEFINE_TEST cases. Put the fresh-object, fd-backed,
and filename-backed checks in one source file. Remove the two extra
files from the build lists.
Exercise atou64() through a TOC numeric element (<inode>) read via the
expat backend, which hands the parser its raw, non-NUL-terminated
character-data buffer. Mirrors test_read_format_xar_base64_oob; build
with ASan for the out-of-bounds coverage.
Malachi Moody
Dustin L. Howett
subotac
Tobias Stoeckmann
Dan Nelson
Jin
0xmrniko
datauwu
Dustin L. Howett
Tim Kientzle
Kaif Khan
Daan De Meyer
Tobias Stoeckmann
datauwu
nvmb3r
Reyad Attiyat