WARC records require a known Content-Length. Reject regular file
entries without a valid size instead of treating them as zero.
Avoid casting archive_entry_size() through size_t before storing it in
the uint64_t WARC Content-Length field, since that can truncate large
sizes on 32-bit platforms.
If enough bytes have to be skipped, a signed integer overflow could
occur (most realistically on 32 bit systems). Use an unsigned type,
which could still overflow but has only negative impact on diagnostic
warning message.
Follow up of 02fa05675f but this time for
function find_newc_header.
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
archive_write_pax_header() set hardlink entries to size zero for the
restricted pax format, then immediately set all hardlink entries to size
zero again.
Remove the narrower check because it is fully subsumed by the following
hardlink check.
No functional change intended.
header_common() parsed the tar mode field once when setting the file
type and again when setting permissions. Cache the parsed value and reuse
it for both setters.
archive_entry_set_filetype() and archive_entry_set_perm() already mask
the relevant bits, so passing the full header mode preserves behavior.
The permission-set guard is unchanged.
No functional change intended.
This is the fix for....
Update the common archive_read_data_into_fd() to truncate
output to the actual entry size from the most recent
archive_read_header result.
This addresses a hole in handling of certain subtly-malformed archives
that can have more data than declared in the header. Before this,
bsdtar explicitly truncates the entry data to the actual declared size
when extracting to a file, but did not do so when extracting to
stdout.
This is the test for....
Update the common archive_read_data_into_fd() to truncate
output to the actual entry size from the most recent
archive_read_header result.
This addresses a hole in handling of certain subtly-malformed archives
that can have more data than declared in the header. Before this,
bsdtar explicitly truncates the entry data to the actual declared size
when extracting to a file, but did not do so when extracting to
stdout.
Reverse an old experiment: When the 32-bit size field is provided, it
now must match the actual size exactly. We no longer permit it to
only match the least-significant 32 bits of the actual size.
--begin-history--
When developing the earliest versions of libarchive's Zip support, I
experimented with the idea of supporting greater-than-4GiB entries in
32-bit Zip archives. This proved to be technically feasible as long
as the _compressed_ data was small enough to keep the resulting
archive under 4GiB. But this also required us to interpret
the 32-bit uncompressed size field as representing only the
least-significant 32 bits of the size.
--end-history--
Test: If the uncompressed size stored in the archive differs
from the actual uncompressed size by a multiple of 2^32,
we should report some kind of error.
This currently goes undiagnosed because of an old attempt
to support larger-than-4GiB entries in plain 32-bit Zip
archives without necessarily requiring Zip64 extensions.
While this is technically feasible, it requires treating
the 32-bit size field ambiguously, which can lead to size
confusion from readers.
ArchiveProperties entries have a property ID, a size, and payload.
The reader parsed the ID and size, but did not consume the payload.
Payload bytes could then be read as the next property ID, which
misaligned the rest of header parsing.
Skip the payload in chunks so parsing stays aligned with 7-Zip/7zz.
If the raw parser ever treats a basically endless stream which
eventually overflows INT64_MAX, fail instead of overflowing
offset + size calculation, leading to subsequent issues in callers.
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
If an entry declares an uncompressed size, we should not return any
data beyond that point. This doesn't impact bsdtar and friends since
they independently truncate to the declared entry size, but other
clients may not be so careful.
Apparently discovered (but not reported) by @bikini
Reported by @dkgkdfg65
Tests for...
If an entry declares an uncompressed size, we should not return any
data beyond that point. This doesn't impact bsdtar and friends since
they independently truncate to the declared entry size, but other
clients may not be so careful.
Apparently discovered (but not reported) by @bikini
Reported by @dkgkdfg65
Add a 7z test archive with a non-empty ArchiveProperties payload.
The archive has one empty file and an ArchiveProperties entry with a
1-byte payload. It is accepted by 7-Zip/7zz.
This covers parsing when ArchiveProperties has payload bytes.
Avoid adding the encryption overhead directly to entry sizes when deciding
whether Zip64 is needed or when updating the stored compressed size.
For the Zip64 decision, compare against ZIP_4GB_MAX - additional_size. For
stored encrypted entries, use archive_ckd_add_i64() so the size update and
overflow check happen together.
Covers basic E8 translations, yet it lacks the handling of negative
values because this would require large archives.
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
The cab specification states that E8 translation shall be performed only
within the first 1 GB range of an uncompressed file, i.e. for the first
0x8000 CFDATA entries in a folder.
Take this limit properly into account.
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
Avoid allocating and filling metadata bitmaps when the 7z header says
that every file has the corresponding value.
Sparse metadata still uses the existing bitmap path.
The Windows disk reader supports wildcards by converting a path like
"foo/*" into a single tree rooted at "foo/". This does not play nicely
with symlink mode `H`, which treats the first item in each tree as a
dereferenced link (L) and all subsequent items as physical links (P). On
Windows, we take the first _match_ in L mode and all subsequent
descendants and matches in P mode.
This creates a behavioral difference between `bsdtar -c -H *` and
`bsdtar -c -H one two three`.
This commit changes how we handle H->L/P transitions when doing wildcard
matches. The transition from H to P is now done when we descend into our
first directory, and we switch back to H mode when we ascend out of
it[^1].
Closes#3222
[^1]: This does not matter in practice because tree is breadth-first
rather than depth-first, but I've done it in the interest of
completeness and not leaving ourselves caltrops to step on later.
The memory allocation in archive_read_set_callback_data2, which is
called by archive_read_set_callback_data, could fail.
Check result and properly handle failure.
Resolves#3210.
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
Flatten nested allAreDefined checks for attribute and timestamp bools.
Also initialize empty-stream entries as directories by default before
applying the optional EmptyFile bitmap.
No archive behavior change is intended.
Both supported BCJ2 layout branches store the chosen unpack size in
remaining.
main_stream_bytes_remaining is set from remaining immediately after
the branch, so the earlier assignments are redundant.
On LoongArch64 with 16KB pages, Linux tmpfs often uses Transparent
Huge Pages (THP) sized at 32MiB. When a test writes data followed
by a 32MiB hole, the kernel may allocate a single huge page folio
to back the entire range in RAM. Because the range is physically
backed by memory, SEEK_HOLE correctly (from the kernel's perspective)
identifies the range as DATA, failing the test's assertion. Increasing
MIN_HOLE to 64MiB ensures the hole exceeds the huge page size on
loongarch64, preserving the sparse nature of the file for the test.
Signed-off-by: Pluto Yang <yangyj.ee@gmail.com>
Dustin L. Howett
datauwu
Tim Kientzle
Tobias Stoeckmann
datauwu
Kaif Khan
Pluto Yang