This significantly changes how end-of-data markers are parsed.
In particular, the spec allows the end-of-data marker to have either
32-bit or 64-bit size values, and there is basically no indication which
is being used. (The spec mentions "Zip64 mode" in many places, but there
is no definitive way for a Zip reader to know whether the writer is
using this mode or not. My mis-reading of another part of the spec
caused me to believe that the Zip64 Extra Data field was such a marker,
but I've been patiently corrected. ;-)
So a Zip reader just has to guess: Try every possible end-of-data marker
format and accept it if any of the four possible forms is correct. In
libarchive's case, this required some non-trivial additional refactoring
to ensure that the CRC32, compressed size, and uncompressed size
statistics are always updated _before_ we need to look for an
end-of-data marker.
This generally follows the strategy outlined by Mark Adler for his
`sunzip` streaming unzip implementation.
While testing this, I played with pmqs/zipdetails which pointed out a
discrepancy in how libarchive writes the `UT` extra field. I folded a
fix for that in here as well.
Resolves#1834
TODO: It would be nice to augment the test suite with some static files
created by Java's implementation to verify that we can read those when
they hold entries of +/- 4GiB. The existing
`test_write_format_zip_large` uses an ad hoc RLE encoding trick to
exercise writing and reading back multi-gigabyte entries. I wonder if
that could be generalized to support deflate-compressed Zip data stored
in test files?
Tim Kientzle
8acb738db6