23 Commits
Author SHA1 Message Date
Sebastian Freundt 96e57afe1f Allow empty files (after filters) in bsdcat command.
This changeset fixes an issue with empty compressed files, i.e.
files that after inflating are of size 0:  bsdcat would report
unrecognized archive format for those because the raw reader is
unable to handle files of zero length.
2014-08-13 07:05:12 +00:00
Sebastian Freundt 850c3c8d37 Provide regression test case for bsdcat on empty compressed files 2014-08-12 15:26:24 +00:00
Sebastian Freundt 1dfc9a2f67 Check for legal HTTP but illegal unix filenames in WARCs archives.
This changeset will refuse to extract WARCs that contain filenames
a la http://example.com/implicit/content/

There is a todo note in archive_read_support_format_warc.c discussing
possible archive options to extract filenames like those either by
explicit user input or by some sort of heuristic as used in wget
for example.
2014-06-10 13:42:41 +00:00
Sebastian Freundt 13ff3063f4 Set archive format, and archive format name.
This changeset adheres to the previously imported read test.
The archive format is hard-set to ARCHIVE_FORMAT_WARC, while the
format name is the stringified WARC/x.y version designator, which
for performance reasons will be cached between calls to the header
reader _warc_rdhdr().
2014-06-10 13:03:31 +00:00
Sebastian Freundt c81efb9912 Tests, provide known-good archive read test. 2014-06-10 12:53:30 +00:00
Sebastian Freundt adc91b6969 Hygiene, set unconsumed slot exactly once for each branch
in _warc_read().  Also kick __archive_read_consume() because the writer
will consume the bytes for us.  So for the EOF case, set unconsumed to 0,
for the non-EOF case set unconsumed to the minimum of the number of bytes
read and the content length.
2014-06-10 12:53:19 +00:00
Sebastian Freundt 75b78d5c13 Hygiene, always use xmemmem() because memmem() is a GNU extension 2014-06-10 12:53:19 +00:00
Sebastian Freundt dd1c7eebb5 Hygiene, use FALLTHROUGH instead of lint's @fallthrough@ 2014-06-10 11:20:08 +00:00
Sebastian Freundt b3cc3bcd93 Hygiene, trust __archive_read_ahead() and kick superfluous check.
Heeding Tim's advice, a non-NULL from __archive_read_ahead() is
guaranteed to be of at least the minimum size, therefore no need to
check for this condition again.
2014-06-10 11:15:42 +00:00
Sebastian Freundt 6e70844f13 Hygiene, protect against NULL filenames in _warc_header()
Return ARCHIVE_WARN immediately.
2014-06-10 11:01:06 +00:00
Sebastian Freundt e251bcfa94 Fix, an empty WARC archive needs a bit more than 256 bytes. 2014-06-10 11:00:20 +00:00
Sebastian Freundt beb4ff6c7a Tests, heed Tim's advice and emit ARCHIVE_FAILED for entries ...
that cannot be stored (natively) in WARC format.
2014-06-10 10:42:06 +00:00
Sebastian Freundt 09100bec9f Hygiene, clean up xmemmem() code a little, use xor sums.
Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
2014-05-23 08:36:58 +00:00
Sebastian Freundt 84114d7db7 Store and read back mtimes through Last-Modified custom header
Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
2014-05-23 08:19:39 +00:00
Sebastian Freundt cb3e79b342 Fix, actually consume data between calls to _warc_read() 2014-05-21 15:18:52 +00:00
Sebastian Freundt 75d16bd095 Obey gcc warnings,
in particular: Don't compare integers of different signedness,
always initialise all members of a struct explicitly.
2014-05-21 14:32:05 +00:00
Sebastian Freundt 3bb9e31050 fix, WARC files urgently need the filesize to be known when the header is written 2014-05-21 14:11:32 +00:00
Sebastian Freundt fa28baa097 fix, request just the bare minimum for a WARC header
Moreover, assume a response of less than the bare minimum header
length to be the archive's EOF.
2014-05-21 14:10:48 +00:00
Sebastian Freundt faa12ea4c2 fix, never write more bytes in _warc_data() than ...
previously announce in _warc_header().

The test suite (as is) is one offender.  It populates a 9-byte string, mimicking an IFREG file
but by the time the header makes it into the archive, the size changes from 0 to 9.
2014-05-21 12:32:50 +00:00
Sebastian Freundt af78448244 Provide WARC read/write tests 2014-05-21 12:32:27 +00:00
Sebastian Freundt 3937c7050d Implant WARC support in tar's get_format_code() 2014-05-21 11:24:10 +00:00
Sebastian Freundt 9693801580 Provide ISO 28500:2009 reader (aka warc, aka web archive) 2014-05-21 11:20:15 +00:00
Sebastian Freundt 667961d7ae Provide ISO 28500:2009 writer (aka warc, aka web archive) 2014-05-21 08:39:40 +00:00