Friday, May 27, 2016

[yvldaeyx] Partial compression with clonable state

Store a compressed partial file in two pieces: one containing all the completed output blocks and one containing the compressed incomplete last block.

Devilish details, partially explored:

There does not seem to be enough rope in zlib to do this efficiently.  We either need a way to snapshot and restore the internal compression state to roll it back, or for the compression function to leave unconsumed input in the input buffer when it produces output.

If the the first feature, then we do some binary search at the next to last block for the minimum amount of input that produces the next to last output block.  I'm not sure where the output of the flush then goes.

We could serialize the final state and write it out as the uncompressed final block, but the unconsumed input serves the same purpose.

No comments :