The fact that, when an ArchiveMember is added via addMember() to a ZipArchive, the data in expandedData is not compressed on the fly, makes it very difficult to create large zip files without running out of memory. I have created a simple workaround by moving the following code from the build() method to the addMember() method:
de.crc32 = std.zlib.crc32(0, cast(void[])de.expandedData);
de.compressedData = cast(ubyte[])std.zlib.compress(cast(void[]))de.expandedData;
de.compressedData = de.compressedData[2 .. de.compressedData.length - 4];
de.compressedSize = de.compressedData.length;
de.expandedSize = de.expandedData.length;
I also added the following to free the memory previously occupied by expandedData:
de.expandedData=null;
This is actually proving to be a significant issue for me, as I am attempting to use std.zip to archive large amounts of DNA sequence, which compresses extremely well, but won't fit in memory uncompressed.
Comment #1 by bugzilla — 2008-08-15T01:51:27Z
Could you please email me the diffs? Thanks!
Comment #2 by dsimcha — 2008-08-15T09:34:28Z
Created attachment 269
Diffs to make std.zip compress on fly.
Comment #3 by dsimcha — 2008-08-15T09:34:48Z
Here they are. Note that I had changed some style stuff to make the code more readable for me when I was trying to understand it, so a little of that may show up in the diffs. Also, I hadn't bothered to include the switch statement to get store mode to work because I didn't need it, but this should be trivial to put back.
Comment #4 by code — 2013-12-22T10:52:33Z
David, how about making a pull for this?
Comment #5 by razvan.nitu1305 — 2017-08-08T08:53:48Z