Commit Graph

2 Commits

Author SHA1 Message Date
sobomax
c9cda0bfe0 o Move logic that determines size of the input image into its own
file. That logic has grown quite significantly now;

o add a special handling for the snapshot images. Those have some
  extra headers at the end of the image and we don't need those
  in the output image really.

MFC after:	6 weeks
2017-06-17 02:58:31 +00:00
sobomax
5be3733785 Improve performance in a few key areas:
o Split the compression across several worker threads. By default, "several"
   matches number of CPUs, capped at 24 for sanity when running on a very big
   hardwares. Provide option to set that number manually;

 o Fix bug inherited from the mkulzma (R.I.P) which degraded already slow LZMA
   compression even further by calling function to release compression state
   after processing each block.

   It is neither documented as required nor actually required by the LZMA
   library. This caused spree of system calls to release memory and then map
   it again for every block. LZMA compression is more than 2x faster after this
   change alone;

 o Record time it takes to do compression and report throughput achieved.

 o Add simple first-level 256 entry hash table for de-dup code, so it's not
   becoming a bottleneck at big files.
2016-04-23 07:23:43 +00:00