Requests for adding internal multithreading for compression libraries #238
Description
The new version of lzbench in master branch introduces the -I# option, which enables compression using # internal (built-in) threads when supported. Currently available only for the following compressors: fast-lzma2, kanzi, lzham, lzma, xz, and zstd.
The supported compressors must provide memory-to-memory compression APIs which allow to specify a number of threads e.g.:
int64_t lzbench_fastlzma2_compress(char *inbuf, size_t insize, char *outbuf, size_t outsize, codec_options_t *codec_options)
{
size_t ret = FL2_compressMt(outbuf, outsize, inbuf, insize, codec_options->level, codec_options->threads);
if (FL2_isError(ret)) return 0;
return ret;
}
The results with 8 threads (-I8) on tarred Silesia Corpus:
lzbench 2.1.1 | GCC 14.2.0 | 64-bit Linux | AMD EPYC 9554 64-Core Processor
Compressor I_Threads Compress. Decompress. Compr. size Ratio Filename
memcpy 1 16265 MB/s 16259 MB/s 211947520 100.00 ../silesia.tar
fastlzma2 1.0.1 -1 8 34.1 MB/s 408 MB/s 59573883 28.11 ../silesia.tar
fastlzma2 1.0.1 -5 8 37.2 MB/s 569 MB/s 51327104 24.22 ../silesia.tar
fastlzma2 1.0.1 -10 8 24.0 MB/s 108 MB/s 48668818 22.96 ../silesia.tar
lzham 1.0 -d26 -1 8 3.71 MB/s 373 MB/s 54899526 25.90 ../silesia.tar
lzham 1.0 -d26 -4 8 2.43 MB/s 407 MB/s 51177148 24.15 ../silesia.tar
lzma 25.01 -4 8 37.7 MB/s 259 MB/s 55968602 26.41 ../silesia.tar
lzma 25.01 -9 8 4.46 MB/s 103 MB/s 48683275 22.97 ../silesia.tar
xz 5.8.1 -4 8 35.7 MB/s 142 MB/s 52458204 24.75 ../silesia.tar
xz 5.8.1 -9 8 3.11 MB/s 146 MB/s 48766532 23.01 ../silesia.tar
zstd 1.5.7 -9 8 384 MB/s 1154 MB/s 59179159 27.92 ../silesia.tar
zstd 1.5.7 -14 8 56.8 MB/s 1180 MB/s 57396270 27.08 ../silesia.tar
zstd 1.5.7 -18 8 19.1 MB/s 1070 MB/s 53313874 25.15 ../silesia.tar
zstd 1.5.7 -22 8 2.27 MB/s 999 MB/s 52288411 24.67 ../silesia.tar
kanzi 2.4.0 -5 8 125 MB/s 261 MB/s 54013491 25.48 ../silesia.tar
kanzi 2.4.0 -6 8 81.1 MB/s 150 MB/s 49517551 23.36 ../silesia.tar
kanzi 2.4.0 -7 8 55.2 MB/s 79.0 MB/s 47308156 22.32 ../silesia.tar
bsc 3.3.11 -m0 -e2 1 14.9 MB/s 16.8 MB/s 48743676 23.00 ../silesia.tar
bsc 3.3.11 -m0 -e1 1 20.0 MB/s 28.1 MB/s 49295208 23.26 ../silesia.tar
bsc 3.3.11 -m5 -e1 1 34.6 MB/s 15.3 MB/s 49609096 23.41 ../silesia.tar
bsc 3.3.11 -m6 -e1 1 33.3 MB/s 14.1 MB/s 49246610 23.24 ../silesia.tar
bzip3 1.5.2 -1 1 13.5 MB/s 16.4 MB/s 50325695 23.74 ../silesia.tar
bzip3 1.5.2 -3 1 13.6 MB/s 16.2 MB/s 48318294 22.80 ../silesia.tar
bzip3 1.5.2 -6 1 12.9 MB/s 13.1 MB/s 47360186 22.35 ../silesia.tar
bzip3 1.5.2 -9 1 11.7 MB/s 11.6 MB/s 48753972 23.00 ../silesia.tar