Debian Bug report logs - #501456
dpkg: parallel compression and decompression

Reply or subscribe to this bug.

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Paul Wise <pabs@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: dpkg: parallel compression and decompression

Date: Tue, 07 Oct 2008 23:08:08 +0800

[Message part 1 (text/plain, inline)]

Package: dpkg
Version: 1.14.22
Severity: wishlist

For those with systems with multiple CPUs, it would be nice if dpkg
could support multi-threaded decompression and compression. Here are
some implementations and possible hints to implementations of these:

gzip:

http://www.c10n.info/archives/505
http://zlib.net/pigz17.c.gz

bzip2:

http://packages.debian.org/pbzip2

lzma:

Seems there is multi-threading in the windows version, needs porting to
pthread though:

http://groups.google.co.id/group/comp.os.linux.development.apps/browse_thread/thread/f8523eeeeb423fee

-- 
bye,
pabs

http://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 501456@bugs.debian.org (full text, mbox, reply):

From: Riku Voipio <riku.voipio@iki.fi>

To: 501456@bugs.debian.org

Cc: pabs@debian.org, debian-dpkg@list.debian.org

Subject: dpkg: parallel compression and decompression

Date: Thu, 3 Apr 2014 11:05:36 +0300

Hi,

At the buildd:

 22477 buildd    30  10 97752  93m  420 R 100.2  2.5  26:15.31 dpkg-deb -Zxz --build debian/libwebkitgtk-3.0-0-dbg ..                 
26 minutes and counting compressing a dbg package. 
The buildd is quadcore, so considerable amount of
time could be saved with parallel implementation.

The mentioned implmentations are now all in debian:

pigz - Parallel Implementation of GZip
 code lines 4304
pixz - parallel, indexing XZ compressor/decompressor
 code lines 2169
pbzip2 - parallel bzip2 implementation
 code lines 6119 (c++)

Unsurprisingly pixz is also the most readable threaded
compressing/uncompressing implementation. From a quick glance
I guess the liblzma api makes it easier than others.

Now that xz is a) default b) most cpu-heavy, perhaps it's also the only
one worth parallelizing.

Riku

Message #15 received at 501456@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Paul Wise <pabs@debian.org>, 501456@bugs.debian.org, Riku Voipio <riku.voipio@iki.fi>

Cc: Raphael Geissert <geissert@debian.org>, Jonathan Nieder <jrnieder@gmail.com>

Subject: Re: Bug#501456: dpkg: parallel compression and decompression

Date: Sun, 8 Jun 2014 17:15:06 +0200

[ CCing Raphael as showing interest off-list, CCing Jonathan for input
  on liblzma MT-support. ]

Hi!

On Tue, 2008-10-07 at 23:08:08 +0800, Paul Wise wrote:
> Package: dpkg
> Version: 1.14.22
> Severity: wishlist

> For those with systems with multiple CPUs, it would be nice if dpkg
> could support multi-threaded decompression and compression. Here are
> some implementations and possible hints to implementations of these:

On Thu, 2014-04-03 at 11:05:36 +0300, Riku Voipio wrote:
> The mentioned implmentations are now all in debian:
> 
> pigz - Parallel Implementation of GZip
>  code lines 4304
> pixz - parallel, indexing XZ compressor/decompressor
>  code lines 2169
> pbzip2 - parallel bzip2 implementation
>  code lines 6119 (c++)
> 
> Unsurprisingly pixz is also the most readable threaded
> compressing/uncompressing implementation. From a quick glance
> I guess the liblzma api makes it easier than others.
> 
> Now that xz is a) default b) most cpu-heavy, perhaps it's also the only
> one worth parallelizing.

Ok, here go my thoughts. I agree it would be nice in principle. Ideally
I'd like to have parallel support natively in libdpkg (indirectly or
directly), to avoid external dependencies, to avoid exposing internal
implementation details, and because I've been pondering about removing
the code to use the command-line tools (which is currently disabled in
Debian anyway). But that might imply using threads and the dpkg codebase
is definitely not yet thread-safe. I guess it could be implemented in
multiple processes at the cost of possibly a slightly more complicated
implementation.

But, meanwhile as a workaround I've also pondered about making both the
compression library and the command-line tools available at the same time
so that the user could request one or the other, which would also allow
to add additional compressor command-line tools, although have not done
anything in that direction because these pose several problems, see
below.

Some comments on external implementations:

 * liblzma does have a native multi-threaded encoder upstream but it's
   disabled in Debian, debian/patches/abi-threaded-encoder. This would
   make adding native support to dpkg pretty easy. But it seems latest
   position from upstream is that this support could even disappear?
   Jonathan any thoughts or further data on this?
 * pigz uses an internal compressing library (zopfli) that claims to
   compress better at the cost of being very slow, so I'm not sure it's
   worth using. It appears to use zlib for decompression. Would need
   more checking and benchmarks.
 * pixz seems nice, but I'm not sure about its file format compatibility
   when it emits the tarball index. Fortunately it can be disabled
   altogether with the -t option. Would need taking a look too. I'm not
   really prepared to even consider making dpkg Pre-Depend on pixz for
   example, because I don't think pulling libarchive as pseudo-essential
   would be acceptable, but otherwise it's pretty small.
 * pxz was not mentioned but I don't think it's really an option, as
   it uses temporary files… Its implementation uses OpenMP so it would
   really not be an option to lift into libdpkg either.
 * pbzip2 is mainly only useful for compression, as decompression
   requires the data to have been compressed with pbzip2 itself.
   bzip2 is deemed deprecated, so probably not worth it I'd say.

So, until this is natively implemented in libdpkg, I guess it might
make some sense to consider at least supporting pixz in some form
(eventual parallel gzip would be nice too, as the other non-deprecated
compressor supported). Tests would need to be performed on single and
multi-core setups, to see the speed/memory-usage/disk-size differences
with small and huge packages. Then there are several ways the user could
enable this in dpkg-deb, none of which are pretty:

 1) Through a new compressor name to dpkg-deb, say -Zpixz, which
    could fallback to use xz if pixz is not available.

    It would require versioned Build-Depends on the dpkg implementing
    that plus a Build-Depends on pixz in the package making use of
    it. This would be an ok option if there's no major difference in
    the implementations behavior, so that anyone trying to build
    a huge package benefits, and so that no special setup is
    required on buildds. A drawback is that this exposes an internal
    implementation detail, and would not allow to, say, change the
    external tool, as the dependencies would stop being valid.

 2) Through a new dpkg-deb option to enable parallel compression,
    say --parallel-compression, so that dpkg-deb would try the
    parallel tool if it's available instead of the default
    implementation.

    This has almost the same requirements and drawbacks from the
    package maintainers PoV than the -Zpixz option.

 3) Through a new environment variable, so that dpkg-deb would try the
    known tool if it's available instead of the default implementation.
    Say DPKG_DEB_COMPRESSOR=parallel or =pixz or similar.

    This would avoid any versioned Build-Depends on dpkg, as the tool
    would not fail on unknown options. But would require any buildd
    wanting to benefit from this to set that explicitly, and to make
    sure that the environment is not cleaned up, and to manually
    install the tools, which could vary depending on the compression
    used by the package, which is also an internal implementation
    detail that could also change. Or to change debian/rules to set
    the envvar and Build-Depend on the compressor.

    The semantics for =parallel or =pixz are going to be different,
    the former could complement -Zxz (or its omission), but the later
    would get overriden by the -Z option. So either has pros/cons.

I presume, any of the above options that automatically falls back to
the non-parallel implementation, or that changes behavior depending on
environment variables, will make the reproducible builds people quite
unhappy, though.


Hmm, it just occurred to me another very viable temporary option could
be to implement minimal threaded xz and gzip (de)compressors as small
helper programs that would not make any use of libdpkg. And change
libdpkg to use those if requested. This seems would have most of the
benefits w/ very minor drawbacks.


So, all in all, my preferred deployment options in decreasing order,
would be:

 * Implement native parallel support in the compression libraries
   (might require adding thread-safey all over the dpkg codebase,
   depending on the implementation).
 * Implement native parallel support in libdpkg (requires first adding
   thread-safey all over the dpkg codebase).
 * Implement native parallel support in the standard compression
   tools, at least gzip and xz (probably unrealistic).
   Change dpkg to allow using the tools in addition to the libraries.
 * Implement native parallel compressor helpers in dpkg, to be used by
   libdpkg.
   Change dpkg to allow using the tools in addition to the libraries.
 * Change the compression tools to either use alternatives (!?) or
   modify the parallel variants to divert the standard ones (at least
   gzip is Essential, so this is probably out of the table). Some might
   need wrapper scripts as they do not implement all command-line options.
   Change dpkg to allow using the tools in addition to the libraries.
 * Change dpkg to explicitly use one of the parallel variants, as
   detailed above.


Hope this gives some light on the issues here.

Thanks,
Guillem

Message #22 received at 501456@bugs.debian.org (full text, mbox, reply):

From: Alexandre Detiste <alexandre.detiste@gmail.com>

To: 501456@bugs.debian.org

Cc: game-data-packager <game-data-packager@packages.debian.org>, vv221 <vv221@dotslashplay.it>

Subject: dpkg: parallel compression with pixz

Date: Tue, 20 Oct 2015 12:33:56 +0200

Hi,

I stumbled on this request, CCing other people
that might be interrested.

game-data-packager would definitively benefit from this.
For this package, having reproducible builds is irrelevant.
(some files are optional, some have various accepted versions)

Alexandre Detiste

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=501456

Message #27 received at 501456@bugs.debian.org (full text, mbox, reply):

From: Konstantin Khlebnikov <koct9i@gmail.com>

To: 501456@bugs.debian.org

Subject: dpkg-deb: Add new option --compress-program=<prog>

Date: Sat, 18 Jun 2016 23:19:16 +0300

[Message part 1 (text/plain, inline)]

Here is tiny patch which adds new option dpkg-deb --compress-program=<prog>
Program is executed as a filter with single argument: compression level -0 .. -9

So, any existing parallel compression tool (pigz/pbzip2/pxz/pixz)
could be used here.
Of course compression type (-Z) must match. Such compression might not
suit for official
builds because it have lower ratio, but will save tons of time for
building debug packages.

[dpkg-deb-add-new-option-compress-program-prog (application/octet-stream, attachment)]

Message #32 received at 501456@bugs.debian.org (full text, mbox, reply):

From: konacher@host.kona.biz

To: 501456@bugs.debian.org

Subject: Please recheck your delivery address (UPS parcel 01245546)

Date: Sun, 19 Feb 2017 02:26:34 -0500

[Message part 1 (text/plain, inline)]

Dear Customer,

This is to confirm that your item has been shipped at February 17.

Download postal receipt attached to e-mail!

With anticipation,
Darren Helton,
UPS Chief Delivery Manager.

[UPS-Package-01245546.zip (application/zip, attachment)]

Message #37 received at 501456@bugs.debian.org (full text, mbox, reply):

From: Thomas Dreibholz <thomas.dreibholz@googlemail.com>

To: 501456@bugs.debian.org

Subject: Re: dpkg-deb: Add new option --compress-program=<prog>

Date: Thu, 13 Dec 2018 12:40:28 +0100

[Message part 1 (text/plain, inline)]

Hi,

is there any progress on this bug? On a recent 72-core system, a kernel
can be build in a few minutes with "make deb-pkg". But dpkg-deb takes
~45 minutes just to pack everything in a single thread. It would be
really nice to get the patch merged.

-- 

-- 
Best regards / Mit freundlichen Grüßen / Med vennlig hilsen

=======================================================================
 Thomas Dreibholz

 Simula@OsloMet -- Simula Metropolitan Centre for Digital Engineering
 Centre for Resilient Networks and Applications
 Pilestredet 52
 0167 Oslo, Norway
-----------------------------------------------------------------------
 E-Mail:     dreibh@simula.no
 Homepage:   http://simula.no/people/dreibh
=======================================================================

[signature.asc (application/pgp-signature, attachment)]

Message #42 received at 501456@bugs.debian.org (full text, mbox, reply):

From: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>

To: 501456@bugs.debian.org, Paul Wise <pabs@debian.org>

Subject: Re: dpkg: parallel compression and decompression

Date: Thu, 9 Apr 2020 11:45:49 +0200

Can this be closed?

Sebastian

Message #47 received at 501456@bugs.debian.org (full text, mbox, reply):

From: Paul Wise <pabs@debian.org>

To: 501456@bugs.debian.org

Subject: Re: dpkg: parallel compression and decompression

Date: Thu, 09 Apr 2020 19:43:34 +0800

[Message part 1 (text/plain, inline)]

On Thu, 9 Apr 2020 11:45:49 +0200 Sebastian Andrzej Siewior wrote:

> Can this be closed?

Since the patch and feature hasn't been merged, I don't think so.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]

Message #52 received at 501456@bugs.debian.org (full text, mbox, reply):

From: Adrien CLERC <bugs-debian@antipoul.fr>

To: 501456@bugs.debian.org

Subject: Re: dpkg: parallel compression and decompression

Date: Tue, 24 Nov 2020 23:44:43 +0100

[Message part 1 (text/plain, inline)]

Hi,

I would also stress that on recent desktop/laptop, having at least 4 
cores is more than common. So parallel decompression is important to 
speed up installations.

As an example, I tried to use xz and pixz to extract data.tar.xz on a 
tmpfs from openjdk-11-jdk-headless_11.0.9.1+1-1_amd64.deb:

$ hyperfine -p 'rm data.tar; ar x 
openjdk-11-jdk-headless_11.0.9.1+1-1_amd64.deb' 'xz -d data.tar.xz'
Benchmark #1: xz -d data.tar.xz
  Time (mean ± σ):      5.866 s ±  0.023 s    [User: 5.762 s, System: 
0.104 s]
  Range (min … max):    5.816 s …  5.894 s    10 runs

$ hyperfine -p 'rm data.tar; ar x 
openjdk-11-jdk-headless_11.0.9.1+1-1_amd64.deb' 'pixz -d data.tar.xz'
Benchmark #1: pixz -d data.tar.xz
  Time (mean ± σ):      1.235 s ±  0.029 s    [User: 6.253 s, System: 
0.220 s]
  Range (min … max):    1.196 s …  1.279 s    10 runs

Nearly five time faster on my 8C/16T system.

Have a nice day!

Adrien

[Message part 2 (text/html, inline)]

Message #57 received at 501456@bugs.debian.org (full text, mbox, reply):

From: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>

To: Paul Wise <pabs@debian.org>, 501456@bugs.debian.org

Subject: Re: Bug#501456: dpkg: parallel compression and decompression

Date: Sat, 31 Dec 2022 13:33:19 +0100

On 2020-04-09 19:43:34 [+0800], Paul Wise wrote:
> On Thu, 9 Apr 2020 11:45:49 +0200 Sebastian Andrzej Siewior wrote:
> 
> > Can this be closed?
> 
> Since the patch and feature hasn't been merged, I don't think so.

What about now given that #956452 has been closed?

Sebastian

Message #62 received at 501456@bugs.debian.org (full text, mbox, reply):

From: Paul Wise <pabs@debian.org>

To: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>, 501456@bugs.debian.org

Subject: Re: Bug#501456: dpkg: parallel compression and decompression

Date: Wed, 04 Jan 2023 11:42:37 +0800

[Message part 1 (text/plain, inline)]

On Sat, 2022-12-31 at 13:33 +0100, Sebastian Andrzej Siewior wrote:

> What about now given that #956452 has been closed?

That bug/patch seems to have added parallel xz decompression, which is
a good start, but this bug requests parallel compression and parallel
decompression for other compression formats too.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]

Message #69 received at 501456@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Konstantin Khlebnikov <koct9i@gmail.com>, 501456@bugs.debian.org

Cc: Paul Wise <pabs@debian.org>

Subject: Re: Bug#501456: dpkg-deb: Add new option --compress-program=<prog>

Date: Thu, 5 Jan 2023 11:52:20 +0100

Control: tags -1 - patch

Hi!

On Sat, 2016-06-18 at 23:19:16 +0300, Konstantin Khlebnikov wrote:
> Here is tiny patch which adds new option dpkg-deb --compress-program=<prog>
> Program is executed as a filter with single argument: compression level -0 .. -9

The code is not reaping the child, nor closing the file descriptors,
this is covered in fd_fd_filter(), but…

> So, any existing parallel compression tool (pigz/pbzip2/pxz/pixz)
> could be used here.
> Of course compression type (-Z) must match. Such compression might not
> suit for official
> builds because it have lower ratio, but will save tons of time for
> building debug packages.

…this is the main problem I have with such patch, that you can specify
any random program and it might or might not match the intended
compression. Also as stated previously several of these tool generate
output that is not ideal anyway.

Thanks,
Guillem

Message #76 received at 501456@bugs.debian.org (full text, mbox, reply):

From: Konstantin Khlebnikov <koct9i@gmail.com>

To: Guillem Jover <guillem@debian.org>

Cc: 501456@bugs.debian.org, Paul Wise <pabs@debian.org>

Subject: Re: Bug#501456: dpkg-deb: Add new option --compress-program=<prog>

Date: Thu, 5 Jan 2023 19:19:26 +0300

[Message part 1 (text/plain, inline)]

On Thu, 5 Jan 2023 at 13:52, Guillem Jover <guillem@debian.org> wrote:

> Control: tags -1 - patch
>
> Hi!
>
> On Sat, 2016-06-18 at 23:19:16 +0300, Konstantin Khlebnikov wrote:
> > Here is tiny patch which adds new option dpkg-deb
> --compress-program=<prog>
> > Program is executed as a filter with single argument: compression level
> -0 .. -9
>
> The code is not reaping the child, nor closing the file descriptors,
>

Oh, maybe. I just slammed it together to show simple solution.

There was no ready to use library solution for parallel compression.
I was limited by the technology of my time. Maybe nowadays it is invented.


> this is covered in fd_fd_filter(), but…
>
> > So, any existing parallel compression tool (pigz/pbzip2/pxz/pixz)
> > could be used here.
> > Of course compression type (-Z) must match. Such compression might not
> > suit for official
> > builds because it have lower ratio, but will save tons of time for
> > building debug packages.
>
> …this is the main problem I have with such patch, that you can specify
> any random program and it might or might not match the intended
> compression. Also as stated previously several of these tool generate
> output that is not ideal anyway.
>

Well, misconfiguration isn't a big deal.
Dpkg could easily validate magic afterwards.

Just to be clear - I have zero interest now.
That's trivial matter, so anybody could finish that.


>
> Thanks,
> Guillem
>

[Message part 2 (text/html, inline)]

Send a report that this bug log contains spam.