Debian Bug report logs - #802241
please store the hash of the installed .deb and allow one to query it

Reply or subscribe to this bug.

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@mapreri.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: please store the hash of the installed .deb and allow to query it

Date: Sun, 18 Oct 2015 18:20:01 +0000

[Message part 1 (text/plain, inline)]

Package: dpkg
Version: 1.18.3
Severity: wishlist
X-Debbugs-CC: reproducible-builds@lists.alioth.debian.org

Hi dpkg people,

in the context of allowing to recreate the same build-environment of a
past build we would need to know which packages where installed.
Currently we rely on (pkgname, arch, version) tuples to uniquely
identify a binary package, but as you can easily imagine this is not
unique at all, definitly not in the multi distro universe, possibly not
even across suites.
This can also help quite some higher level package manager to identify
which archive is providing the installed package, as David Kalnischkies
pointed out in https://lists.debian.org/20150624164233.GA25413@crossbow

I would think to just add a field in /var/lib/dpkg/status but YMMV and
I'm happy with everything.

As a side effect this allows enables anyone easily whether a package
came from the Debian archive or from somewhere else.


This matter was already briefly discussed in ML, and ended up with some
open questions in https://lists.debian.org/20150623073105.GE5719@loar so
let's file this bug to way easily track it.

To me it seems that:
* we are mostly interested in the hash of the whole container: all the
  use cases highlighted above would require this;
* If ↑ then the hash can't be pre-computed and stored inside the
  container.


Thanks in advance for everything!

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  http://mapreri.org                              : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Message #12 received at 802241@bugs.debian.org (full text, mbox, reply):

From: Julian Andres Klode <jak@debian.org>

To: Mattia Rizzolo <mattia@mapreri.org>, 802241@bugs.debian.org

Subject: Re: Bug#802241: please store the hash of the installed .deb and allow to query it

Date: Sat, 26 Aug 2017 14:14:16 +0200

On Sun, Oct 18, 2015 at 06:20:01PM +0000, Mattia Rizzolo wrote:
> Package: dpkg
> Version: 1.18.3
> Severity: wishlist
> X-Debbugs-CC: reproducible-builds@lists.alioth.debian.org
> 
> Hi dpkg people,
> 
> in the context of allowing to recreate the same build-environment of a
> past build we would need to know which packages where installed.
> Currently we rely on (pkgname, arch, version) tuples to uniquely
> identify a binary package, but as you can easily imagine this is not
> unique at all, definitly not in the multi distro universe, possibly not
> even across suites.

I also want this for delta debs, to identify local rebuilds being
installed, and prevent delta installation failure in such cases.

> 
> To me it seems that:
> * we are mostly interested in the hash of the whole container: all the
>   use cases highlighted above would require this;
> * If ↑ then the hash can't be pre-computed and stored inside the
>   container.

Practically speaking, for your use case you only need a hash of the
file tree. My proposal for a package id is to use the md5sum of
DEBIAN/md5sums. This can be stored in DEBIAN/control in an id
field and generated at build time. 

We can also use cat DEBIAN/md5sums DEBIAN/control | md5sum (without an
Id field in control) as the ID, and then append that to control. This
means that dependency relations and stuff is included as well. That's
useful for the solver use case; but it's not really relevant for
the reproducible build use case - dependencies on the installed
system, description, etc should not matter for you.

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
                  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Message #15 received at 802241@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@mapreri.org>

To: Julian Andres Klode <jak@debian.org>, 802241@bugs.debian.org

Subject: Re: Bug#802241: please store the hash of the installed .deb and allow to query it

Date: Sat, 26 Aug 2017 15:16:52 +0200

[Message part 1 (text/plain, inline)]

On Sat, Aug 26, 2017 at 02:14:16PM +0200, Julian Andres Klode wrote:
> I also want this for delta debs, to identify local rebuilds being
> installed, and prevent delta installation failure in such cases.

yay another user!

> > To me it seems that:
> > * we are mostly interested in the hash of the whole container: all the
> >   use cases highlighted above would require this;
> > * If ↑ then the hash can't be pre-computed and stored inside the
> >   container.
> 
> Practically speaking, for your use case you only need a hash of the
> file tree. My proposal for a package id is to use the md5sum of
> DEBIAN/md5sums. This can be stored in DEBIAN/control in an id
> field and generated at build time. 

That's not true, as we need the hash also (for example) of all the
maintainer scripts which are not in data.tar (I assume that's what you
meant by "file tree").  Also, we have seen packages where the only
difference is the order of entry in the md5sums file, therefore making
the build not reproducible by our (higher than policy) standards.
We really need the whole container.

> We can also use cat DEBIAN/md5sums DEBIAN/control | md5sum (without an
> Id field in control) as the ID, and then append that to control. This
> means that dependency relations and stuff is included as well. That's
> useful for the solver use case; but it's not really relevant for
> the reproducible build use case - dependencies on the installed
> system, description, etc should not matter for you.

Well, DEBIAN/control contains the dependencies generated during the
package build, and we do are interested in them as well…
In short: we do care about both data.tar and control.tar.  After all, we
do compare the hashes of the final .deb container.


As I saw it when I originally thought of the problem the only sane
solution to this for me would be to have dpkg compute the hash of the
.deb before unpacking it, and store in it's $admindir/status file, but
that makes the installation process very CPU-intensive, to the point
that very probably it's too much to be bareable in many systems.

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 802241@bugs.debian.org (full text, mbox, reply):

From: Julian Andres Klode <jak@debian.org>

To: Mattia Rizzolo <mattia@mapreri.org>

Cc: 802241@bugs.debian.org

Subject: Re: Bug#802241: please store the hash of the installed .deb and allow to query it

Date: Sat, 26 Aug 2017 15:34:26 +0200

On Sat, Aug 26, 2017 at 03:16:52PM +0200, Mattia Rizzolo wrote:
> On Sat, Aug 26, 2017 at 02:14:16PM +0200, Julian Andres Klode wrote:
> > I also want this for delta debs, to identify local rebuilds being
> > installed, and prevent delta installation failure in such cases.
> 
> yay another user!
> 
> > > To me it seems that:
> > > * we are mostly interested in the hash of the whole container: all the
> > >   use cases highlighted above would require this;
> > > * If ↑ then the hash can't be pre-computed and stored inside the
> > >   container.
> > 
> > Practically speaking, for your use case you only need a hash of the
> > file tree. My proposal for a package id is to use the md5sum of
> > DEBIAN/md5sums. This can be stored in DEBIAN/control in an id
> > field and generated at build time. 
> 
> That's not true, as we need the hash also (for example) of all the
> maintainer scripts which are not in data.tar (I assume that's what you
> meant by "file tree").  Also, we have seen packages where the only
> difference is the order of entry in the md5sums file, therefore making
> the build not reproducible by our (higher than policy) standards.
> We really need the whole container.

I don't see why you need maintainer scripts. When you are building
a package what you care about is the state you are building in. And
the maintainer scripts are irrelevant in that state. But well, you could
hash them too.

> 
> > We can also use cat DEBIAN/md5sums DEBIAN/control | md5sum (without an
> > Id field in control) as the ID, and then append that to control. This
> > means that dependency relations and stuff is included as well. That's
> > useful for the solver use case; but it's not really relevant for
> > the reproducible build use case - dependencies on the installed
> > system, description, etc should not matter for you.
> 
> Well, DEBIAN/control contains the dependencies generated during the
> package build, and we do are interested in them as well…
> In short: we do care about both data.tar and control.tar.  After all, we
> do compare the hashes of the final .deb container.

I fail to see why you'd be interested in dependencies. Your stated use
case is "allowing to recreate the same build-environment of a past build
we would need to know which packages where installed.". To recreate a 
build environment you do not have to care about dependencies, nor do
you have to care about maintainer scripts (as soon as you involve
upgrades, the result might be different anyway, if you always bootstrap
the exact tree, it might be useful). Because the matter of fact is that
neither maintainer scripts nor dependencies affect how a package is
built (once build-depends are installed).

There's also the question of conffiles. They might affect the environment,
but: The actually installed ones might be different from the ones in the
package (which essentially is the same upgrade problem you have for
maintainer scripts); hence it's not really useful to include them in
the hash (but it does not hurt either).

> As I saw it when I originally thought of the problem the only sane
> solution to this for me would be to have dpkg compute the hash of the
> .deb before unpacking it, and store in it's $admindir/status file, but
> that makes the installation process very CPU-intensive, to the point
> that very probably it's too much to be bareable in many systems.

If you really want everything, the wise choice is to just hash the
entire tree dpkg-deb is packaging up and then add that to the ID
field. You can easily reconstruct the ID by unpacking the package,
removing the ID field from debian/control and rehashing.

Once mtree is in, this also includes stuff like permissions, xattrs.
Then we can also SHA256 everything and use that as an ID.

The only thing not covered by this is the layout of the tar files,
the compression, and the layout of the final ar file. But that's
not really relevant to any of our use cases.

For APT, we specifically need the ID to be in the package and dpkg
not to add any missing IDs, otherwise we cannot rely on the ID as
the installed one might have one but the remote one not.

For deltas, we only care that it has *some* ID, for what it's worth,
it could be a random UUID (but that's not reproducible). I do need
that to be in debian/control, as this will allow us to change the
ID algorithm at any point in time and not require us to recompute
the id when generating the delta.

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
                  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Message #25 received at 802241@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: 802241@bugs.debian.org

Subject: Re: Bug#802241: please store the hash of the installed .deb and allow to query it

Date: Sat, 26 Aug 2017 16:11:29 +0000

[Message part 1 (text/plain, inline)]

hi,

while I very much like this idea, please don't store md5sums, but rather
sha256sums.

Thank you!


-- 
cheers,
	Holger (wondering why I seem to have to write this in 2017)

[signature.asc (application/pgp-signature, inline)]

Message #30 received at 802241@bugs.debian.org (full text, mbox, reply):

From: Drew Parsons <dparsons@debian.org>

To: 802241@bugs.debian.org

Subject: Re: Bug#802241: please store the hash of the installed .deb and allow to query it

Date: Tue, 24 Apr 2018 16:43:48 +0800

I got caught out by this apt bug when handling sasview. I had installed
a local build of sasview with my patch for 4.2.0~git20180309-3. Stuart
Prescott added another patch and uploaded.

I assumed apt had picked up on the update and installed the final,
official version.  But it hadn't, even though the contents and
changelog had changed.  I was left wondering why the "new" version was
misbehaving (still showing the bug that Stuart had patched).

Drew

Message #35 received at 802241@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: 802241@bugs.debian.org

Cc: Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Subject: re: #802241: dpkg: please store the hash of the installed .deb and allow to query it

Date: Wed, 6 Jun 2018 23:40:54 +0000

[Message part 1 (text/plain, inline)]

Hi Guillem,

ping on this bug, you haven't replied to it yet and it's a blocker for
"#774415 sbuild: please add the srebuild sbuild wrapper to reproduce builds"
which is a rather important one to give users the means to easily
reproduce Debian packages, which is a core feature of reproducible
builds and which we would love to see for Buster…! 

:)


-- 
cheers,
	Holger

[signature.asc (application/pgp-signature, inline)]

Message #40 received at 802241@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Holger Levsen <holger@layer-acht.org>, 802241@bugs.debian.org

Cc: Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Subject: Re: Bug#802241: #802241: dpkg: please store the hash of the installed .deb and allow to query it

Date: Fri, 8 Jun 2018 04:30:45 +0200

Hi!

On Wed, 2018-06-06 at 23:40:54 +0000, Holger Levsen wrote:
> ping on this bug, you haven't replied to it yet and it's a blocker for
> "#774415 sbuild: please add the srebuild sbuild wrapper to reproduce builds"

Oh, had not noticed this was a blocker for anything sorry. And thought
the previous discussion on the list and on the bug was complete enough
for now. :)

> which is a rather important one to give users the means to easily
> reproduce Debian packages, which is a core feature of reproducible
> builds and which we would love to see for Buster…! 

Having reread the blocking bug, and the specific message where josch
says this one is a blocker (https://bugs.debian.org/774415#44), I
think this is actually an artificial blocker!

I think this specific bug should eventually be fixed, how, I don't
know yet. It would fix some rough edges and make life easier for apt
and other frontends.

But for the specific case in the reproducibility effort, I think even
if we fixed this tomorrow you would not be able to rely on it, because
it would require feeding the hashes for all pre-installed packages, as
David Kalnischkies already mentioned previously.

I say it's an artificial blocker, because it is based on the problem
faced while implementing the srebuild script to use the current
snapshot.d.o API. And I think that's your actual blocker. Fixing that
API would also mean you can use it right away independently of what's
already installed on the system and might be useful for other users
too. I think the fix would imply adding an API entry point based on
the name-version-arch tuple.

Thanks,
Guillem

Message #47 received at 802241@bugs.debian.org (full text, mbox, reply):

From: Johannes Schauer <josch@debian.org>

To: 802241@bugs.debian.org, Guillem Jover <guillem@debian.org>, Holger Levsen <holger@layer-acht.org>

Cc: Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Subject: Re: Bug#802241: #802241: dpkg: please store the hash of the installed .deb and allow to query it

Date: Fri, 08 Jun 2018 08:38:49 +0200

[Message part 1 (text/plain, inline)]

Hi,

Quoting Guillem Jover (2018-06-08 04:30:45)
> Having reread the blocking bug, and the specific message where josch
> says this one is a blocker (https://bugs.debian.org/774415#44), I
> think this is actually an artificial blocker!
>
> [...]
>
> I say it's an artificial blocker, because it is based on the problem
> faced while implementing the srebuild script to use the current
> snapshot.d.o API. And I think that's your actual blocker. Fixing that
> API would also mean you can use it right away independently of what's
> already installed on the system and might be useful for other users
> too. I think the fix would imply adding an API entry point based on
> the name-version-arch tuple.

yes, that would also solve the problem.

I unblocked the bug, because it's not a hard blocker but just an inconvenience.
The bigger blocker of #774415 is, that the script needs somebody who feels
responsible for it and who is willing to maintain it. I only wrote it but I
have no intention of being its maintainer.

Thanks!

cheers, josch

[signature.asc (application/pgp-signature, inline)]

Message #52 received at 802241@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: 802241@bugs.debian.org

Cc: Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Subject: Re: Bug#802241: #802241: dpkg: please store the hash of the installed .deb and allow to query it

Date: Fri, 8 Jun 2018 15:47:47 +0000

[Message part 1 (text/plain, inline)]

Guillem, josch:

thanks for your feedback, much appreciated.

On Fri, Jun 08, 2018 at 08:38:49AM +0200, Johannes Schauer wrote:
> > I say it's an artificial blocker, because it is based on the problem
> > faced while implementing the srebuild script to use the current
> > snapshot.d.o API. And I think that's your actual blocker. Fixing that
> > API would also mean you can use it right away independently of what's
> > already installed on the system and might be useful for other users
> > too. I think the fix would imply adding an API entry point based on
> > the name-version-arch tuple.
> 
> yes, that would also solve the problem.
 
as I'm not an sbuild user (yet) myself, I was hesistant to try this
myself, so I'm confused now: does it work as it is now? (or does it need
changes to snapshot.d.o?)

> I unblocked the bug, because it's not a hard blocker but just an inconvenience.

thanks

> The bigger blocker of #774415 is, that the script needs somebody who feels
> responsible for it and who is willing to maintain it. I only wrote it but I
> have no intention of being its maintainer.

I'd be happy to maintain it, once I'm a user of it :) (which might
happen quite soon via tests.r-b.o…)


-- 
cheers,
	Holger

[signature.asc (application/pgp-signature, inline)]

Message #57 received at 802241@bugs.debian.org (full text, mbox, reply):

From: Johannes Schauer <josch@debian.org>

To: 802241@bugs.debian.org, Holger Levsen <holger@layer-acht.org>

Cc: Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Subject: Re: Bug#802241: #802241: dpkg: please store the hash of the installed .deb and allow to query it

Date: Fri, 08 Jun 2018 19:54:20 +0200

[Message part 1 (text/plain, inline)]

Hi Holger,

Quoting Holger Levsen (2018-06-08 17:47:47)
> as I'm not an sbuild user (yet) myself, I was hesistant to try this
> myself, so I'm confused now: does it work as it is now? (or does it need
> changes to snapshot.d.o?)

yes, it does work as it is now.

Just supply the script with a buildinfo file to see it in action.

It does not require superuser privileges.

The script will query snapshot.debian.org to retrieve the right snapshot
timestamp that contains all the package versions specified in the buildinfo
file.

At the end of execution the script will print how to either reproduce the
buildinfo manually via dpkg-buildpackage or how to run sbuild such that it does
it for you.

People who know how to use pbuilder could easily add a section that outputs how
to run pbuilder to do the same.

Naturally, instead of just printing how to use sbuild or pbuilder, the script
could also be made actually run either.

The main two limitations of the script are:

 1. it will fail if there is not a single snapshot that contains all the right
    package versions

 2. it will instruct sbuild/pbuilder to use the last stable release as the base
    which might not allow upgrading to the right package versions

Both issues can be fixed by manually downloading exactly the required binary
package set and creating a completely new chroot with exactly the required
packages. But I didn't get around to doing that yet.

Thanks!

cheers, josch

[signature.asc (application/pgp-signature, inline)]

Message #62 received at 802241@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: Johannes Schauer <josch@debian.org>

Cc: 774415@bugs.debian.org, 802241@bugs.debian.org, Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Subject: Re: Bug#802241: #802241: dpkg: please store the hash of the installed .deb and allow to query it

Date: Sat, 9 Jun 2018 20:12:33 +0000

[Message part 1 (text/plain, inline)]

Hi josch,

adding #774415 to to: and reply-to:…

On Fri, Jun 08, 2018 at 07:54:20PM +0200, Johannes Schauer wrote:
> > as I'm not an sbuild user (yet) myself, I was hesistant to try this
> > myself, so I'm confused now: does it work as it is now? (or does it need
> > changes to snapshot.d.o?)
> 
> yes, it does work as it is now.
> 
> Just supply the script with a buildinfo file to see it in action.
> 
> It does not require superuser privileges.
> 
> The script will query snapshot.debian.org to retrieve the right snapshot
> timestamp that contains all the package versions specified in the buildinfo
> file.
> 
> At the end of execution the script will print how to either reproduce the
> buildinfo manually via dpkg-buildpackage or how to run sbuild such that it does
> it for you.
> 
> People who know how to use pbuilder could easily add a section that outputs how
> to run pbuilder to do the same.
> 
> Naturally, instead of just printing how to use sbuild or pbuilder, the script
> could also be made actually run either.
> 
> The main two limitations of the script are:
> 
>  1. it will fail if there is not a single snapshot that contains all the right
>     package versions
> 
>  2. it will instruct sbuild/pbuilder to use the last stable release as the base
>     which might not allow upgrading to the right package versions
> 
> Both issues can be fixed by manually downloading exactly the required binary
> package set and creating a completely new chroot with exactly the required
> packages. But I didn't get around to doing that yet.

thank you very much for this nice summary!

As it sounds, I now believe this script would better live in
src:devscripts and as such I would like to reassign #774415 to
devscripts - or do you see any issue with that?


-- 
cheers,
	Holger

[signature.asc (application/pgp-signature, inline)]

Message #67 received at 802241@bugs.debian.org (full text, mbox, reply):

From: Johannes Schauer <josch@debian.org>

To: 774415@bugs.debian.org, 802241@bugs.debian.org, Holger Levsen <holger@layer-acht.org>, Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Cc: 774415@bugs.debian.org, 802241@bugs.debian.org, Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Subject: Re: Bug#802241: #802241: dpkg: please store the hash of the installed .deb and allow to query it

Date: Sat, 09 Jun 2018 22:33:16 +0200

[Message part 1 (text/plain, inline)]

Quoting Holger Levsen (2018-06-09 22:12:33)
> As it sounds, I now believe this script would better live in src:devscripts
> and as such I would like to reassign #774415 to devscripts - or do you see
> any issue with that?

I see no issues with that from my side.

[signature.asc (application/pgp-signature, inline)]

Send a report that this bug log contains spam.