Debian Bug report logs - #742012
dpkg fails to upgrade gyp (symlink gets replaced with dir)

Reply or subscribe to this bug.

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Michael Vogt <mvo@debian.org>

To: submit@bugs.debian.org

Subject: dpkg fails to upgrade gyp (symlink gets replaced with dir)

Date: Tue, 18 Mar 2014 10:22:02 +0100

[Message part 1 (text/plain, inline)]

Package: dpkg
Version: 1.17.6

I got the following error when upgrading the "gyp" package. I ran into
this from a ubuntu machine but I reproduced it in a minimal sid chroot
as well:

# dpkg -i gyp_0.1~svn1654-1_all.deb 
(Reading database ... 12267 files and directories currently
installed.)
Preparing to unpack gyp_0.1~svn1654-1_all.deb ...
Unpacking gyp (0.1~svn1654-1) over (0.1~svn1654-1) ...
Setting up gyp (0.1~svn1654-1) ...

root@bod:/tmp# dpkg -i gyp_0.1~svn1729-3_all.deb 
(Reading database ... 12267 files and directories currently
installed.)
Preparing to unpack gyp_0.1~svn1729-3_all.deb ...
Unpacking gyp (0.1~svn1729-3) over (0.1~svn1654-1) ...
dpkg: error processing archive gyp_0.1~svn1729-3_all.deb (--install):
 unable to open
 '/usr/share/pyshared/gyp-0.1.egg-info/dependency_links.txt.dpkg-new':
 No such file or directory
Errors were encountered while processing:
 gyp_0.1~svn1729-3_all.deb

After some debugging I think the following is what happens:

1. The old package contains a symlink 
   /usr/lib/python2.7/dist-packages/gyp-0.1.egg-info 
2. The new package contains a directory 
   /usr/lib/python2.7/dist-packages/gyp-0.1.egg-info 
3. On upgrade the symlink is not replaced with the directory, it still
   is a symlink, this causes the error.

I can send a full dpkg debug log. The debs in question can be
downloaded from e.g.:
http://launchpadlibrarian.net/154612335/gyp_0.1%7Esvn1729-3_all.deb
http://launchpadlibrarian.net/145640491/gyp_0.1%7Esvn1654-1_all.deb

But that should not be needed, I also created two minimal debs that
seem to trigger the issue:

$ sudo dpkg -i symlink-becomes-dir_1.0_all.deb && sudo dpkg -i symlink-becomes-dir_2.0_all.deb && echo "have:" && ls -ald /usr/share/symlink-becomes-dir && printf "\nwant:\n" && dpkg -c symlink-becomes-dir_2.0_all.deb |grep /usr/share/symlink-becomes-dir
(Reading database ... 436479 files and directories currently installed.)
Removing symlink-becomes-dir ...
Selecting previously unselected package symlink-becomes-dir.
(Reading database ... 436479 files and directories currently installed.)
Unpacking symlink-becomes-dir (from symlink-becomes-dir_1.0_all.deb) ...
Setting up symlink-becomes-dir (1.0) ...
(Reading database ... 436481 files and directories currently installed.)
Preparing to replace symlink-becomes-dir 1.0 (using symlink-becomes-dir_2.0_all.deb) ...
Unpacking replacement symlink-becomes-dir ...
Setting up symlink-becomes-dir (2.0) ...
have:
lrwxrwxrwx 1 root root 3 Mar 18 10:07 /usr/share/symlink-becomes-dir -> foo

want:
drwxr-xr-x root/root         0 2014-03-18 10:08 ./usr/share/symlink-becomes-dir/


As for the real gyp deb:

When I inspect the system before dpkg cleans up again, I see the
following:
# ls -al /usr/share/pyshared/gyp-0.1.egg-info
total 16
drwxr-xr-x 2 root root 4096 Mar 18 08:46 .
drwxr-xr-x 4 root root 4096 Mar 18 08:46 ..
lrwxrwxrwx 1 root root   52 Oct 22 02:42 PKG-INFO.dpkg-new ->
../../../../share/pyshared/gyp-0.1.egg-info/PKG-INFO
lrwxrwxrwx 1 root root   55 Oct 22 02:42 SOURCES.txt.dpkg-new ->
../../../../share/pyshared/gyp-0.1.egg-info/SOURCES.txt
lrwxrwxrwx 1 root root   64 Oct 22 02:42 dependency_links.txt.dpkg-new
-> ../../../../share/pyshared/gyp-0.1.egg-info/dependency_links.txt
lrwxrwxrwx 1 root root   60 Oct 22 02:42 entry_points.txt.dpkg-new ->
../../../../share/pyshared/gyp-0.1.egg-info/entry_points.txt
lrwxrwxrwx 1 root root   57 Oct 22 02:42 top_level.txt.dpkg-new ->
../../../../share/pyshared/gyp-0.1.egg-info/top_level.txt

Which looks incorrect, a properly installed deb has no symlinks here:
# ls -la /usr/share/pyshared/gyp-0.1.egg-info/
total 28
drwxr-xr-x 2 root root 4096 Mar 18 08:47 .
drwxr-xr-x 4 root root 4096 Mar 18 08:47 ..
-rw-r--r-- 1 root root  242 Oct 22 02:42 PKG-INFO
-rw-r--r-- 1 root root 1145 Oct 22 02:42 SOURCES.txt
-rw-r--r-- 1 root root    1 Oct 22 02:42 dependency_links.txt
-rw-r--r-- 1 root root   41 Oct 22 02:42 entry_points.txt
-rw-r--r-- 1 root root    4 Oct 22 02:42 top_level.txt

That is because:
# ls -la /usr/lib/python2.7/dist-packages/gyp-0.1.egg-info
lrwxrwxrwx 1 root root 40 Jul 23  2013 /usr/lib/python2.7/dist-packages/gyp-0.1.egg-info -> ../../../share/pyshared/gyp-0.1.egg-info

Which is a directory on a correctly installed package:
# ls -lad /usr/lib/python2.7/dist-packages/gyp-0.1.egg-info/
drwxr-xr-x 2 root root 4096 Mar 18 08:47 /usr/lib/python2.7/dist-packages/gyp-0.1.egg-info/


This is also reported in launchpad as
https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/1246730 


Thanks,
 Michael

[symlink-becomes-dir_1.0_all.deb (application/x-debian-package, attachment)]

[symlink-becomes-dir_2.0_all.deb (application/x-debian-package, attachment)]

Message #10 received at 742012@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Michael Vogt <mvo@debian.org>, 742012@bugs.debian.org

Subject: Re: Bug#742012: dpkg fails to upgrade gyp (symlink gets replaced with dir)

Date: Tue, 18 Mar 2014 15:33:35 +0100

Hi!

On Tue, 2014-03-18 at 10:22:02 +0100, Michael Vogt wrote:
> Package: dpkg
> Version: 1.17.6
> 
> I got the following error when upgrading the "gyp" package. I ran into
> this from a ubuntu machine but I reproduced it in a minimal sid chroot
> as well:
[…]

> After some debugging I think the following is what happens:
> 
> 1. The old package contains a symlink 
>    /usr/lib/python2.7/dist-packages/gyp-0.1.egg-info 
> 2. The new package contains a directory 
>    /usr/lib/python2.7/dist-packages/gyp-0.1.egg-info 
> 3. On upgrade the symlink is not replaced with the directory, it still
>    is a symlink, this causes the error.

This is “expected” behaviour, I documented it in the dpkg FAQ because it
seems to trip people over. The Debian policy also documents this (§6.6.4).

<https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Will_dpkg_replace_a_symlink_with_a_directory_or_vice_versa.3F>

Given the above, I'll be closing this report if there's no other
issues besides this one.

Thanks,
Guillem

Message #15 received at 742012@bugs.debian.org (full text, mbox, reply):

From: Michael Vogt <mvo@debian.org>

To: Guillem Jover <guillem@debian.org>

Subject: Re: Bug#742012: dpkg fails to upgrade gyp (symlink gets replaced with dir)

Date: Tue, 18 Mar 2014 18:50:41 +0100

[Message part 1 (text/plain, inline)]

On Tue, Mar 18, 2014 at 03:33:35PM +0100, Guillem Jover wrote:
> Hi!
Hi Guillem,

[..]
> This is “expected” behaviour, I documented it in the dpkg FAQ because it
> seems to trip people over. The Debian policy also documents this (§6.6.4).
> 
> <https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Will_dpkg_replace_a_symlink_with_a_directory_or_vice_versa.3F>
> 
> Given the above, I'll be closing this report if there's no other
> issues besides this one.

Thanks for your reply and the link to the FAQ and sorry for this
unneeded bugreport. I should have been move careful in my
research. Feel free to close the report.

Given that this behavior is not that (that) intuitive I wonder if dpkg
could issue a warning during unpack? Maybe something like the attached
patch? I'm happy to improve the warning message and add one for the
other way around (dir is replaced with symlink).

Could you elaborate a little bit what the FAQ means with metadata
tracking? Would the $pkg.lists file need to be amended with
information like:
/path/to/dir =d
/path/to/file =f
/path/to/symlink =s
?

Thanks for your consideration.

Cheers,
 Michael

[warn-on-switch-dirtype.diff (text/x-diff, attachment)]

Message #20 received at 742012@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Michael Vogt <mvo@debian.org>, 742012@bugs.debian.org

Subject: Re: Bug#742012: dpkg fails to upgrade gyp (symlink gets replaced with dir)

Date: Fri, 21 Mar 2014 19:35:34 +0100

Control: forcemerge 406715 -1

Hi!

On Tue, 2014-03-18 at 18:50:41 +0100, Michael Vogt wrote:
> On Tue, Mar 18, 2014 at 03:33:35PM +0100, Guillem Jover wrote:
> > This is “expected” behaviour, I documented it in the dpkg FAQ because it
> > seems to trip people over. The Debian policy also documents this (§6.6.4).
> > 
> > <https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Will_dpkg_replace_a_symlink_with_a_directory_or_vice_versa.3F>
> > 
> > Given the above, I'll be closing this report if there's no other
> > issues besides this one.

> Thanks for your reply and the link to the FAQ and sorry for this
> unneeded bugreport. I should have been move careful in my
> research. Feel free to close the report.

No problem. Merging it now with previous incarnations of this.

> Given that this behavior is not that (that) intuitive I wonder if dpkg
> could issue a warning during unpack? Maybe something like the attached
> patch? I'm happy to improve the warning message and add one for the
> other way around (dir is replaced with symlink).

The problem is that this has been the intended behaviour for dpkg all
along, mainly to support local admins switching directories to symlinks
and the reverse to be able to rearrange filesystems and disks w/o
needing whole reinstalls and similar. So starting to issue warnings
would be quite annoying in those cases. The issue here is that dpkg does
not have enough information to distinguish if the discrepancy is due to
a switch in the packaging or due to sysadmin actions.

> Could you elaborate a little bit what the FAQ means with metadata
> tracking? Would the $pkg.lists file need to be amended with
> information like:
> /path/to/dir =d
> /path/to/file =f
> /path/to/symlink =s
> ?

More or less, in addition to file size, uid/gid, mode, symlink target,
device IDs, etc. Reusing the .list files is probably not a good idea
though, because unfortunately other programs try to use those internal
files and they might silently break then. And if those are to disappear,
then I might as well try to store the data in a way that makes it faster
to load, as .list files are the currently slowest part of dpkg startup,
in many situations.

My current thinking is to use a new mtree(5) formatted file to track
those and also the file checksums. I think I might have the beginning
of some code for that somewhere, but I'm not sure. In any case, I still
need to consider how to internally handle shared paths with different
metadata, like directories or ref-counted paths in M-A:same packages,
and metadata switched for those shared paths, for example.

Thanks,
Guillem

Send a report that this bug log contains spam.