Debian Bug report logs - #852476
file: /etc/magic no longer searched by default

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Olly Betts <olly@survex.com>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: file: /etc/magic no longer searched by default

Date: Wed, 25 Jan 2017 08:59:04 +1300

[Message part 1 (text/plain, inline)]

Package: file
Version: 1:5.29-1
Severity: important

In jessie the default magic path is "/etc/magic:/usr/share/misc/magic":

$ dpkg -l file|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version            Architecture Description
+++-==============-==================-============-==========================================
ii  file           1:5.22+15-2+deb8u3 amd64        Determines file type using "magic" numbers
$ file -v
file-5.22
magic file from /etc/magic:/usr/share/misc/magic
$ cat /etc/debian_version 
8.7

Somewhere between then and current unstable this has changed to instead
look for ~/.magic instead of /etc/magic:

$ dpkg -l file|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-==========================================================
ii  file           1:5.29-2     amd64        Recognize the type of data in a file using "magic" numbers
$ file -v
file-5.29
magic file from /home/olly/.magic:/usr/share/misc/magic
$ cat /etc/debian_version 
stretch/sid

I can't see any mention of this change in either
/usr/share/doc/file/changelog.Debian.gz or
/usr/share/doc/file/changelog.gz so it isn't clear to me if this change
was intentional or not.  But this change is unhelpful (custom magic
rules in /etc/magic will quietly no longer be searched).  This seems to
be a regression of https://bugs.debian.org/279324 and comments there
suggest that both the reporter and maintainer thought that searching
/etc/magic is the correct and expected behaviour.

It looks like the cause is that two hunks were dropped from the patch
local.support-local-definitions-in-etc-magic.patch between 5.28-4 and
5.29-1:

$ interdiff file-{5.28-4,5.29-1}/debian/patches/local.support-local-definitions-in-etc-magic.patch
reverted:
--- b/src/Makefile.am
+++ a/src/Makefile.am
@@ -4,7 +4,7 @@
 
 bin_PROGRAMS = file
 
+AM_CPPFLAGS = -DMAGIC='"$(MAGIC)"'
-AM_CPPFLAGS = -DMAGIC='"/etc/magic:$(MAGIC)"'
 AM_CFLAGS = $(CFLAG_VISIBILITY) @WARNINGS@
 
 libmagic_la_SOURCES = magic.c apprentice.c softmagic.c ascmagic.c \
reverted:
--- b/src/Makefile.in
+++ a/src/Makefile.in
@@ -340,7 +340,7 @@
 MAGIC = $(pkgdatadir)/magic
 lib_LTLIBRARIES = libmagic.la
 nodist_include_HEADERS = magic.h
+AM_CPPFLAGS = -DMAGIC='"$(MAGIC)"'
-AM_CPPFLAGS = -DMAGIC='"/etc/magic:$(MAGIC)"'
 AM_CFLAGS = $(CFLAG_VISIBILITY) @WARNINGS@
 libmagic_la_SOURCES = magic.c apprentice.c softmagic.c ascmagic.c \
 	encoding.c compress.c is_tar.c readelf.c print.c fsmagic.c \

The documentation seems to not exactly match either behaviour - "man
file" says:

| The information identifying these files is read from /etc/magic and
| the compiled magic file /usr/share/misc/magic.mgc, or the files in the
| directory /usr/share/misc/magic if the compiled file does not exist.
| In addition, if $HOME/.magic.mgc or $HOME/.magic exists, it will be
| used in preference to the system magic files.

The wording could perhaps be clearer, but that seems to imply that the
default should actually be to search:

$HOME/.magic:/etc/magic:/usr/share/misc/magic

Cheers,
    Olly

-- System Information:
Debian Release: stretch/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'unstable'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.7.0-1-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_NZ.UTF-8, LC_CTYPE=en_NZ.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages file depends on:
ii  libc6      2.24-8
ii  libmagic1  1:5.29-2
ii  zlib1g     1:1.2.8.dfsg-4

file recommends no packages.

file suggests no packages.

-- no debconf information

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 852476@bugs.debian.org (full text, mbox, reply):

From: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

To: Olly Betts <olly@survex.com>, 852476@bugs.debian.org

Subject: Re: Bug#852476: file: /etc/magic no longer searched by default

Date: Tue, 24 Jan 2017 22:05:28 +0100

[Message part 1 (text/plain, inline)]

Control: tags 852476 confirmed pending

Olly Betts wrote...

> I can't see any mention of this change in either
> /usr/share/doc/file/changelog.Debian.gz or
> /usr/share/doc/file/changelog.gz so it isn't clear to me if this change
> was intentional or not.

> It looks like the cause is that two hunks were dropped from the patch
> local.support-local-definitions-in-etc-magic.patch between 5.28-4 and
> 5.29-1:

Thanks for the catch and the analysis, this was not intended. Just a
fallout in the attempt to debug #840754. Will take care of that in due
course.

> The documentation seems to not exactly match either behaviour - "man
> file" says:
> 
> | The information identifying these files is read from /etc/magic and
> | the compiled magic file /usr/share/misc/magic.mgc, or the files in the
> | directory /usr/share/misc/magic if the compiled file does not exist.
> | In addition, if $HOME/.magic.mgc or $HOME/.magic exists, it will be
> | used in preference to the system magic files.
> 
> The wording could perhaps be clearer, but that seems to imply that the
> default should actually be to search:
> 
> $HOME/.magic:/etc/magic:/usr/share/misc/magic

Um, are you saying the wording short rather be in the order of actual
probing so it was clearer - or is there something plain wrong?

According to strace, file(1) checks at the moment:

stat("$HOME/.magic.mgc", 0x7fff7a46d870) = -1 ENOENT (No such file or directory)
stat("$HOME/.magic", 0x7fff7a46d870) = -1 ENOENT (No such file or directory)
open("/usr/share/misc/magic.mgc", O_RDONLY) = 3

... which is how I read the manpage - modulo the lost /etc/magic I will
restore.

    Christoph

[signature.asc (application/pgp-signature, inline)]

Message #17 received at 852476@bugs.debian.org (full text, mbox, reply):

From: Olly Betts <olly@survex.com>

To: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

Cc: 852476@bugs.debian.org

Subject: Re: Bug#852476: file: /etc/magic no longer searched by default

Date: Tue, 24 Jan 2017 22:49:24 +0000

On Tue, Jan 24, 2017 at 10:05:28PM +0100, Christoph Biedl wrote:
> Olly Betts wrote...
> > The documentation seems to not exactly match either behaviour - "man
> > file" says:
> > 
> > | The information identifying these files is read from /etc/magic and
> > | the compiled magic file /usr/share/misc/magic.mgc, or the files in the
> > | directory /usr/share/misc/magic if the compiled file does not exist.
> > | In addition, if $HOME/.magic.mgc or $HOME/.magic exists, it will be
> > | used in preference to the system magic files.
> > 
> > The wording could perhaps be clearer, but that seems to imply that the
> > default should actually be to search:
> > 
> > $HOME/.magic:/etc/magic:/usr/share/misc/magic
> 
> Um, are you saying the wording short rather be in the order of actual
> probing so it was clearer - or is there something plain wrong?

Mostly that it would be clearer if the orders matched.

But I'm also not sure how to interpret "in preference" there - it could
mean "~/.magic is used if it exists, otherwise the system magic files are",
but as you say strace shows that's not what actually happens.  So it seems
"will be used in preference to" really just means "will be read before".

Cheers,
    Olly

Message #22 received at 852476-close@bugs.debian.org (full text, mbox, reply):

From: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

To: 852476-close@bugs.debian.org

Subject: Bug#852476: fixed in file 1:5.29-3

Date: Thu, 26 Jan 2017 01:48:40 +0000

Source: file
Source-Version: 1:5.29-3

We believe that the bug you reported is fixed in the latest version of
file, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 852476@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Christoph Biedl <debian.axhn@manchmal.in-ulm.de> (supplier of updated file package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Thu, 26 Jan 2017 00:29:24 +0100
Source: file
Binary: file libmagic1 libmagic-mgc libmagic-dev python-magic python3-magic
Architecture: source powerpc all
Version: 1:5.29-3
Distribution: unstable
Urgency: medium
Maintainer: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>
Changed-By: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>
Description:
 file       - Recognize the type of data in a file using "magic" numbers
 libmagic-dev - Recognize the type of data in a file using "magic" numbers - deve
 libmagic-mgc - File type determination library using "magic" numbers (compiled m
 libmagic1  - Recognize the type of data in a file using "magic" numbers - libr
 python-magic - Recognize the type of data in a file using "magic" numbers - Pyth
 python3-magic - Recognize the type of data in a file using "magic" numbers - Pyth
Closes: 838860 852476
Changes:
 file (1:5.29-3) unstable; urgency=medium
 .
   * Restore full local.support-local-definitions-in-etc-magic patch.
     Closes: #852476
   * Include all upstream commits since the 5.29 release
   * Improve detection of Flash data. Closes: #838860
Checksums-Sha1:
 c52d158b6f0a0ee08270f3ba8ede2c05e64662eb 2124 file_5.29-3.dsc
 75e0d09a4cebfed521d516c6123bb1eee6dc2a7c 41052 file_5.29-3.debian.tar.xz
 0984aae9455e27ebd10f1cab3f016dd8cacd9080 14118 file-dbgsym_5.29-3_powerpc.deb
 c2539a91298a2905acb655a1d6049755bc12fae8 6836 file_5.29-3_powerpc.buildinfo
 4f84255c33e689ae9d0e9f6cc0308342b1eada77 63652 file_5.29-3_powerpc.deb
 6819c81cd5c68f9274bd67874bd95a0eac3b1376 115776 libmagic-dev_5.29-3_powerpc.deb
 0f073252673b476b8828bd87b14a298f7571802b 221996 libmagic-mgc_5.29-3_powerpc.deb
 014942600d88a905b00b96c6d98b20daa11f2aea 152734 libmagic1-dbgsym_5.29-3_powerpc.deb
 04a659f1cf0e1056069c238c8baeda66899606fd 108606 libmagic1_5.29-3_powerpc.deb
 18390f355a242ef06452367abd5665549b4f114c 47860 python-magic_5.29-3_all.deb
 1a23720b895815ecec411ec98a6ea0ce4f8ac591 47928 python3-magic_5.29-3_all.deb
Checksums-Sha256:
 e72d2d4b53a2872f36fefe6a8bc38068ec5882ab44f59b54b8357debd9d64315 2124 file_5.29-3.dsc
 0c4265ea108b6f25cc8cb742542ed013ef207ce5a66b8cdfd121b1c596d3f28d 41052 file_5.29-3.debian.tar.xz
 3830dc2c1e80aa8fd0cf1d91ab659f04efea751a3d8565a77b842eb4ef1379a3 14118 file-dbgsym_5.29-3_powerpc.deb
 858ad02d35d87bd88c7203e966f297a822c0a252b7f3a93398486862f0559d60 6836 file_5.29-3_powerpc.buildinfo
 6c198c174949df76c7ac7602cf260d2d2373d62c3796b307f12ebfe6cbc27a07 63652 file_5.29-3_powerpc.deb
 34bfdac21f777056aaf53aa66d907a5d03790407ac7c618dd846c0e693f38bf7 115776 libmagic-dev_5.29-3_powerpc.deb
 8591bca019ba4dbd1cb35f0b6a84449ffc64cacb0c611efb54dae5539cf1fc80 221996 libmagic-mgc_5.29-3_powerpc.deb
 141355d450901acf42afce535e7eff847a280321cb72a0185044500a291841d5 152734 libmagic1-dbgsym_5.29-3_powerpc.deb
 4dfdf2f90c42ae7fdd22ab6fe815afeb03286e6805167a967a4be90428f25354 108606 libmagic1_5.29-3_powerpc.deb
 5cb59dc176157d6f9dd88585a81d369317ac381e86f44052821d911207b6dd33 47860 python-magic_5.29-3_all.deb
 e88e03ac9445a52ef2bf14fec160380752708a94bcc36e1847b26bcfc798913b 47928 python3-magic_5.29-3_all.deb
Files:
 677843489a933c3207832bf5f6cd718d 2124 utils standard file_5.29-3.dsc
 cc72c748d7e75ac55d87bde9da4b50e7 41052 utils standard file_5.29-3.debian.tar.xz
 b35d1ffde9f796941217f0ea30d69ed0 14118 debug extra file-dbgsym_5.29-3_powerpc.deb
 53d0918ec98a6be81dd5779227f98ab9 6836 utils standard file_5.29-3_powerpc.buildinfo
 6cf834d69b450dbd9420f5a118188db3 63652 utils standard file_5.29-3_powerpc.deb
 e8026b2faa0f309d86171d281e7c2132 115776 libdevel optional libmagic-dev_5.29-3_powerpc.deb
 245eb1eeb1b453c465172315e0b1364b 221996 libs standard libmagic-mgc_5.29-3_powerpc.deb
 a6738ecda11706f8db8d5e48ba7834b2 152734 debug extra libmagic1-dbgsym_5.29-3_powerpc.deb
 d1c62b8334731f087adf8ce19fd5c9ca 108606 libs standard libmagic1_5.29-3_powerpc.deb
 3a4a78ddba1da926b543358185deb18a 47860 python optional python-magic_5.29-3_all.deb
 00a9c345ef8145f0af7b794619d9c195 47928 python optional python3-magic_5.29-3_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJYiU5qAAoJEMQsWOtZFJL9XhoQANHZp7zfXFuMgK8wX8FrFbpJ
M7tj4LEcZ3K9P3gT/ALZLmqSDf6DrqUnBzHIlTc+hZI7OfFJ16HKfleBlnFX7L1G
RK1fLdn5I16oPLOYfF+nt+UN7gqFlCenI0W3SFlLR5uvylA+FcZ9eONsChONIOCi
uIieN4PxrHSqdVepqL9uyW0t5koEgQGTFAuW9/LJE9PlBN9ygxZTzXn9CzcW8bRi
1MnSqiZ/ZknXWwTSULuc1E/D/k5hm29+sRXPaIwSaVuGbXcYOoOsWkWWLn9IYMrh
hcvxX6kyXv/ZvVhZJEmit6PxuHnMIdlw7ppNBcV9Jtuw1jMVlyJj3oP4I6e/B0zY
057VefYnIlfm5Jl4Uv36v9O7/kYSEYwudCruWz4uwXLpKoagpWNApez2MZQnkxGx
WwjLkG09AqzW+SOHhfuIjceTWdphRCOiLOwOil2BMs2Y/vlvTckmMZXpF1ugt2NA
hzbFSnP4f+tII/hjV4vlxVn3nJECoVOtpJa+zOGutK3EOPCsPCTGV50zvsArDf3D
p7eF/KFdz8p8abkwSWZBuVAWT4MwZR8xwSfRUE9oxCyMXnS5y0YJYGn1oQUG9ORf
3gKUs21ttI2LzZwr8LpFgZ6WUYb33h32RsPYskEGZASjEV0lZn7pgDR4WYfq3hUw
m+1qkXJOLurh7g2kfIMf
=gym/
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.