Debian Bug report logs - #1017996
dpkg: please provide `--force-really-unsafe-io` (or similar) option

Reply or subscribe to this bug.

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Ansgar <ansgar@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: dpkg: please provide `--force-really-unsafe-io` (or similar) option

Date: Tue, 23 Aug 2022 22:40:37 +0200

Package: dpkg
Version: 1.21.9
Severity: wishlist
Tags: upstream d-i

Hi,

please reconsider to provide a `--force-really-unsafe-io` (or similar)
option that skips the calls to `fsync()` & friends in dpkg.

I tried installing a larger package set (in stable), including
texlive-full, KDE, GCC and other packages. It took:

  Default dpkg: ~4h10m = 250m
  --force-unsafe-io: ~2h20m = 140m
  eatmydata: ~22m

So skipping all fsync() calls provides a speedup of 11 compared to
dpkg's defaults and still 6 compared to --force-unsafe-io. This is a
very noticable change.

The dpkg FAQ[1] asks users to use a different filesystem, non-default
mount options, `--force-unsafe-io` or `eatmydata`.

I don't think asking users to use a different filesystem is a good
option (and the installer doesn't ask in many cases); same for
non-default mount options. Doing so also has other downsides.

Options like `--force-unsafe-io` are okay (and d-i can and does use
it), but are suboptimal in many situations where one really does not
need any calls to `fsync`.  `eatmydata` is a bit of a hack and harder
to use (make sure apt-get calls a wrapper dpkg script or everything
calls apt-get/aptitude/* via some wrapper) or not possible (d-i
doesn't use eatmydata).  I think dpkg supporting to skip fsync on its
own is the better solution (compared to implementing eatmydata support
in d-i and other places).

There are various cases where this is useful:

- Initial installation (d-i or initial setup with ansible/puppet/scripts/*).
- Containers.
- CI systems.
- Anything else using throw-away systems (chroots, containers, VMs, ...).

Even dpkg itself uses eatmydata to avoid `fsync()`s (in .gitlab-ci.yml).

I know this was previously proposed and not implemented, but I think
the numer of cases where this is useful has increased over time and
will probably increase more in the future.

Ansgar

  [1]: https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slow_when_using_new_filesystems_such_as_btrfs_or_ext4.3F

Message #10 received at 1017996@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Ansgar <ansgar@debian.org>, 1017996@bugs.debian.org

Subject: Re: Bug#1017996: dpkg: please provide `--force-really-unsafe-io` (or similar) option

Date: Wed, 14 Sep 2022 05:14:50 +0200

Control: forcemerge 985888 -1

On Tue, 2022-08-23 at 22:40:37 +0200, Ansgar wrote:
> Package: dpkg
> Version: 1.21.9
> Severity: wishlist
> Tags: upstream d-i

> please reconsider to provide a `--force-really-unsafe-io` (or similar)
> option that skips the calls to `fsync()` & friends in dpkg.

I think I've mentioned this elsewhere, but the main blocker has been
lack of database taint tracking support. I do have a draft branch for
a --force-reckless-io.

> I tried installing a larger package set (in stable), including
> texlive-full, KDE, GCC and other packages. It took:
> 
>   Default dpkg: ~4h10m = 250m
>   --force-unsafe-io: ~2h20m = 140m
>   eatmydata: ~22m
> 
> So skipping all fsync() calls provides a speedup of 11 compared to
> dpkg's defaults and still 6 compared to --force-unsafe-io. This is a
> very noticable change.

All fsync()s from dpkg, which are not necessarily all fsync()s
performed indirectly by maintscripts/triggers.

Guillem

Send a report that this bug log contains spam.