-
-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[DO NOT MERGE] [TRACKER] Apt 3.1.x #24212
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
|
That part of @@ -611,8 +619,27 @@ struct pkgCache::DescFile
map_pointer<DescFile> NextFile;
/** \brief position in the file */
map_filesize_t Offset; // File offset
- /** @TODO document pkgCache::DescFile::Size */
- map_filesize_t Size;
+};
+ /*}}}*/
+// SourceVersion structure /*{{{*/
+/** \brief information for a single version of a source package
+
+ The version list is always sorted from highest version to lowest
+ version by the generator. Equal version numbers are either merged
+ or handled as separate versions based on the Hash value. */
+struct pkgCache::SourceVersion
+{
+ /** \brief unique sequel ID */
+ map_id_t ID;
+ /** \brief Group the source package belongs too */
+ map_pointer<pkgCache::Group> Group;
+ /** \brief complete version string */
+ map_stringitem_t VerStr;
+ map_pointer<Version> VersionList [[gnu::unavailable("not yet available")]];
+ map_pointer<SourceVersion> NextSourceVersion [[gnu::unavailable("not yet available")]];
+
+ /** \brief Private pointer */
+ map_pointer<void> d;https://salsa.debian.org/apt-team/apt/-/blob/3.0.0/apt-pkg/pkgcache.h?ref_type=tags#L638 |
|
Thanks for the help @licy183, I just saw this now. |
|
I'm doing on-device testing with the DEBs from this PR now. I have already encountered one issue. Warning: Could not execute pager - PagerSetup (2: No such file or directory)Edit: Okay this is "newly" introduced in 2.9.181, and defaults to Footnotes |
|
Okay, that fixed ~$ pkg up
Checking availability of current mirror:
[*] https://packages-cf.termux.dev/apt/termux-main: ok
CANNOT LINK EXECUTABLE "/data/data/com.termux/files/usr/lib/apt/methods/https": library "libseccomp.so" not found: needed by main executable
CANNOT LINK EXECUTABLE "/data/data/com.termux/files/usr/lib/apt/methods/https": library "libseccomp.so" not found: needed by main executable
CANNOT LINK EXECUTABLE "/data/data/com.termux/files/usr/lib/apt/methods/https": library "libseccomp.so" not found: needed by main executable
CANNOT LINK EXECUTABLE "/data/data/com.termux/files/usr/lib/apt/methods/https": library "libseccomp.so" not found: needed by main executable
Error: Method https has died unexpectedly!
Error: Sub-process https returned an error code (1)
Error: Method /data/data/com.termux/files/usr/lib/apt/methods/https did not start correctly
Error: Method https has died unexpectedly!
Error: Sub-process https returned an error code (1)
Error: Method /data/data/com.termux/files/usr/lib/apt/methods/https did not start correctly
Error: Method https has died unexpectedly!
Error: Sub-process https returned an error code (1)
Error: Method /data/data/com.termux/files/usr/lib/apt/methods/https did not start correctly
Error: Method https has died unexpectedly!
Error: Sub-process https returned an error code (1)
Error: Method /data/data/com.termux/files/usr/lib/apt/methods/https did not start correctly
Error: Failed to fetch https://packages-cf.termux.dev/apt/termux-main/dists/stable/InRelease
Error: Failed to fetch https://packages-cf.termux.dev/apt/termux-root/dists/root/InRelease
Error: Failed to fetch https://tur.kcubeterm.com/dists/tur-packages/InRelease
Error: Failed to fetch https://packages-cf.termux.dev/apt/termux-x11/dists/x11/InRelease
Error: Some index files failed to download. They have been ignored, or old ones used instead.
Notice: Some sources can be modernized. Run 'apt modernize-sources' to do so.Edit: looks like a missing dependency. ~$ apt modernize-sources
The following files need modernizing:
- /data/data/com.termux/files/usr/etc/apt/sources.list
- /data/data/com.termux/files/usr/etc/apt/sources.list.d/root.list
- /data/data/com.termux/files/usr/etc/apt/sources.list.d/tur.list
- /data/data/com.termux/files/usr/etc/apt/sources.list.d/x11.list
Modernizing will replace .list files with the new .sources format,
add Signed-By values where they can be determined automatically,
and save the old files into .list.bak files.
This command supports the 'signed-by' and 'trusted' options. If you
have specified other options inside [] brackets, please transfer them
manually to the output files; see sources.list(5) for a mapping.
For a simulation, respond N in the following prompt.
Rewrite 4 sources? [Y/n] N
Simulating only...
Modernizing /data/data/com.termux/files/usr/etc/apt/sources.list...
# Would write to: /data/data/com.termux/files/usr/etc/apt/sources.list.d/termux-main-stable.sources
# Modernized from /data/data/com.termux/files/usr/etc/apt/sources.list
Types: deb
URIs: https://packages-cf.termux.dev/apt/termux-main/
Suites: stable
Components: main
Signed-By:
Warning: Could not determine Signed-By for URIs: https://packages-cf.termux.dev/apt/termux-main/, Suites: stable
Modernizing /data/data/com.termux/files/usr/etc/apt/sources.list.d/root.list...
# Would write to: /data/data/com.termux/files/usr/etc/apt/sources.list.d/root.sources
Types: deb
URIs: https://packages-cf.termux.dev/apt/termux-root/
Suites: root
Components: stable
Signed-By:
Warning: Could not determine Signed-By for URIs: https://packages-cf.termux.dev/apt/termux-root/, Suites: root
Modernizing /data/data/com.termux/files/usr/etc/apt/sources.list.d/tur.list...
# Would write to: /data/data/com.termux/files/usr/etc/apt/sources.list.d/tur.sources
Types: deb
URIs: https://tur.kcubeterm.com/
Suites: tur-packages
Components: tur tur-on-device tur-continuous
Signed-By: /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/tur.gpg
Modernizing /data/data/com.termux/files/usr/etc/apt/sources.list.d/x11.list...
# Would write to: /data/data/com.termux/files/usr/etc/apt/sources.list.d/x11.sources
Types: deb
URIs: https://packages-cf.termux.dev/apt/termux-x11/
Suites: x11
Components: main
Signed-By:
Warning: Could not determine Signed-By for URIs: https://packages-cf.termux.dev/apt/termux-x11/, Suites: x11Edit: gonna have to look into that, we'll probably wanna migrate the repo files. |
Probably, seccomp need to be removed from cmake https://salsa.debian.org/apt-team/apt/-/blob/3.0.0/CMakeLists.txt?ref_type=tags#L137 |
It was missing as a dependency. |
libseccomp is still downloaded in CI and cmake picks it up. |
Yes. I force pushed the branch to fix that. |
|
I'm having some trouble wrapping my head around the new deb822 format for package sources (e.g. repos). If I run # ~$ tree $TERMUX__PREFIX/etc/apt
/data/data/com.termux/files/usr/etc/apt
├── apt.conf.d
│ └── 50apt-file.conf
├── preferences.d
├── sources.list
├── sources.list.bak
├── sources.list.d
│ ├── root.list.bak
│ ├── root.sources
│ ├── termux-main-stable.sources
│ ├── tur.list.bak
│ ├── tur.sources
│ ├── x11.list.bak
│ └── x11.sources
├── sources.list.dpkg-old
├── trusted.gpg
└── trusted.gpg.d
├── 2096779623.gpg -> /data/data/com.termux/files/usr/share/termux-keyring/2096779623.gpg
├── agnostic-apollo.gpg -> /data/data/com.termux/files/usr/share/termux-keyring/agnostic-apollo.gpg
├── grimler.gpg -> /data/data/com.termux/files/usr/share/termux-keyring/grimler.gpg
├── kcubeterm.gpg -> /data/data/com.termux/files/usr/share/termux-keyring/kcubeterm.gpg
├── landfillbaby.gpg -> /data/data/com.termux/files/usr/share/termux-keyring/landfillbaby.gpg
├── mradityaalok.gpg -> /data/data/com.termux/files/usr/share/termux-keyring/mradityaalok.gpg
├── termux-autobuilds.gpg -> /data/data/com.termux/files/usr/share/termux-keyring/termux-autobuilds.gpg
└── tur.gpg
5 directories, 20 filesAnd it's complaining about duplicates and missing Checking availability of current mirror:
[*] https://packages-cf.termux.dev/apt/termux-main: ok
Hit:1 https://turdl.kcubeterm.com tur-packages InRelease
Hit:2 https://packages-cf.termux.dev/apt/termux-main stable InRelease
Hit:3 https://packages-cf.termux.dev/apt/termux-root root InRelease
Hit:4 https://packages-cf.termux.dev/apt/termux-x11 x11 InRelease
All packages are up to date.
Warning: Target Packages (main/binary-aarch64/Packages) is configured multiple times in /data/data/com.termux/files/usr/etc/apt/sources.list:1 and /data/data/com.termux/files/usr/etc/apt/sources.list.d/termux-main-stable.sources:1
Warning: Target Packages (main/binary-all/Packages) is configured multiple times in /data/data/com.termux/files/usr/etc/apt/sources.list:1 and /data/data/com.termux/files/usr/etc/apt/sources.list.d/termux-main-stable.sources:1
Warning: Target Contents-deb (main/Contents-aarch64) is configured multiple times in /data/data/com.termux/files/usr/etc/apt/sources.list:1 and /data/data/com.termux/files/usr/etc/apt/sources.list.d/termux-main-stable.sources:1
Warning: Target Contents-deb (main/Contents-all) is configured multiple times in /data/data/com.termux/files/usr/etc/apt/sources.list:1 and /data/data/com.termux/files/usr/etc/apt/sources.list.d/termux-main-stable.sources:1
Warning: Target Packages (main/binary-aarch64/Packages) is configured multiple times in /data/data/com.termux/files/usr/etc/apt/sources.list:1 and /data/data/com.termux/files/usr/etc/apt/sources.list.d/termux-main-stable.sources:1
Warning: Target Packages (main/binary-all/Packages) is configured multiple times in /data/data/com.termux/files/usr/etc/apt/sources.list:1 and /data/data/com.termux/files/usr/etc/apt/sources.list.d/termux-main-stable.sources:1
Warning: Target Contents-deb (main/Contents-aarch64) is configured multiple times in /data/data/com.termux/files/usr/etc/apt/sources.list:1 and /data/data/com.termux/files/usr/etc/apt/sources.list.d/termux-main-stable.sources:1
Warning: Target Contents-deb (main/Contents-all) is configured multiple times in /data/data/com.termux/files/usr/etc/apt/sources.list:1 and /data/data/com.termux/files/usr/etc/apt/sources.list.d/termux-main-stable.sources:1
Notice: Missing Signed-By in the sources.list(5) entry for 'https://packages-cf.termux.dev/apt/termux-main'
Notice: Missing Signed-By in the sources.list(5) entry for 'https://packages-cf.termux.dev/apt/termux-root'
Notice: Missing Signed-By in the sources.list(5) entry for 'https://packages-cf.termux.dev/apt/termux-x11'
Summary:
Upgrading: 0, Installing: 0, Removing: 0, Not Upgrading: 0 |
|
We'll probably need to change the apt sources handling in:
I'll prep a PR for |
|
That's the repos moved over to |
2ef3a0d to
d45e3d3
Compare
|
Also updated |
|
|
4940db5 to
4f1bcf3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This gets us up to date with master again and all packages building.
The already conducted code reviews should still hold.
So that only leaves the rollback plan as the final unfinished piece.
Actually, did we ever work out if we want to roll updating termux-tools to 1.47.0 into this PR as well?
- That would mean additional revalidation of
termux/termux-tools#165 and
termux/termux-tools#167
| #!@TERMUX_PREFIX@/bin/sh | ||
| # shellcheck shell=dash | ||
|
|
||
| set -eu | ||
|
|
||
| is_installed() { | ||
| case "$(dpkg-query -W -f '${db:Status-Status}' "$1" 2>/dev/null)" in | ||
| 'not-installed'|'config-files') return 1;; | ||
| esac | ||
| } | ||
|
|
||
| was_upgraded() { | ||
| case "$1" in | ||
| # Versions of the rebuilt packages prior to the Apt 3.x merge | ||
| 'apt') OLD_VERSION='2.8.1-1';; | ||
| 'dpkg') OLD_VERSION='1.22.6-1';; | ||
| 'apt-file') OLD_VERSION='3.3';; | ||
| 'libapt-pkg-perl') OLD_VERSION='0.1.40-10';; | ||
| 'nala') OLD_VERSION='0.16.0';; | ||
| 'synaptic') OLD_VERSION='0.91.7-1';; | ||
| 'termux-tools') OLD_VERSION='1.46.0+really1.45.0';; | ||
| *) # How did we get here? | ||
| echo "Package '$1' isn't covered by this script." | ||
| return 1 | ||
| ;; | ||
| esac | ||
|
|
||
| INSTALLED_VERSION="$(dpkg-query -W -f '${Version}' "$1")" | ||
|
|
||
| dpkg --compare-versions "$INSTALLED_VERSION" le "$OLD_VERSION" | ||
| } | ||
|
|
||
| download_backup() { | ||
| PKG_NAME="$1" | ||
| was_upgraded "$PKG_NAME" || return 1 | ||
|
|
||
| TERMUX_ARCH="@TERMUX_ARCH@" | ||
| OUTPUT_FILE"${TMPDIR:-@TERMUX_PREFIX@/tmp}/${PKG_NAME}_${OLD_VERSION}_${TERMUX_ARCH}.deb" | ||
| PKG_URL="$MIRROR_URL" | ||
| curl -L "$PKG_URL" -o "$OUTPUT_FILE" | ||
| } | ||
|
|
||
| downgrade() { | ||
| # Check that 'dpkg' is in a sane enough state to mark itself 'ok' | ||
| case "$(dpkg-query -W -f '${Status}' dpkg)" in | ||
| *' ok '*);; | ||
| *) # If 'dpkg' isn't in a known 'ok' state something has gone very wrong. | ||
| echo "Something is seriously wrong with 'dpkg'." | ||
| echo "Unable to proceed with automatic restore." | ||
| exit 1 | ||
| ;; | ||
| esac | ||
|
|
||
| for package in 'dpkg' 'apt' 'libapt-pkg-perl' 'apt-file' 'nala' 'synaptic' 'termux-tools'; do | ||
| echo "Checking: '$package'" | ||
|
|
||
| # Do we even have this package? | ||
| is_installed "$package" || continue | ||
|
|
||
| # Download the old DEB if needed | ||
| download_backup "$package" | ||
|
|
||
| done | ||
| # Force install the old version. | ||
| dpkg -i "$OUTPUT_FILE" | ||
| } | ||
|
|
||
| downgrade |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd appreciate some suggestions for the restore script.
I do not recall how far I got with it, but it isn't yet fully implemented.
Notably, the way we'd want to handle keeping the old versions of the packages in the PR around hasn't been hashed out yet, and the script isn't currently being installed as part of the apt package.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Personally, if the decision were up to me, I would prefer to treat apt 3.0+ as a normal rolling-release package upgrade and not include a "rollback script", which is (in my opinion) not idiomatic to the general philosophy of rolling-release distributions.
In other rolling-release distributions, there is an expectation that, if users do not already know how, but encounter any unfixable problem with the new package versions, users should learn how to manually build old packages on another system and perform the rollback manually by transferring and force-reinstalling the old package versions in order to fix their broken environment.
There have been many instances throughout history when a broken pacman has occurred for almost the entire user base of Arch Linux, and it has been necessary for users of Arch Linux to manually retrieve and force-reinstall an older pacman to recover.
I'm also not sure myself how to safely implement a "fully automated" rollback script of this kind in any rolling-release distribution, because, since the exact versions of dependencies in rolling-release distributions are never locked to any "baseline release" set of package versions that could be expected, it is, as far as I know, impossible to predict exactly what dependency-related errors will occur for any given user when force-reinstalling older package versions, and that is something that users just have to figure out on their own using standard problem-solving steps for rolling-release distributions.
However, if the decision is not up to me, then I would be happy to test any rollback script prototype on my devices once it is ready to run, and report the results, so that it could at least be a rollback script that has consistent and possibly-useful behavior, even if it's not possible for it to handle every single edge case that could occur.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Notably, the way we'd want to handle keeping the old versions of the packages in the PR around hasn't been hashed out yet
Arch Linux has an archive for all older packages of all older versions released anytime in the last several months or years.
That is actually part of Arch Linux's official wiki page documenting how to downgrade packages in Arch Linux.
however, the important reason that works in Arch Linux is because they actually store an archive of all their packages, not just a few, and unlike Debian sid and Termux, they don't delete old packages arbitrarily, but have many old packages in their archive going back a really long time.
That makes it simple for Arch Linux users to perform the process of manually rolling back packages and all their dependencies, since they can try the package first -> if the package at the version they need reports a runtime error because of incorrect dependencies, they can go find the dependencies at the needed versions in the package archive -> repeat,
The equivalent steps in Termux (since Termux does not archive all old packages) are:
- to first download two clones of the termux-packages git repository
- use
git checkoutto rollback one of the clones to the last commit that contained the desired package(s) - manually override the latest-clone with the needed packages copied and pasted from the rolled-back clone
- build all of the desired old package versions using the
scriptsfolder and other packages present in the latest-clone - and then transfer the newly-rebuilt old package versions to the Termux device where they are needed, and manually reinstall them there,
- repeat until no more errors
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Additionally skill set of average arch users and Termux users would be slightly different ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Additionally skill set of average arch users and Termux users would be slightly different ;)
I agree, but part of my point is that fully-automated rollback scripts that handle every single possible edge-case are, as far as I know, impossible to implement in rolling-release distributions because of properties shared between all rolling-release distributions, so unless a "termux-packages 'release' package set for last apt 2.X" is made and fully backed up so that it can be treated as the "stable reference point" for rolling-back to (simulating/implementing the effects of a non-rolling-release distro), it might be impossible for users encountering any hypothetical issue with the apt package (or any other package bump, since the same considerations technically apply for any upgrade of any package in rolling-release distributions) to fix their issue without learning the troubleshooting and recovery steps that are shared between all rolling-release distributions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do publish bootstraps which contain all dependencies of apt at snapshots confirmed to work going back for quite a while, here:
https://github.com/termux/termux-packages/releases
If a rollback script is important, maybe one of these could be selected and used as the download from which the script could extract and install a working copy of apt 2.X and its dependencies?
There could be other problems with that approach, but it's one idea since the bootstraps might represent functioning "releases" of all the necessary components to run apt 2.X (just not, unlike real non-rolling-release distros, any other packages that are not already included in the bootstraps, which could easily result in any non-bootstrap packages the user has installed becoming broken after the rollback, but maybe if the goal is recovery of apt alone, those aren't as important in the opinion of some - in my opinion, preserving the functionality of all other packages the user has installed is important, but others might disagree).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TL;DR of all I wrote is, that in my opinion, a "rollback script" in Termux is unlikely to work reliably and could easily cause more problems than it fixes, in a cascading whack-a-mole effect that the more hypothetical errors it attempts to fix, the more errors are likely to appear. Others might have other opinions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Extracting package and dependencies from bootstraps would be too much work as they would exist as a prefix directory. It might be possible to create a pool-archive directory on the repo with all the required debs for apt and recursive dependencies that "currently exist", that state is currently working, that should be enough to restore, without running into dependency linking issues. It could break other packages that may rely on newer dependencies, but we could fix the apt 3.0 issue that was found and release an update, after which users could upgrade all packages using apt 2.0 to get all packages working.
The goal should be to provide a reasonable/basic recovery mechanism, not cater to every possibly scenario. The emergency-restore.sh.in looks fine from a quick look, would have to check it more closely and ensure all dependencies are restored, etc and change url to pool-archive, etc.
Quoting @agnostic-apollo on may 27th in this thread:
if the situation has changed since then and that comment no longer applies, agnostic-apollo could write an updated response to explain. |
|
Some other notes regarding very minor changes to very heavy changes to the behavior of |
Same as before, do it from here. |
actually there is some other review I might have for this PR about other stuff, but I have posted a lot of messages here recently, so to avoid spamming more text walls, I will wait until the other discussion topics are resolved before starting new discussions about other parts of the PR. |
4f1bcf3 to
74e98e4
Compare
Co-authored-by: Jia Yuan Lo <jylo06g@gmail.com> Co-authored-by: Robert Kirkman <rkirkman@termux.dev> Co-authored-by: Twaik Yont <9674930+twaik@users.noreply.github.com>
Co-authored-by: Jia Yuan Lo <jylo06g@gmail.com>
939e571 to
59637f0
Compare
Co-authored-by: Chongyun Lee <45286352+licy183@users.noreply.github.com> Co-authored-by: Biswapriyo Nath <nathbappai@gmail.com>
| [1]: https://github.com/llvm/llvm-project/pull/90152 | ||
|
|
||
| --- a/apt-pkg/cacheset.h | ||
| +++ b/apt-pkg/cacheset.h |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have discovered that NDK r29
appears that it will cause this patch to not be necessary anymore
|
It might be good to actually articulate this. We can probably part this out into a couple different batches and do the apt bump as the final step if we want to take a more gradual approach to this instead of adding a bunch of important changes all at once with this PR. |
Currently fails with: