Getting `termux-tools` ready for `apt` 3.0.0 #165

TomJo2000 · 2025-04-23T22:25:39Z

Part of [DO NOT MERGE] [TRACKER] Apt 3.1.x termux-packages#24212

Apt 3.0.0 makes the "multiline" deb822 format the standard for package sources.
This PR aims to introduce support for this repo source format in termux-tools.

This initial commit only adds support in pkg.
I plan to add support in termux-info tomorrow, however that turned into half a refactor and I'm tired.

The changes to pkg introduce two possible return values from has_repo.

deb822, the new standard format (determined by the .sources file extension)
legacy, the old oneline format (determined by the .list file extension)
Support for the legacy oneline .list format isn't slated to be removed until 2029¹,
so it is kept for backwards compatibility in the meantime.
An empty return string ("") indicates that the repo is not present in either format.

I also fixed a couple shellcheck warnings and some indentation.
I may do a more extensive refactor of pkg as part of this PR later.

https://man.archlinux.org/man/sources.list.5#ONE-LINE-STYLE_FORMAT ↩

robertkirkman · 2025-04-23T22:45:20Z

I tested this, and I think it is working for me. There is something strange-looking about the indentation in some places, but I think you might already have noticed, and will be able to resolve that.

TomJo2000 · 2025-04-24T15:32:36Z

There is something strange-looking about the indentation in some places, but I think you might already have noticed, and will be able to resolve that.

Haven't noticed anything of the sort.
There's some over-indenting from conditionals that could be turned into guard clauses, but the indentation as is seems correct to me.
If you have any specific examples please do point them out.

robertkirkman · 2025-04-24T18:12:40Z

Oh ok, sure, here is one of them

https://github.com/TomJo2000/termux-tools-prs/blob/924edb6859681eb1b57da7f15d2fab9307a58f18/scripts/pkg.in#L432-L434

TomJo2000 · 2025-04-24T18:15:49Z

That part was already lined up with the line above, it just used spaces instead of tabs for indentation, which doesn't match the rest of the script.

robertkirkman · 2025-04-24T18:20:04Z

yes I understand that, but all I am trying to explain is that it is no longer "lined up" after this. This is how it looks on my screen.

TomJo2000 · 2025-04-24T18:22:16Z

Looks fine on my device.

Probably an issue with your set tapstop?

TomJo2000 · 2025-04-24T18:23:35Z

There's snapped it back to 3 tabs.

robertkirkman · 2025-04-24T18:25:13Z

The problem is also visible in your screenshot, but then you fixed it in a force push so I guess you are now able to see the problem with how it was before

TomJo2000 · 2025-04-24T18:40:24Z

The problem is also visible in your screenshot

Then I'm not seeing what you meant.

robertkirkman · 2025-04-24T18:42:37Z

I screenshotted your screenshot and drew a line on it, in my opinion the line should be straight BUT it is ok and does not matter very much because:

it is purely whitespace-related and does not affect the script operation at all
you already fixed it by force pushing a commit which makes the line straight

TomJo2000 · 2025-04-24T18:44:22Z

It was aligned with un*|rem*|rm|del*, so I kept it aligned that way when I fixed the indentation from spaces to tabs.
But yeah it should just be straight.

robertkirkman · 2025-04-24T18:47:42Z

On my screen, the current change does not change the straight shape of the "line", but this discrepancy can come as a side effect of the different settings of tab width for each user, like you mentioned.

scripts/pkg.in

TomJo2000 · 2025-04-27T16:27:17Z

Just finished testing the termux-info changes.
The new version does away with the previous special casing for the sources.list file outside the sources.list.d directory by consolidating the find into a for loop, glob and case for classification instead.

TomJo2000 · 2025-05-04T21:35:52Z

Seeing as [DO NOT MERGE] [TRACKER] Apt 3.1.x termux-packages#24212 is just about ready,
and as I've mentioned in that PR's reviews,
I think we should keep the Apt 3 transition focused and leave additional structural changes to related packages and tooling for later revisions.

Grimler91 · 2025-05-08T07:18:59Z

For some reason that I have not figured out termux-change-repo/pkg empties my .sources files:

$ cat $PREFIX/etc/apt/sources.list.d/main.sources
# The main termux repository, with cloudflare cache
Types: deb
URIs: https://packages-cf.termux.dev/apt/termux-main/
Suites: stable
Components: main
Signed-By: /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/grimler.gpg, /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/termux-autobuilds.gpg

# The main termux repository, without cloudflare cache
#Types: deb
#URIs: https://packages-cf.termux.dev/apt/termux-main/
#Suites: stable
#Components: main
#Signed-By: /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/grimler.gpg, /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/termux-autobuilds.gpg
$ termux-change-repo
[ ... ]
[*] Mirror grimler.se selected
[*] pkg --check-mirror update
grep: /data/data/com.termux/files/usr/etc/apt/sources.list: No such file or directory
Testing the available mirrors:
[*] (4) https://grimler.se/termux/termux-main: ok
Picking mirror: (1) /data/data/com.termux/files/usr/etc/termux/mirrors/europe/grimler.se
Hit:1 https://turdl.kcubeterm.com tur-packages InRelease
All packages are up to date.
$ more $PREFIX/etc/apt/sources.list.d/* $PREFIX/etc/apt/sources.list
::::::::::::::
/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources
::::::::::::::
::::::::::::::
/data/data/com.termux/files/usr/etc/apt/sources.list.d/root.sources
::::::::::::::
::::::::::::::
/data/data/com.termux/files/usr/etc/apt/sources.list.d/tur.sources
::::::::::::::
# A single and trusted place for all unofficial/less popular termux packages
Types: deb
URIs: https://tur.kcubeterm.com/
Suites: tur-packages
Components: tur tur-on-device tur-continuous
Signed-By: /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/tur.gpg
::::::::::::::
/data/data/com.termux/files/usr/etc/apt/sources.list.d/x11.sources
::::::::::::::

Grimler91 · 2025-05-08T07:22:16Z

scripts/pkg.in

+		source "$selected_mirror"
+
+		case "$(has_repo main)" in
+			'deb822') sed -ni -e "s|URIs:.*|URIs: $MAIN|" "@TERMUX_PREFIX@/etc/apt/sources.list.d/main.sources";;


Might be better to keep original *.sources in $PREFIX/etc/termux/ (or another suitable place), and then sed -n -e "s|URIs:.*|URIs: $MAIN|" "$some_path" > "@TERMUX_PREFIX@/etc/apt/sources.list.d/main.sources". That would make it possible to recover even if *.sources are emptied as in my issue

Seems like it is the -n option that empties the .sources files:

$ cat main.sources # The main termux repository, with cloudflare cache Types: deb URIs: https://mirror.accum.se/mirror/termux.dev/termux-main Suites: stable Components: main Signed-By: /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/grimler.gpg, /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/termux-autobuilds.gpg # The main termux repository, without cloudflare cache #Types: deb #URIs: https://mirror.accum.se/mirror/termux.dev/termux-main #Suites: stable #Components: main #Signed-By: /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/grimler.gpg, /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/termux-autobuilds.gpg $ sed -ni -e "s|URIs:.*|URIs: https://mirror.accum.se/mirror/termux.dev/termux-main|" main.sources $ cat main.sources [ empty ]

Yep sorry that was leftover from testing, thought I had already removed those.
-n disables the implied p command causing the pattern space to be printed automatically.

I was using it during testing because I using --debug to step through the sed command.

# ~$ sed --debug -n -e "s/URIs:.*/URIs: testing/" $TERMUX__PREFIX/etc/apt/sources.list.d/main.sources SED PROGRAM: s/URIs:.*/URIs: testing/ INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 1 PATTERN: # The main termux repository, with cloudflare cache COMMAND: s/URIs:.*/URIs: testing/ PATTERN: # The main termux repository, with cloudflare cache END-OF-CYCLE: INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 2 PATTERN: Types: deb COMMAND: s/URIs:.*/URIs: testing/ PATTERN: Types: deb END-OF-CYCLE: INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 3 PATTERN: URIs: https://packages-cf.termux.dev/apt/termux-main/ COMMAND: s/URIs:.*/URIs: testing/ MATCHED REGEX REGISTERS regex[0] = 0-53 'URIs: https://packages-cf.termux.dev/apt/termux-main/' PATTERN: URIs: testing END-OF-CYCLE: INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 4 PATTERN: Suites: stable COMMAND: s/URIs:.*/URIs: testing/ PATTERN: Suites: stable END-OF-CYCLE: INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 5 PATTERN: Components: main COMMAND: s/URIs:.*/URIs: testing/ PATTERN: Components: main END-OF-CYCLE: INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 6 PATTERN: Signed-By: /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/grimler.gpg, /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/termux-autobuilds.gpg COMMAND: s/URIs:.*/URIs: testing/ PATTERN: Signed-By: /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/grimler.gpg, /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/termux-autobuilds.gpg END-OF-CYCLE: INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 7 PATTERN: COMMAND: s/URIs:.*/URIs: testing/ PATTERN: END-OF-CYCLE: INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 8 PATTERN: # The main termux repository, without cloudflare cache COMMAND: s/URIs:.*/URIs: testing/ PATTERN: # The main termux repository, without cloudflare cache END-OF-CYCLE: INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 9 PATTERN: #Types: deb COMMAND: s/URIs:.*/URIs: testing/ PATTERN: #Types: deb END-OF-CYCLE: INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 10 PATTERN: #URIs: https://packages-cf.termux.dev/apt/termux-main/ COMMAND: s/URIs:.*/URIs: testing/ MATCHED REGEX REGISTERS regex[0] = 1-54 'URIs: https://packages-cf.termux.dev/apt/termux-main/' PATTERN: #URIs: testing END-OF-CYCLE: INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 11 PATTERN: #Suites: stable COMMAND: s/URIs:.*/URIs: testing/ PATTERN: #Suites: stable END-OF-CYCLE: INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 12 PATTERN: #Components: main COMMAND: s/URIs:.*/URIs: testing/ PATTERN: #Components: main END-OF-CYCLE: INPUT: '/data/data/com.termux/files/usr/etc/apt/sources.list.d/main.sources' line 13 PATTERN: #Signed-By: /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/grimler.gpg, /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/termux-autobuilds.gpg COMMAND: s/URIs:.*/URIs: testing/ PATTERN: #Signed-By: /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/grimler.gpg, /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/termux-autobuilds.gpg END-OF-CYCLE:

scripts/pkg.in

Grimler91 · 2025-05-08T17:08:23Z

scripts/pkg.in

+		source "$selected_mirror"
+
+		case "$(has_repo main)" in
+			'deb822') sed -ni -e "s|URIs:.*|URIs: $MAIN|" "@TERMUX_PREFIX@/etc/apt/sources.list.d/main.sources";;


Seems like it is the -n option that empties the .sources files:

$ cat main.sources # The main termux repository, with cloudflare cache Types: deb URIs: https://mirror.accum.se/mirror/termux.dev/termux-main Suites: stable Components: main Signed-By: /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/grimler.gpg, /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/termux-autobuilds.gpg # The main termux repository, without cloudflare cache #Types: deb #URIs: https://mirror.accum.se/mirror/termux.dev/termux-main #Suites: stable #Components: main #Signed-By: /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/grimler.gpg, /data/data/com.termux/files/usr/etc/apt/trusted.gpg.d/termux-autobuilds.gpg $ sed -ni -e "s|URIs:.*|URIs: https://mirror.accum.se/mirror/termux.dev/termux-main|" main.sources $ cat main.sources [ empty ]

Grimler91 · 2025-05-08T17:33:04Z

I opened a PR: github.com/TomJo2000/pull/2 with fixes for my comments above, and also fixed so it can find current_mirror in deb822 format as well

TomJo2000 · 2025-05-09T15:50:13Z

I opened a PR: github.com/TomJo2000/pull/2 with fixes for my comments above, and also fixed so it can find current_mirror in deb822 format as well

Those changes have been manually merged into 4ba21f3.

Grimler91 · 2025-05-10T09:53:28Z

Those changes have been manually merged into 4ba21f3.

Thanks! Here's another PR which checks if both main.sources and sources.list exists, and deletes sources.list if main.sources seems to be a valid mirror: TomJo2000#3
If we get something like that in then I am happy and will gladly merge this!

TomJo2000 · 2025-05-10T10:24:32Z

Thanks! Here's another PR which checks if both main.sources and sources.list exists, and deletes sources.list if main.sources seems to be a valid mirror: TomJo2000#3 If we get something like that in then I am happy and will gladly merge this!

The way I was gonna handle that is for each *-repo package (and apt) to move their old .list entry to a .list.dpkg-old file instead.
That way each repo only modifies its own sources.

Grimler91 · 2025-05-10T12:00:44Z

@TomJo2000 right, that does sound better!

TomJo2000 · 2025-05-10T15:37:59Z

@TomJo2000 right, that does sound better!

I have now pushed those changes to the postinst scripts to [DO NOT MERGE] [TRACKER] Apt 3.1.x termux-packages#24212
The general structure for the message is:

[ -f "$TERMUX_PREFIX/etc/apt/sources.list.d/{{repo}}.list" ] && {
	echo "Legacy repo file found for '{{repo}}' repository."
	echo "Migrating sources.list.d/{{repo}}.list to sources.list.d/{{repo}}.sources"
	mv -v "$TERMUX_PREFIX"/etc/apt/sources.list.d/{{repo}}.list{,.dpkg-old}
	echo "A backup copy of the old {{repo}}.list repo file"
	echo "has been saved to $TERMUX_PREFIX/etc/apt/sources.list.d/{{repo}}.list.dpkg-old"
}

(apt's does of course check for $TERMUX_PREFIX/etc/apt/sources.list instead)
It's currently duplicated for each repo, I don't think it's worth the additional effort to make it DRY.

Grimler91

This PR looks good to me, I will aim at merging and releasing it tomorrow evening unless there are more questions/comments by then!

TomJo2000 · 2025-05-10T21:37:08Z

This PR looks good to me, I will aim at merging and releasing it tomorrow evening unless there are more questions/comments by then!

We can merge this PR ahead of merging Apt 3.0.1,
I'd like to run final validation and testing tonight so we're ready to merge everything tomorrow.

scripts/pkg.in

scripts/termux-info.in

agnostic-apollo · 2025-05-11T07:42:34Z

Looks good otherwise.

scripts/termux-info.in

scripts/pkg.in

scripts/termux-info.in

agnostic-apollo · 2025-05-11T14:30:03Z

scripts/pkg.in

+				sed -i -e "s|URIs:.*|$(sed_escape_replacement "URIs: $ROOT")|" \
+				"@TERMUX_PREFIX@/etc/apt/sources.list.d/root.sources"
+			;;
+			'legacy') echo "deb $ROOT root stable" > @TERMUX_PREFIX@/etc/apt/sources.list.d/root.list;;


Derp, yeah missed that one.

Checked and quoted all remaining instances.
https://github.com/termux/termux-tools/compare/7db010555f20e228d8e3d3301c2f93f3070d4e77..aa8327da102febe132be2378a1797e7bb723d21a

Grimler91 · 2025-05-11T19:25:56Z

Needs one more update in update_apt_cache, where we run: sources_modified=$(last_modified "@TERMUX_PREFIX@/etc/apt/sources.list"), I will fix that and get this in

Co-authored-by: Henrik Grimler <grimler@termux.dev> Co-authored-by: agnostic-apollo <agnosticapollo@gmail.com>

Grimler91 · 2025-05-11T19:41:46Z

Thanks!

TomJo2000 force-pushed the migrate-to-deb822-format branch from 924edb6 to 8caa682 Compare April 24, 2025 18:23

twaik reviewed Apr 26, 2025

View reviewed changes

scripts/pkg.in Show resolved Hide resolved

Grimler91 reviewed May 8, 2025

View reviewed changes

TomJo2000 mentioned this pull request May 8, 2025

[DO NOT MERGE] [TRACKER] Apt 3.1.x termux/termux-packages#24212

Draft

Grimler91 reviewed May 8, 2025

View reviewed changes

TomJo2000 force-pushed the migrate-to-deb822-format branch from b26d819 to 8246af6 Compare May 9, 2025 15:47

Grimler91 approved these changes May 10, 2025

View reviewed changes