Thanks for this. The relevant issue was termux/termux-packages#11091 (comment)

I will test later, looks fine, other than missing else condition for if [ -n "$selected_mirror" ]; then statement and main fosshost repo would be better to be under default repo in single mirrors list.

Creating the menu for selecting a single mirror is a bit slow, might be nice to decrease the overhead there somehow

Weird, how slow?

Also any reason why you didn't preserve history?

Weird, how slow?

On my old arm tablet it takes maybe 2 seconds, its noticeable.

Also any reason why you didn't preserve history?

Me being lazy. There are between 200 and 300 commits affecting these scripts, with various conflicts that would have to be resolved, I don't think that's something I want to spend time on right now anyways but can perhaps revisit it once I'm done with some of the other stuff I am doing at the moment

Bug: If TERMUX_PKG_NO_MIRROR_SELECT=1 is set in advance, the mirror group selection will not take effect.

(This is the F-droid version of termux, I just added an imageview control to display the background, there is no difference from the original in other places.)

Creating the menu for selecting a single mirror is a bit slow, might be nice to decrease the overhead there somehow.

I think it can be split into another file.

Bug:

[*] Mirror mirrors.pku.edu.cn.dpkg-new selected      
  [*] pkg --check-mirror update
Testing the available mirrors:
/data/data/com.termux/files/usr/bin/pkg: line 99: [: missing `]'
[*] (1) https://mirrors.pku.edu.cn/termux/termux-main/: ok

/data/data/com.termux/files/usr/bin/pkg: line 99: [: missing `]'                                              Picking mirror: (0) /data/data/com.termux/files/usr/etc/termux/mirrors/china/mirrors.pku.edu.cn.dpkg-new

[*] Mirror mirrors.ustc.edu.cn selected                
[*] pkg --check-mirror update
Testing the available mirrors:                         /data/data/com.termux/files/usr/bin/pkg: line 99: [: missing `]'                                              [*] (1) https://mirrors.ustc.edu.cn/termux/apt/termux-main: bad                                               Hit:1 https://turdl.kcubeterm.com tur-packages InRelease                                                      Ign:2 https://mirrors.tuna.tsinghua.edu.cn/termux/apt/termux-main stable InRelease                            Get:3 https://mirrors.tuna.tsinghua.edu.cn/termux/apt/termux-x11 x11 InRelease [14.0 kB]
Get:2 https://mirrors.tuna.tsinghua.edu.cn/termux/apt/termux-main stable InRelease [14.0 kB]
Get:4 https://mirrors.tuna.tsinghua.edu.cn/termux/apt/termux-x11 x11/main aarch64 Packages [103 kB]
Get:5 https://mirrors.tuna.tsinghua.edu.cn/termux/apt/termux-main stable/main aarch64 Packages [432 kB]
Fetched 549 kB in 36s (15.2 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

Repeatedly select two ways, invalid when selecting a single mirror.

Bug:

[*] Mirror mirrors.pku.edu.cn.dpkg-new selected      
  [*] pkg --check-mirror update
Testing the available mirrors:
/data/data/com.termux/files/usr/bin/pkg: line 99: [: missing `]'
[*] (1) https://mirrors.pku.edu.cn/termux/termux-main/: ok

/data/data/com.termux/files/usr/bin/pkg: line 99: [: missing `]'                                              Picking mirror: (0) /data/data/com.termux/files/usr/etc/termux/mirrors/china/mirrors.pku.edu.cn.dpkg-new

Alright, the '[' issue was fixed at the same time as addressing @agnostic-apollo's comments. *.dpkg-new has been added as a pattern to ignore

I will test later, looks fine, other than missing else condition for if [ -n "$selected_mirror" ]; then statement

And now fallback is default mirror as well. Changed it to an error message and exit instead.

main fosshost repo would be better to be under default repo in single mirrors list.

Yeah, agree, but can't think of a convenient way to sort the find output to make that happen

failed to get access selected mirror

None of the mirrors are accessible would be better. Issue would occur if all mirrors are bad and get unset. Check with internet off.

but can't think of a convenient way to sort the find output to make that happen

Add it manually after default and skip it in the loop.

None of the mirrors are accessible would be better. Issue would occur if all mirrors are bad and get unset. Check with internet off.

Message is updated, works as it should when I test by subscribing to a dummy, non-existing mirror. Mirror order still has to be fixed

I have pushed https://github.com/termux/termux-tools/tree/full-history with complete history. Had you just run following when creating repo, wouldn't have add to waste time cherry picking and rebasing... Alas... ;)

Doesn't have any diffs from master, you can force push to master after renaming it locally.

git log --pretty=email --patch-with-stat --reverse --full-index --binary -- packages/termux-tools > termux-tools.patch
cd ../termux-tools
git am --committer-date-is-author-date ../termux-packages/termux-tools.patch

There was an empty patch issue while running git am for https://github.com/termux/termux-packages/commits/8d57afa4b0b8f4179e8ef65c82fd30575aab6cf1/packages/termux-tools, commit has two parents and dual 0.61 commits, which required a rebase, hence dates are wrong after it. Can be redone I guess by manually fixing termux-tools.patch file and reapplying, we have latest commits anyways after 1.24 that can be cherrypicked directly now.

Fixed branch at https://github.com/termux/termux-tools/tree/full-history-redo

I have pushed https://github.com/termux/termux-tools/tree/full-history with complete history. Had you just run following when creating repo, wouldn't have add to waste time cherry picking and rebasing... Alas... ;)

Doesn't have any diffs from master, you can force push to master after renaming it locally.

git log --pretty=email --patch-with-stat --reverse --full-index --binary -- packages/termux-tools > termux-tools.patch
cd ../termux-tools
git am --committer-date-is-author-date ../termux-packages/termux-tools.patch

Thanks git sensei, several options there I am not familiar with! I went through your branch and removed all references to build.sh: https://github.com/termux/termux-tools/tree/full-history-redo-merge, giving 262 commits instead of 338.

git sensei

That is so not true! ;)

But thanks!

I went through your branch and removed all references to build.sh: https://github.com/termux/termux-tools/tree/full-history-redo-merge, giving 262 commits instead of 338.

Is there any specific reasons lesser commits would be better? The build.sh was very important part before and would have important changes that commits messages would have been for. I even modified a few of your post merge commits in which you created dedicated files that were previously being generated by build.sh to show that they were previously a part of build.sh.

https://github.com/termux/termux-tools/commits/full-history-redo?after=a3048f59758ccf85acbf227201622b440e587396+0&branch=full-history-redo&qualified_name=refs%2Fheads%2Ffull-history-redo

Yeah, my branch might need more cleaning and re-writing of commit messages to make sense. I'll go with your branch and force push over main so that we can start getting stuff into the new repo

Great. Holler at me when everything is final and I'll test.

Great work though overall!

Should be ready for testing again, mirror order has been fixed. Apart from slowness on single mirror selection I am rather happy with it

I don't think 2d082e7 should be done since user may not have any main source set and in that case we should not be automatically creating one. Secondly, single mirror requires it to be done with termux-change-repo, but user may have some custom mirror that's not in the list of approved mirrors and would have to manually create symlink, which is not user friendly.

Also can you update termux/termux-packages#11393 with this branch so that debs get built online. Thanks.

Another thing I want implemented is a blacklisted mirrors list file that is hosted on fosshost and as fallback on github as well in case its inaccessible, and file possibly signed with our key. This should contain all the mirrors that must not be used or whose domains have been freed or url sub path changed, like bintray, packages.termux.org, kcubeterm.me, recent one, etc. This file should likely automatically be downloaded on termux app start and incrementally as well to $PREFIX/etc/apt/blacklisted-mirrors.list file. The mirrors in these files should be checked by termux-change-repo and excluded from showing and pkg should exclude it too from mirrors group and switch to default mirror if none remain or single mirror is blacklisted.

An additional default mirror site file should also be hosted, that is used by the app to replace any blacklisted mirror from existing apt sources list files so that if user uses apt instead of pkg, then blacklisted mirrors don't get used. The default mirror should also be used by pkg script as default.

With the above, the blacklisted mirrors will be independent of whether users updates termux-tools or the app or has old bootstrap and we hopefully won't get any (or lower) broken mirror issues anymore. Hopefully, fosshost and github will be accessible in all countries. There is of course an ethical issue of pinging either fossthost or github continuously that could be used to track termux users, so users should explicitly opt in with a prompt. The user should also be able to disable this in termux app settings.

We could also download the files in pkg script instead so that app doesn't need to and it doesn't ping in background and only when users wants to update, but then users that may be running apt directly would still remain affected, unless we change relevant apt binaries to a wrapper scripts or patch them to run a script at start, depending on command install/update/upgrade/search.

What do you think? Or any better ideas.

Hopefully, fosshost and github will be accessible in all countries.

Try using a static page host like cloudflare pages, github pages, vercel, etc.

Maybe we can learn yum plugin: fastmirror .

Cloudflare is blocked in some countries, and our github pages was too.

Maybe we can learn yum plugin: fastmirror .

It does have exclude mirrors support but not meant for apt. But could learn from it. Faster mirrors can have more weight. But a fast mirror does not always mean that it has more monthly bandwidth limit.

We should instead have a single config file that's downloaded that contains all the data.

~ $ pkg install libarrow-cpp-static
Testing the available mirrors:
[*] (1) https://mirrors.ustc.edu.cn/termux/apt/termux-main: bad
Error: None of the mirrors are accessible
~ $

If mirror can't connect, do you want to test all mirrors/mirrors in a mirror group?

I don't think 2d082e7 should be done since user may not have any main source set and in that case we should not be automatically creating one.

I'll drop that.

Secondly, single mirror requires it to be done with termux-change-repo, but user may have some custom mirror that's not in the list of approved mirrors and would have to manually create symlink, which is not user friendly.

User can, with these changes, keep their custom mirror in $PREFIX/etc/termux/mirrors/<continent>/ as well and termux-change-repo should pick it up automatically. I think that this is the best way to handle custom mirrors, if user wants to override sources.list{,.d/*} directly then they are free to do so, but have to stick to using apt and not our wrappers

Ability to have a list of blacklisted mirrors is a very good idea, but lets do that in another PR, I'll fix this up and merge it so we can update main termux-tools package