Make auto updates rate-limit aware #22994

TomJo2000 · 2025-01-21T21:49:02Z

This PR aims to make the auto-update workflow rate-limit aware in order to avoid future shadowbans of @termuxbot2 for just doing its job.

In this current state it only implements basic debugging functionality by printing out the .resources.core table of the rate limit query API response which contains the limit_(int), used_(int), remaining_(int) and reset_{(unix_epoch)} keys.

For local testing I have generated a fine grained access token for GitHub, which I am sourcing from gh.env inside the root directory of the git repo.
(This token only has read permissions to the rate-limit API endpoint, I may need to replace it for further testing)

# file: gh.env
export GITHUB_TOKEN='<your_github_token_here>'

The token is obviously passed in as part of the environment in the CI, so this file exists just for local testing.
Speaking of which, my local testing setup consists of running:

./scripts/run-docker.sh ./scripts/bin/update-packages '@all'

Which simulates a full auto update run.
(It also eats through the rate limit quite fast, which is why I'm opening this PR now, since I need to wait an hour to do more testing)

twaik · 2025-01-21T22:01:39Z

I am pretty much sure we should not spam logs with ratelimit status. The most of ratelimit consuming job is done in _fetch_and_cache_tags which invokes a lot of simultaneous requests to github (to save some CI time by avoiding requesting tags one by one). And not every single package invokes github rest api. I see at least two ways to optimize it.
a. Moving ratelimit checking to termux_github_api_get_tag.
b. Invoking ratelimit check every 50 packages.
termux_repology_api_get_latest_version creates repology metadata file on the first invocation and after this simply reuses it. We can do similar thing for ratelimits. We can cache the used field of github ratelimit reply and decrement it on every github request. This counter will be used to report null or current version of package when ~50 requests are remaining. AFAIK null should make tag checker report the current version as latest without throwing errors.

TomJo2000 · 2025-01-21T22:06:03Z

I know we don't want to spam logs, I'm not suggesting this is ready to merge in this state, this is an initial draft.

If you got any code suggestions do feel free to put them in here, or push them to my branch, or I can move the branch over to this repository if that is easier in terms of workflow.

twaik · 2025-01-21T22:09:06Z

I can try this a bit later, currently I should go to sleep.

TomJo2000 · 2025-01-21T22:10:19Z

Yeah I'm about to as well, I just wanted to push this before I do.

TomJo2000 · 2025-01-21T22:19:41Z

Alright I did a full run now that the limit reset for me and it looks like it takes ~820 API calls for the full auto-update cycle.

TomJo2000 · 2025-01-21T22:23:28Z

That limit is shared globally between all CI runs, and as such also resets globally for all CI runs.
We could probably make it idle or reschedule for after the next reset if there isn't even API limit remaining is the current time window.

(That limit being 5000 per hour)

scripts/bin/update-packages

TomJo2000 · 2025-01-30T16:59:13Z

I changed it now so it only outputs the rate limit information when the package has a GitHub source URL.
This is still a proof of concept at this stage, but it is closer to being functional.

scripts/bin/update-packages

TomJo2000 · 2025-02-01T13:26:47Z

This latest version is based on @twaik's suggestion to use the GraphQL API for querying github packages.

The GraphQL queries need to be sent as a JSON string and as such are very particular about containing no tabs or newlines.

It still needs some cleanup, but a batch size of 100 repos per request seems to work consistently.
I'd like to test separate named queries instead of separate batches when I have the time.
This also still lacks error handling and rate limit awareness.

The queries are built dynamically by the batching loop.

Inside this dropdown I have provided a prettified JSON object for a single batch and the extracted GraphQL query

// Begin test: Sat Feb  1 01:10:15 PM UTC 2025
// Batch: 0
{
"query": "
fragment _latest_release_tag on Repository {
  releases(first: 1, orderBy: {field: CREATED_AT, direction: DESC}) {
    nodes { tagName }
  }
}

fragment _newest_tag on Repository {
  refs( refPrefix: \"refs/tags/\" first: 1 orderBy: {field: TAG_COMMIT_DATE, direction: DESC}) {
    nodes { name }
  }
}

query {

_0: repository(owner: \"mevdschee\", name: \"2048.c\") { ..._latest_release_tag }
_1: repository(owner: \"radare\", name: \"acr\") { ..._latest_release_tag }
_2: repository(owner: \"Qucs\", name: \"ADMS\") { ..._latest_release_tag }
_3: repository(owner: \"mbrubeck\", name: \"agate\") { ..._latest_release_tag }
_4: repository(owner: \"FiloSottile\", name: \"age\") { ..._latest_release_tag }
_5: repository(owner: \"asciinema\", name: \"agg\") { ..._latest_release_tag }
_6: repository(owner: \"theZiz\", name: \"aha\") { ..._newest_tag }
_7: repository(owner: \"sigoden\", name: \"aichat\") { ..._latest_release_tag }
_8: repository(owner: \"alembic\", name: \"alembic\") { ..._latest_release_tag }
_9: repository(owner: \"xyproto\", name: \"algernon\") { ..._latest_release_tag }
_10: repository(owner: \"alsa-project\", name: \"alsa-lib\") { ..._newest_tag }
_11: repository(owner: \"alsa-project\", name: \"alsa-plugins\") { ..._newest_tag }
_12: repository(owner: \"alsa-project\", name: \"alsa-utils\") { ..._newest_tag }
_13: repository(owner: \"dalance\", name: \"amber\") { ..._latest_release_tag }
_14: repository(owner: \"makeworld-the-better-one\", name: \"amfora\") { ..._latest_release_tag }
_15: repository(owner: \"nmeum\", name: \"android-tools\") { ..._latest_release_tag }
_16: repository(owner: \"ysf\", name: \"anewer\") { ..._latest_release_tag }
_17: repository(owner: \"rcoh\", name: \"angle-grinder\") { ..._latest_release_tag }
_18: repository(owner: \"pystardust\", name: \"ani-cli\") { ..._latest_release_tag }
_19: repository(owner: \"getantibody\", name: \"antibody\") { ..._latest_release_tag }
_20: repository(owner: \"EFForg\", name: \"apkeep\") { ..._latest_release_tag }
_21: repository(owner: \"ximion\", name: \"appstream\") { ..._newest_tag }
_22: repository(owner: \"aptly-dev\", name: \"aptly\") { ..._latest_release_tag }
_23: repository(owner: \"argp-standalone\", name: \"argp-standalone\") { ..._newest_tag }
_24: repository(owner: \"aria2\", name: \"aria2\") { ..._newest_tag }
_25: repository(owner: \"opencollab\", name: \"arpack-ng\") { ..._newest_tag }
_26: repository(owner: \"asciidoc\", name: \"asciidoc-py3\") { ..._latest_release_tag }
_27: repository(owner: \"asciinema\", name: \"asciinema\") { ..._latest_release_tag }
_28: repository(owner: \"bergercookie\", name: \"asm-lsp\") { ..._newest_tag }
_29: repository(owner: \"assimp\", name: \"assimp\") { ..._latest_release_tag }
_30: repository(owner: \"OpenVisionE2\", name: \"astra-sm\") { ..._newest_tag }
_31: repository(owner: \"wez\", name: \"atomicparsley\") { ..._latest_release_tag }
_32: repository(owner: \"atomvm\", name: \"AtomVM\") { ..._latest_release_tag }
_33: repository(owner: \"ellie\", name: \"atuin\") { ..._latest_release_tag }
_34: repository(owner: \"wting\", name: \"autojump\") { ..._latest_release_tag }
_35: repository(owner: \"Ro5bert\", name: \"avra\") { ..._newest_tag }
_36: repository(owner: \"slavaGanzin\", name: \"await\") { ..._latest_release_tag }
_37: repository(owner: \"axel-download-accelerator\", name: \"axel\") { ..._latest_release_tag }
_38: repository(owner: \"BLAKE3-team\", name: \"BLAKE3\") { ..._latest_release_tag }
_39: repository(owner: \"fumiama\", name: \"base16384\") { ..._latest_release_tag }
_40: repository(owner: \"scop\", name: \"bash-completion\") { ..._latest_release_tag }
_41: repository(owner: \"sharkdp\", name: \"bat\") { ..._latest_release_tag }
_42: repository(owner: \"gavinhoward\", name: \"bc\") { ..._latest_release_tag }
_43: repository(owner: \"beanshell\", name: \"beanshell\") { ..._latest_release_tag }
_44: repository(owner: \"itchyny\", name: \"bed\") { ..._latest_release_tag }
_45: repository(owner: \"facebookincubator\", name: \"below\") { ..._newest_tag }
_46: repository(owner: \"rsharo\", name: \"bgrep\") { ..._latest_release_tag }
_47: repository(owner: \"louiz\", name: \"biboumi\") { ..._newest_tag }
_48: repository(owner: \"WebAssembly\", name: \"binaryen\") { ..._latest_release_tag }
_49: repository(owner: \"bitcoin\", name: \"bitcoin\") { ..._latest_release_tag }
_50: repository(owner: \"blade-lang\", name: \"blade\") { ..._latest_release_tag }
_51: repository(owner: \"jart\", name: \"blink\") { ..._latest_release_tag }
_52: repository(owner: \"blogc\", name: \"blogc\") { ..._latest_release_tag }
_53: repository(owner: \"tgraf\", name: \"bmon\") { ..._latest_release_tag }
_54: repository(owner: \"ekzhang\", name: \"bore\") { ..._latest_release_tag }
_55: repository(owner: \"borgbackup\", name: \"borg\") { ..._latest_release_tag }
_56: repository(owner: \"ascii-boxes\", name: \"boxes\") { ..._latest_release_tag }
_57: repository(owner: \"tmewett\", name: \"BrogueCE\") { ..._latest_release_tag }
_58: repository(owner: \"txthinking\", name: \"brook\") { ..._latest_release_tag }
_59: repository(owner: \"Canop\", name: \"broot\") { ..._latest_release_tag }
_60: repository(owner: \"google\", name: \"brotli\") { ..._latest_release_tag }
_61: repository(owner: \"bittorrent\", name: \"go-btfs\") { ..._latest_release_tag }
_62: repository(owner: \"bufbuild\", name: \"buf\") { ..._latest_release_tag }
_63: repository(owner: \"cabinpkg\", name: \"cabin\") { ..._latest_release_tag }
_64: repository(owner: \"caddyserver\", name: \"caddy\") { ..._latest_release_tag }
_65: repository(owner: \"lcn2\", name: \"calc\") { ..._latest_release_tag }
_66: repository(owner: \"c-ares\", name: \"c-ares\") { ..._latest_release_tag }
_67: repository(owner: \"karlstav\", name: \"cava\") { ..._latest_release_tag }
_68: repository(owner: \"kornelski\", name: \"cavif-rs\") { ..._latest_release_tag }
_69: repository(owner: \"ccache\", name: \"ccache\") { ..._latest_release_tag }
_70: repository(owner: \"0l1v3rr\", name: \"cli-file-manager\") { ..._latest_release_tag }
_71: repository(owner: \"CGAL\", name: \"cgal\") { ..._latest_release_tag }
_72: repository(owner: \"dloebl\", name: \"cgif\") { ..._latest_release_tag }
_73: repository(owner: \"hpjansson\", name: \"chafa\") { ..._latest_release_tag }
_74: repository(owner: \"libcheck\", name: \"check\") { ..._latest_release_tag }
_75: repository(owner: \"twpayne\", name: \"chezmoi\") { ..._latest_release_tag }
_76: repository(owner: \"theryangeary\", name: \"choose\") { ..._newest_tag }
_77: repository(owner: \"mitnk\", name: \"cicada\") { ..._latest_release_tag }
_78: repository(owner: \"CNugteren\", name: \"CLBlast\") { ..._latest_release_tag }
_79: repository(owner: \"leo-arch\", name: \"clifm\") { ..._latest_release_tag }
_80: repository(owner: \"Oblomov\", name: \"clinfo\") { ..._newest_tag }
_81: repository(owner: \"cloudflare\", name: \"cloudflared\") { ..._latest_release_tag }
_82: repository(owner: \"commonmark\", name: \"cmark\") { ..._latest_release_tag }
_83: repository(owner: \"abishekvashok\", name: \"cmatrix\") { ..._latest_release_tag }
_84: repository(owner: \"cmus\", name: \"cmus\") { ..._latest_release_tag }
_85: repository(owner: \"Arkq\", name: \"cmusfm\") { ..._newest_tag }
_86: repository(owner: \"exaexa\", name: \"codecrypt\") { ..._newest_tag }
_87: repository(owner: \"coin-or\", name: \"Cbc\") { ..._latest_release_tag }
_88: repository(owner: \"coin-or\", name: \"Clp\") { ..._latest_release_tag }
_89: repository(owner: \"cointop-sh\", name: \"cointop\") { ..._latest_release_tag }
_90: repository(owner: \"composer\", name: \"composer\") { ..._newest_tag }
_91: repository(owner: \"ros\", name: \"console_bridge\") { ..._latest_release_tag }
_92: repository(owner: \"cooklang\", name: \"cookcli\") { ..._latest_release_tag }
_93: repository(owner: \"DrakeW\", name: \"corgi\") { ..._latest_release_tag }
_94: repository(owner: \"cowsay-org\", name: \"cowsay\") { ..._latest_release_tag }
_95: repository(owner: \"danmar\", name: \"cppcheck\") { ..._latest_release_tag }
_96: repository(owner: \"Dr-Noob\", name: \"cpufetch\") { ..._latest_release_tag }
_97: repository(owner: \"opsengine\", name: \"cpulimit\") { ..._newest_tag }
_98: repository(owner: \"crawl\", name: \"crawl\") { ..._latest_release_tag }
_99: repository(owner: \"schollz\", name: \"croc\") { ..._latest_release_tag }
ratelimit: rateLimit { cost limit remaining used resetAt }
}"
}

Extracted query:

fragment _latest_release_tag on Repository {
  releases(first: 1, orderBy: {field: CREATED_AT, direction: DESC}) {
    nodes { tagName }
  }
}

fragment _newest_tag on Repository {
  refs( refPrefix: \"refs/tags/\" first: 1 orderBy: {field: TAG_COMMIT_DATE, direction: DESC}) {
    nodes { name }
  }
}

query {

_0: repository(owner: \"mevdschee\", name: \"2048.c\") { ..._latest_release_tag }
_1: repository(owner: \"radare\", name: \"acr\") { ..._latest_release_tag }
_2: repository(owner: \"Qucs\", name: \"ADMS\") { ..._latest_release_tag }
_3: repository(owner: \"mbrubeck\", name: \"agate\") { ..._latest_release_tag }
_4: repository(owner: \"FiloSottile\", name: \"age\") { ..._latest_release_tag }
_5: repository(owner: \"asciinema\", name: \"agg\") { ..._latest_release_tag }
_6: repository(owner: \"theZiz\", name: \"aha\") { ..._newest_tag }
_7: repository(owner: \"sigoden\", name: \"aichat\") { ..._latest_release_tag }
_8: repository(owner: \"alembic\", name: \"alembic\") { ..._latest_release_tag }
_9: repository(owner: \"xyproto\", name: \"algernon\") { ..._latest_release_tag }
_10: repository(owner: \"alsa-project\", name: \"alsa-lib\") { ..._newest_tag }
_11: repository(owner: \"alsa-project\", name: \"alsa-plugins\") { ..._newest_tag }
_12: repository(owner: \"alsa-project\", name: \"alsa-utils\") { ..._newest_tag }
_13: repository(owner: \"dalance\", name: \"amber\") { ..._latest_release_tag }
_14: repository(owner: \"makeworld-the-better-one\", name: \"amfora\") { ..._latest_release_tag }
_15: repository(owner: \"nmeum\", name: \"android-tools\") { ..._latest_release_tag }
_16: repository(owner: \"ysf\", name: \"anewer\") { ..._latest_release_tag }
_17: repository(owner: \"rcoh\", name: \"angle-grinder\") { ..._latest_release_tag }
_18: repository(owner: \"pystardust\", name: \"ani-cli\") { ..._latest_release_tag }
_19: repository(owner: \"getantibody\", name: \"antibody\") { ..._latest_release_tag }
_20: repository(owner: \"EFForg\", name: \"apkeep\") { ..._latest_release_tag }
_21: repository(owner: \"ximion\", name: \"appstream\") { ..._newest_tag }
_22: repository(owner: \"aptly-dev\", name: \"aptly\") { ..._latest_release_tag }
_23: repository(owner: \"argp-standalone\", name: \"argp-standalone\") { ..._newest_tag }
_24: repository(owner: \"aria2\", name: \"aria2\") { ..._newest_tag }
_25: repository(owner: \"opencollab\", name: \"arpack-ng\") { ..._newest_tag }
_26: repository(owner: \"asciidoc\", name: \"asciidoc-py3\") { ..._latest_release_tag }
_27: repository(owner: \"asciinema\", name: \"asciinema\") { ..._latest_release_tag }
_28: repository(owner: \"bergercookie\", name: \"asm-lsp\") { ..._newest_tag }
_29: repository(owner: \"assimp\", name: \"assimp\") { ..._latest_release_tag }
_30: repository(owner: \"OpenVisionE2\", name: \"astra-sm\") { ..._newest_tag }
_31: repository(owner: \"wez\", name: \"atomicparsley\") { ..._latest_release_tag }
_32: repository(owner: \"atomvm\", name: \"AtomVM\") { ..._latest_release_tag }
_33: repository(owner: \"ellie\", name: \"atuin\") { ..._latest_release_tag }
_34: repository(owner: \"wting\", name: \"autojump\") { ..._latest_release_tag }
_35: repository(owner: \"Ro5bert\", name: \"avra\") { ..._newest_tag }
_36: repository(owner: \"slavaGanzin\", name: \"await\") { ..._latest_release_tag }
_37: repository(owner: \"axel-download-accelerator\", name: \"axel\") { ..._latest_release_tag }
_38: repository(owner: \"BLAKE3-team\", name: \"BLAKE3\") { ..._latest_release_tag }
_39: repository(owner: \"fumiama\", name: \"base16384\") { ..._latest_release_tag }
_40: repository(owner: \"scop\", name: \"bash-completion\") { ..._latest_release_tag }
_41: repository(owner: \"sharkdp\", name: \"bat\") { ..._latest_release_tag }
_42: repository(owner: \"gavinhoward\", name: \"bc\") { ..._latest_release_tag }
_43: repository(owner: \"beanshell\", name: \"beanshell\") { ..._latest_release_tag }
_44: repository(owner: \"itchyny\", name: \"bed\") { ..._latest_release_tag }
_45: repository(owner: \"facebookincubator\", name: \"below\") { ..._newest_tag }
_46: repository(owner: \"rsharo\", name: \"bgrep\") { ..._latest_release_tag }
_47: repository(owner: \"louiz\", name: \"biboumi\") { ..._newest_tag }
_48: repository(owner: \"WebAssembly\", name: \"binaryen\") { ..._latest_release_tag }
_49: repository(owner: \"bitcoin\", name: \"bitcoin\") { ..._latest_release_tag }
_50: repository(owner: \"blade-lang\", name: \"blade\") { ..._latest_release_tag }
_51: repository(owner: \"jart\", name: \"blink\") { ..._latest_release_tag }
_52: repository(owner: \"blogc\", name: \"blogc\") { ..._latest_release_tag }
_53: repository(owner: \"tgraf\", name: \"bmon\") { ..._latest_release_tag }
_54: repository(owner: \"ekzhang\", name: \"bore\") { ..._latest_release_tag }
_55: repository(owner: \"borgbackup\", name: \"borg\") { ..._latest_release_tag }
_56: repository(owner: \"ascii-boxes\", name: \"boxes\") { ..._latest_release_tag }
_57: repository(owner: \"tmewett\", name: \"BrogueCE\") { ..._latest_release_tag }
_58: repository(owner: \"txthinking\", name: \"brook\") { ..._latest_release_tag }
_59: repository(owner: \"Canop\", name: \"broot\") { ..._latest_release_tag }
_60: repository(owner: \"google\", name: \"brotli\") { ..._latest_release_tag }
_61: repository(owner: \"bittorrent\", name: \"go-btfs\") { ..._latest_release_tag }
_62: repository(owner: \"bufbuild\", name: \"buf\") { ..._latest_release_tag }
_63: repository(owner: \"cabinpkg\", name: \"cabin\") { ..._latest_release_tag }
_64: repository(owner: \"caddyserver\", name: \"caddy\") { ..._latest_release_tag }
_65: repository(owner: \"lcn2\", name: \"calc\") { ..._latest_release_tag }
_66: repository(owner: \"c-ares\", name: \"c-ares\") { ..._latest_release_tag }
_67: repository(owner: \"karlstav\", name: \"cava\") { ..._latest_release_tag }
_68: repository(owner: \"kornelski\", name: \"cavif-rs\") { ..._latest_release_tag }
_69: repository(owner: \"ccache\", name: \"ccache\") { ..._latest_release_tag }
_70: repository(owner: \"0l1v3rr\", name: \"cli-file-manager\") { ..._latest_release_tag }
_71: repository(owner: \"CGAL\", name: \"cgal\") { ..._latest_release_tag }
_72: repository(owner: \"dloebl\", name: \"cgif\") { ..._latest_release_tag }
_73: repository(owner: \"hpjansson\", name: \"chafa\") { ..._latest_release_tag }
_74: repository(owner: \"libcheck\", name: \"check\") { ..._latest_release_tag }
_75: repository(owner: \"twpayne\", name: \"chezmoi\") { ..._latest_release_tag }
_76: repository(owner: \"theryangeary\", name: \"choose\") { ..._newest_tag }
_77: repository(owner: \"mitnk\", name: \"cicada\") { ..._latest_release_tag }
_78: repository(owner: \"CNugteren\", name: \"CLBlast\") { ..._latest_release_tag }
_79: repository(owner: \"leo-arch\", name: \"clifm\") { ..._latest_release_tag }
_80: repository(owner: \"Oblomov\", name: \"clinfo\") { ..._newest_tag }
_81: repository(owner: \"cloudflare\", name: \"cloudflared\") { ..._latest_release_tag }
_82: repository(owner: \"commonmark\", name: \"cmark\") { ..._latest_release_tag }
_83: repository(owner: \"abishekvashok\", name: \"cmatrix\") { ..._latest_release_tag }
_84: repository(owner: \"cmus\", name: \"cmus\") { ..._latest_release_tag }
_85: repository(owner: \"Arkq\", name: \"cmusfm\") { ..._newest_tag }
_86: repository(owner: \"exaexa\", name: \"codecrypt\") { ..._newest_tag }
_87: repository(owner: \"coin-or\", name: \"Cbc\") { ..._latest_release_tag }
_88: repository(owner: \"coin-or\", name: \"Clp\") { ..._latest_release_tag }
_89: repository(owner: \"cointop-sh\", name: \"cointop\") { ..._latest_release_tag }
_90: repository(owner: \"composer\", name: \"composer\") { ..._newest_tag }
_91: repository(owner: \"ros\", name: \"console_bridge\") { ..._latest_release_tag }
_92: repository(owner: \"cooklang\", name: \"cookcli\") { ..._latest_release_tag }
_93: repository(owner: \"DrakeW\", name: \"corgi\") { ..._latest_release_tag }
_94: repository(owner: \"cowsay-org\", name: \"cowsay\") { ..._latest_release_tag }
_95: repository(owner: \"danmar\", name: \"cppcheck\") { ..._latest_release_tag }
_96: repository(owner: \"Dr-Noob\", name: \"cpufetch\") { ..._latest_release_tag }
_97: repository(owner: \"opsengine\", name: \"cpulimit\") { ..._newest_tag }
_98: repository(owner: \"crawl\", name: \"crawl\") { ..._latest_release_tag }
_99: repository(owner: \"schollz\", name: \"croc\") { ..._latest_release_tag }
ratelimit: rateLimit { cost limit remaining used resetAt }
}

TomJo2000 · 2025-02-10T19:22:20Z

This is finally getting closer to being ready to merge.
All the code is here and working as separate pieces now.
It just needs to be plumbed and reorganized nicely.

It's not done, but I think it's in a good enough state that I'm gonna do ahead and mark this PR as ready for review.

scripts/bin/update-packages

twaik · 2025-02-18T16:48:29Z

scripts/bin/update-packages

+		fi
+
+		# This holds the source repositories, and query type for each repo we need to query.
+		# Newlines need to be escaped for the query parser


We can strip newlines and hard tabs in the final HTTP request composing with something like tr -d '\n\t' and drop this restriction in here.

As below, yes we can.
I still think it's important to have in mind.

twaik · 2025-02-18T16:49:02Z

scripts/bin/update-packages

+		# JSON strings cannot contain tabs or newlines
+		# so shutup shellcheck complaining about escapes in single quotes


We can strip newlines and hard tabs in the final HTTP request composing with something like tr -d '\n\t' and drop this restriction in here.

We absolutely can strip them with tr.
But I think it's an important thing to be aware of either way.

You are right. But I never said we should remove this comment completely. But probably we should use tr and place the comment right before the invocation. There is no comment about this here and IIRC you spent a lot of time investigating why query did not work and was throwing JSON exceptions.

twaik · 2025-02-18T16:52:31Z

scripts/bin/update-packages

+		# Start the GraphQL query with our two fragments for getting the latest tag from a release, and from refs/tags
+		# These are defined only if needed, so this one is for '_latest_release_tag'
+		grep -q '_latest_release_tag' <<< "${GITHUB_GRAPHQL_QUERIES[@]:$BATCH * $BATCH_SIZE:$BATCH_SIZE}" && {
+			QUERY+="$(printf '%s\n' \
+			'fragment _latest_release_tag on Repository {' \
+			'  latestRelease { tagName }' \
+			'}')"
+		}
+
+		grep -q '_newest_tag' <<< "${GITHUB_GRAPHQL_QUERIES[@]:$BATCH * $BATCH_SIZE:$BATCH_SIZE}" && {
+			QUERY+="$(printf '%s\n' \
+			'fragment _newest_tag on Repository {' \
+			'  refs( refPrefix: \"refs/tags/\" first: 1 orderBy: {field: TAG_COMMIT_DATE, direction: DESC}) {' \
+			'    nodes { name }' \
+			'  }' \
+			'}')"
+		}


We can use heredoc or multiline string syntax and drop newlines, hard tabs and unnecessary spaces during HTTP request composition (tr -d '\n\t' | sed 's/[ ][ ]*/ /g'?).

I'd prefer to keep this simple.
And introducing heredoc semantics, tr and sed into this decidedly does not do that.
It's unfortunately a bit messy either way.

Actually we already use this solution in auto-update code. Here.

twaik · 2025-02-18T16:55:44Z

scripts/bin/update-packages

+				'}' \
+
+		printf '%s' "${QUERY}" >> /tmp/query-12345
+		response="$(printf '{ "query": "%s" }' "${QUERY//$'\n'/ }" | curl -fsL \


You mentioned that github does not accept queries with more than 250 version fetch requests. Did you try to put more than one query to a single curl invocation? In this case we will have only one call to github API even if it will cost us 11 graphql points so it should not trigger any simultaneous request restriction checker.

250 was an overestimation on my part.
And yes I did try multiple queries in in single POST, that didn't work.
We might be able to do multiple separate POST requests, but that isn't really something I wanna be messing with right now.

100 per request works.
And 11 GraphQL API requests per auto-update cycle keeps us well under the rate limit.

As far as I understand the restrictions also include one request per second and requests being non-simultaneous. So probably we should add sleep 1 here. Or sleep 0.5 if you want to save 5 secs of CI time. I am not sure how much time the single curl invocation takes.

twaik · 2025-02-18T17:00:37Z

scripts/bin/update-packages


 	# First invocation of termux_repology_api_get_latest_version fetches and caches repology metadata.
-	quiet termux_repology_api_get_latest_version ' '
+	#quiet termux_repology_api_get_latest_version ' '


There is an alternative solution for invoking ninja and separating this to other script. All of non-github package versions are fetched with gitlab (~60 packages) or repology methods. We can drop fetching gitlab versions (leaving this to the regular termux_step_auto_update, it will add ~60-120 secs to CI run) and fetch only repology.
The cache file created by quiet termux_repology_api_get_latest_version ' ' contains only versions for packages which have updates so we can end up with

Declaring _LATEST_TAGS_REPOLOGY hashtable containing package names as keys and current package version in value.

Parsing repology cache data file replacing versions in our hashtable.

Pushing this array content to the regular _LATEST_TAGS.

Also this solution will let us not invoke python (in termux_repology_api_get_latest_version function) 1k+ times so resources required for this will not be wasted.

TomJo2000 · 2025-02-19T06:05:31Z

Alright I got an appointing I need to leave for in a couple minutes.
I just pushed my initial refactoring to this branch for review.
I'm not entirely happy with this yet and there's some debugging stuff I haven't cleaned up yet.

TomJo2000

@twaik I'd appreciate it if you could give this another look.
It passes my end-to-end testing but it's quite possible I've missed something.

I'd also appreciate it if you could group any suggestions you still have into a single review for easier organization.

TomJo2000 · 2025-02-23T08:02:26Z

scripts/bin/update-packages

 . "${TERMUX_SCRIPTDIR}/scripts/properties.sh"

+# my GH_TOKEN for debugging, this is already passed in for the CI.
+. "${TERMUX_SCRIPTDIR}/gh.env" || :


This is the last piece of debugging code left from testing.
Will be removed before final merge.

twaik

LGTM.

…ed GraphQL queries

thunder-coding reviewed Jan 22, 2025

View reviewed changes

scripts/bin/update-packages Show resolved Hide resolved

TomJo2000 force-pushed the ci-rate-limiting branch from f9cc350 to bfe5130 Compare January 30, 2025 16:57

twaik reviewed Jan 31, 2025

View reviewed changes

scripts/bin/update-packages Outdated Show resolved Hide resolved

TomJo2000 force-pushed the ci-rate-limiting branch from bfe5130 to f84e3dc Compare February 1, 2025 13:16

TomJo2000 force-pushed the ci-rate-limiting branch 3 times, most recently from eacdeed to 3be74ae Compare February 3, 2025 21:54

TomJo2000 mentioned this pull request Feb 6, 2025

bump(main/dart): 3.6.2 #23112

Merged

TomJo2000 force-pushed the ci-rate-limiting branch from 3be74ae to 1c5b346 Compare February 10, 2025 19:20

TomJo2000 marked this pull request as ready for review February 10, 2025 19:22

TomJo2000 requested a review from Grimler91 as a code owner February 10, 2025 19:22

TomJo2000 changed the title ~~[DRAFT]: Make auto updates rate-limit aware~~ [DO NOT MERGE YET]: Make auto updates rate-limit aware Feb 10, 2025

TomJo2000 force-pushed the ci-rate-limiting branch from 1c5b346 to 6536170 Compare February 11, 2025 11:16

twaik marked this pull request as draft February 11, 2025 12:19

TomJo2000 force-pushed the ci-rate-limiting branch 2 times, most recently from 9c26b4b to 25ef2aa Compare February 15, 2025 19:50

twaik reviewed Feb 18, 2025

View reviewed changes

scripts/bin/update-packages Show resolved Hide resolved

twaik reviewed Feb 18, 2025

View reviewed changes

scripts/bin/update-packages Show resolved Hide resolved

twaik reviewed Feb 18, 2025

View reviewed changes

scripts/bin/update-packages Show resolved Hide resolved

twaik reviewed Feb 18, 2025

View reviewed changes

scripts/bin/update-packages Show resolved Hide resolved

twaik reviewed Feb 18, 2025

View reviewed changes

scripts/bin/update-packages Show resolved Hide resolved

twaik reviewed Feb 18, 2025

View reviewed changes

scripts/bin/update-packages Show resolved Hide resolved

twaik reviewed Feb 18, 2025

View reviewed changes

scripts/bin/update-packages Show resolved Hide resolved

twaik reviewed Feb 18, 2025

View reviewed changes

TomJo2000 mentioned this pull request Feb 18, 2025

chore(scripts/bin/update-packages): cosmetic changes #23195

Closed

TomJo2000 force-pushed the ci-rate-limiting branch from 25ef2aa to ac00c90 Compare February 19, 2025 06:03

TomJo2000 force-pushed the ci-rate-limiting branch from ac00c90 to 114e5f4 Compare February 19, 2025 09:27

chore(script/update-packages): Script cleanup

545f675

TomJo2000 force-pushed the ci-rate-limiting branch 3 times, most recently from 448af35 to 1b007fa Compare February 23, 2025 08:07

TomJo2000 commented Feb 23, 2025

View reviewed changes

TomJo2000 marked this pull request as ready for review February 23, 2025 08:08

twaik approved these changes Feb 23, 2025

View reviewed changes

TomJo2000 changed the title ~~[DO NOT MERGE YET]: Make auto updates rate-limit aware~~ Make auto updates rate-limit aware Feb 23, 2025

TomJo2000 force-pushed the ci-rate-limiting branch from 1b007fa to d3dfce7 Compare February 23, 2025 08:41

TomJo2000 requested a review from thunder-coding February 23, 2025 08:48

enhance(script/update-packages): Fetch github releases/tags via batch…

9245d4e

…ed GraphQL queries

TomJo2000 force-pushed the ci-rate-limiting branch from d3dfce7 to 9245d4e Compare February 23, 2025 11:32

TomJo2000 merged commit c92f60e into termux:master Feb 23, 2025

TomJo2000 deleted the ci-rate-limiting branch February 23, 2025 12:55

This was referenced Feb 23, 2025

[Bug]: python-grpcio is not the latest version #23209

Closed

scripts(toolchain): set ASFLAGS=-c #23456

Open

Some packages cannot be built 3: Revenge of the Xiph #23492

Open

This comment was marked as spam.

Sign in to view

termux locked and limited conversation to collaborators Oct 24, 2025

		# JSON strings cannot contain tabs or newlines
		# so shutup shellcheck complaining about escapes in single quotes

Uh oh!

Make auto updates rate-limit aware #22994

Make auto updates rate-limit aware #22994

Uh oh!

Conversation

TomJo2000 commented Jan 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

twaik commented Jan 21, 2025

Uh oh!

TomJo2000 commented Jan 21, 2025

Uh oh!

twaik commented Jan 21, 2025

Uh oh!

TomJo2000 commented Jan 21, 2025

Uh oh!

TomJo2000 commented Jan 21, 2025

Uh oh!

TomJo2000 commented Jan 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

TomJo2000 commented Jan 30, 2025

Uh oh!

Uh oh!

TomJo2000 commented Feb 1, 2025

Uh oh!

TomJo2000 commented Feb 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

twaik Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomJo2000 commented Feb 19, 2025

Uh oh!

TomJo2000 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

twaik left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

Uh oh!

Reviewers

Assignees

Labels

Projects

TomJo2000 commented Jan 21, 2025 •

edited

Loading

TomJo2000 commented Jan 21, 2025 •

edited

Loading

twaik Feb 18, 2025 •

edited

Loading