ci: breakdown benchmarks #7641
ci: breakdown benchmarks #7641
Conversation
|
WalkthroughUpdates benchmarking CI: narrows the existing Benchmarks Configuration workflow to only biome_configuration and biome_rowan, and adds six new workflows — benchmark_css, benchmark_graphql, benchmark_js, benchmark_json, benchmark_manifests, and benchmark_module_graph. Each workflow runs CodSpeed benchmarks on ARM Linux runners using actions/checkout, a Rust setup action with cargo-codspeed, per-package cargo codspeed build (limited CARGO_BUILD_JOBS) and codspeed run. Triggers include workflow_dispatch, pull_request (opened, synchronize) and push with path filters. Checkout PR branch step now prefers the PR head SHA with a github.sha fallback. No exported/public API changes. Suggested labels
Suggested reviewers
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches🧪 Generate unit tests
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (6)
.github/workflows/benchmark_configuration.yml (1)
39-43: Checkout step references a missing outputSame missing
steps.shaoutput here; please apply the fix noted in the JSON workflow review..github/workflows/benchmark_module_graph.yml (1)
39-43: Checkout step references a missing outputDitto: no
shastep exists, so the checkout fails. Reuse the${{ github.event.pull_request.head.sha || github.sha }}fix..github/workflows/benchmark_graphql.yml (1)
46-50: Checkout step references a missing outputRepeated issue:
steps.shais undefined. Please align with the fix suggested earlier..github/workflows/benchmark_manifests.yml (1)
39-43: Checkout step references a missing outputOnce more,
steps.shaisn’t defined. Please patch it as recommended in the JSON workflow file..github/workflows/benchmark_js.yml (1)
49-53: Checkout step references a missing outputStill pointing at the nonexistent
steps.sha. Apply the same${{ github.event.pull_request.head.sha || github.sha }}tweak..github/workflows/benchmark_css.yml (1)
49-53: Checkout step references a missing outputIdentical issue here—
steps.shaisn’t defined. Please update the ref as in the other workflows.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
.github/workflows/benchmark_configuration.yml(1 hunks).github/workflows/benchmark_css.yml(1 hunks).github/workflows/benchmark_graphql.yml(1 hunks).github/workflows/benchmark_js.yml(1 hunks).github/workflows/benchmark_json.yml(1 hunks).github/workflows/benchmark_manifests.yml(1 hunks).github/workflows/benchmark_module_graph.yml(1 hunks)
🧰 Additional context used
🪛 actionlint (1.7.7)
.github/workflows/benchmark_manifests.yml
31-31: label "depot-ubuntu-24.04-arm-16" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2022", "windows-2019", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "ubuntu-20.04", "macos-latest", "macos-latest-xl", "macos-latest-xlarge", "macos-latest-large", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xl", "macos-14-xlarge", "macos-14-large", "macos-14", "macos-13-xl", "macos-13-xlarge", "macos-13-large", "macos-13", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
42-42: property "sha" is not defined in object type {}
(expression)
.github/workflows/benchmark_module_graph.yml
31-31: label "depot-ubuntu-24.04-arm-16" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2022", "windows-2019", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "ubuntu-20.04", "macos-latest", "macos-latest-xl", "macos-latest-xlarge", "macos-latest-large", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xl", "macos-14-xlarge", "macos-14-large", "macos-14", "macos-13-xl", "macos-13-xlarge", "macos-13-large", "macos-13", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
42-42: property "sha" is not defined in object type {}
(expression)
.github/workflows/benchmark_js.yml
39-39: label "depot-ubuntu-24.04-arm-16" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2022", "windows-2019", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "ubuntu-20.04", "macos-latest", "macos-latest-xl", "macos-latest-xlarge", "macos-latest-large", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xl", "macos-14-xlarge", "macos-14-large", "macos-14", "macos-13-xl", "macos-13-xlarge", "macos-13-large", "macos-13", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
52-52: property "sha" is not defined in object type {}
(expression)
.github/workflows/benchmark_graphql.yml
37-37: label "depot-ubuntu-24.04-arm-16" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2022", "windows-2019", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "ubuntu-20.04", "macos-latest", "macos-latest-xl", "macos-latest-xlarge", "macos-latest-large", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xl", "macos-14-xlarge", "macos-14-large", "macos-14", "macos-13-xl", "macos-13-xlarge", "macos-13-large", "macos-13", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
49-49: property "sha" is not defined in object type {}
(expression)
.github/workflows/benchmark_css.yml
39-39: label "depot-ubuntu-24.04-arm-16" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2022", "windows-2019", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "ubuntu-20.04", "macos-latest", "macos-latest-xl", "macos-latest-xlarge", "macos-latest-large", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xl", "macos-14-xlarge", "macos-14-large", "macos-14", "macos-13-xl", "macos-13-xlarge", "macos-13-large", "macos-13", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
52-52: property "sha" is not defined in object type {}
(expression)
.github/workflows/benchmark_json.yml
39-39: label "depot-ubuntu-24.04-arm-16" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2022", "windows-2019", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "ubuntu-20.04", "macos-latest", "macos-latest-xl", "macos-latest-xlarge", "macos-latest-large", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xl", "macos-14-xlarge", "macos-14-large", "macos-14", "macos-13-xl", "macos-13-xlarge", "macos-13-large", "macos-13", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
52-52: property "sha" is not defined in object type {}
(expression)
There was a problem hiding this comment.
We could use the workflow_call trigger to avoid copy-pasting the steps: https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows
|
Definitely! I will add it to my todo list once we confirm that the new workflows run :) |
There was a problem hiding this comment.
Actionable comments posted: 6
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
.github/workflows/benchmark_configuration.yml(2 hunks).github/workflows/benchmark_css.yml(1 hunks).github/workflows/benchmark_graphql.yml(1 hunks).github/workflows/benchmark_js.yml(1 hunks).github/workflows/benchmark_json.yml(1 hunks).github/workflows/benchmark_manifests.yml(1 hunks).github/workflows/benchmark_module_graph.yml(1 hunks)
🧰 Additional context used
🪛 actionlint (1.7.7)
.github/workflows/benchmark_js.yml
39-39: label "depot-ubuntu-24.04-arm-16" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2022", "windows-2019", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "ubuntu-20.04", "macos-latest", "macos-latest-xl", "macos-latest-xlarge", "macos-latest-large", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xl", "macos-14-xlarge", "macos-14-large", "macos-14", "macos-13-xl", "macos-13-xlarge", "macos-13-large", "macos-13", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
.github/workflows/benchmark_json.yml
39-39: label "depot-ubuntu-24.04-arm-16" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2022", "windows-2019", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "ubuntu-20.04", "macos-latest", "macos-latest-xl", "macos-latest-xlarge", "macos-latest-large", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xl", "macos-14-xlarge", "macos-14-large", "macos-14", "macos-13-xl", "macos-13-xlarge", "macos-13-large", "macos-13", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
.github/workflows/benchmark_css.yml
39-39: label "depot-ubuntu-24.04-arm-16" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2022", "windows-2019", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "ubuntu-20.04", "macos-latest", "macos-latest-xl", "macos-latest-xlarge", "macos-latest-large", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xl", "macos-14-xlarge", "macos-14-large", "macos-14", "macos-13-xl", "macos-13-xlarge", "macos-13-large", "macos-13", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
.github/workflows/benchmark_graphql.yml
37-37: label "depot-ubuntu-24.04-arm-16" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2022", "windows-2019", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "ubuntu-20.04", "macos-latest", "macos-latest-xl", "macos-latest-xlarge", "macos-latest-large", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xl", "macos-14-xlarge", "macos-14-large", "macos-14", "macos-13-xl", "macos-13-xlarge", "macos-13-large", "macos-13", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
.github/workflows/benchmark_manifests.yml
31-31: label "depot-ubuntu-24.04-arm-16" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2022", "windows-2019", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "ubuntu-20.04", "macos-latest", "macos-latest-xl", "macos-latest-xlarge", "macos-latest-large", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xl", "macos-14-xlarge", "macos-14-large", "macos-14", "macos-13-xl", "macos-13-xlarge", "macos-13-large", "macos-13", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
.github/workflows/benchmark_module_graph.yml
31-31: label "depot-ubuntu-24.04-arm-16" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2022", "windows-2019", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "ubuntu-20.04", "macos-latest", "macos-latest-xl", "macos-latest-xlarge", "macos-latest-large", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xl", "macos-14-xlarge", "macos-14-large", "macos-14", "macos-13-xl", "macos-13-xlarge", "macos-13-large", "macos-13", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
(runner-label)
| permissions: | ||
| pull-requests: write |
There was a problem hiding this comment.
Don’t forget contents: read.
The checkout step needs that scope; otherwise every push run falls over instantly.
Same fix as elsewhere:
permissions:
- pull-requests: write
+ pull-requests: write
+ contents: read📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| permissions: | |
| pull-requests: write | |
| permissions: | |
| pull-requests: write | |
| contents: read |
🤖 Prompt for AI Agents
In .github/workflows/benchmark_css.yml around lines 36 to 37, the workflow
grants pull-requests: write but is missing the required contents: read
permission for the checkout action; update the permissions block to include
contents: read alongside the existing pull-requests permission so the checkout
step can access the repository contents and workflow runs won't fail.
| permissions: | ||
| pull-requests: write |
There was a problem hiding this comment.
Restore contents read scope.
Checkout needs contents: read; without it, pushes to main/next will keep failing right at the first step.
Suggested tweak:
permissions:
- pull-requests: write
+ pull-requests: write
+ contents: read📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| permissions: | |
| pull-requests: write | |
| permissions: | |
| pull-requests: write | |
| contents: read |
🤖 Prompt for AI Agents
In .github/workflows/benchmark_graphql.yml around lines 34 to 35, the workflow
permissions only grant pull-requests: write but lack the required contents: read
permission needed by actions/checkout; update the permissions block to include
contents: read (in addition to pull-requests: write) so the checkout step can
access the repository contents and subsequent pushes to main/next won't fail.
| permissions: | ||
| pull-requests: write |
There was a problem hiding this comment.
Needs contents scope for checkout.
Right now the token can’t read the repo on push events, so the workflow will fail before building.
Patch proposal:
permissions:
- pull-requests: write
+ pull-requests: write
+ contents: read📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| permissions: | |
| pull-requests: write | |
| permissions: | |
| pull-requests: write | |
| contents: read |
🤖 Prompt for AI Agents
.github/workflows/benchmark_js.yml around lines 36 to 37: the workflow only
grants pull-requests: write, but checkout on push requires repository contents
permission, so add a contents permission (at least read) under permissions;
update the permissions block to include contents: read (or contents: write if
the job needs push access) alongside the existing pull-requests: write so the
GITHUB_TOKEN can access the repo during checkout.
| permissions: | ||
| pull-requests: write |
There was a problem hiding this comment.
Add contents read permission.
With only pull-requests: write, the token cannot fetch the repo on push events, so checkout will bomb out. Please include contents: read alongside it.
Apply this diff:
permissions:
- pull-requests: write
+ pull-requests: write
+ contents: read📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| permissions: | |
| pull-requests: write | |
| permissions: | |
| pull-requests: write | |
| contents: read |
🤖 Prompt for AI Agents
.github/workflows/benchmark_json.yml around lines 36 to 37: the workflow only
grants pull-requests: write which prevents the checkout from fetching repo
contents on push events; update the permissions block to include contents: read
alongside pull-requests: write (i.e., add a contents: read entry under
permissions) so the token can read repository contents during checkout.
| permissions: | ||
| pull-requests: write |
There was a problem hiding this comment.
Missing contents permission breaks checkout.
The job token needs contents: read; otherwise the initial clone dies on any push run.
Please patch:
permissions:
- pull-requests: write
+ pull-requests: write
+ contents: read📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| permissions: | |
| pull-requests: write | |
| permissions: | |
| pull-requests: write | |
| contents: read |
🤖 Prompt for AI Agents
.github/workflows/benchmark_manifests.yml around lines 28 to 29: the workflow
only grants pull-requests: write which does not allow the job token to access
repository files and causes checkout to fail; update the permissions block to
include contents: read (e.g., add a contents: read entry alongside
pull-requests: write) so the checkout step can access repository contents.
| permissions: | ||
| pull-requests: write |
There was a problem hiding this comment.
Grant contents: read here too.
Without contents: read, the checkout step will choke on push triggers.
Recommended change:
permissions:
- pull-requests: write
+ pull-requests: write
+ contents: read📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| permissions: | |
| pull-requests: write | |
| permissions: | |
| pull-requests: write | |
| contents: read |
🤖 Prompt for AI Agents
In .github/workflows/benchmark_module_graph.yml around lines 28-29, the workflow
permissions only grant pull-requests: write but missing contents: read which
causes the checkout step to fail on push triggers; update the permissions block
to include contents: read (alongside pull-requests: write) so the workflow token
can read repository contents for the checkout step.
|
Applied suggestions here acbd9cc |
Co-authored-by: siketyan <12772118+siketyan@users.noreply.github.com>
Summary
Now that codspeed supports partial runs, we can split the benchmarks by feature/language.
This PR split the benchmarks in languages for formatter, parser and analyzer. I splits the benchmarks by feature for the rest: module graph, package and configuration.
The code has been generated by Claude Code
Test Plan
CI should pass
Docs
N/A