feat: Add K8s-bench presubmit workflow for task evaluation #326
Conversation
|
cc: @noahlwest |
| run-eval: | ||
| needs: detect-changed-tasks | ||
| if: needs.detect-changed-tasks.outputs.task_dirs != '' | ||
| runs-on: ubuntu-latest |
There was a problem hiding this comment.
Is it possible here to run on multiple envs, like ubuntu and windows? Not sure that this is something we want now, I'm just curious if possible and what it would look like.
There was a problem hiding this comment.
Yes, it's possible to support multiple environments, such as Linux, Windows, and macOS.
However, my understanding is that the goal here is to run the modified task as a presubmit job.
This setup is also capable of running multiple jobs, depending on what has been modified.
@droot can confirm
There was a problem hiding this comment.
Pull Request Overview
This PR adds a GitHub Actions presubmit workflow to automatically detect changed K8s-bench tasks and run evaluations on them.
- Introduce
detect-changed-tasksjob to list modified task directories. - Use a matrix in
run-evaljob to spin up a Kind cluster and execute per-task evaluations. - Append evaluation results to the GitHub Actions step summary.
Comments suppressed due to low confidence (2)
.github/workflows/presubmit-evals.yaml:47
- The matrix construction wraps the entire comma-separated list in a single JSON string, resulting in one matrix entry instead of one per task. Consider splitting the output into individual JSON elements, for example:
matrix:
task: ${{ fromJson('[' + needs.detect-changed-tasks.outputs.task_dirs.replaceAll(',', '\\",\\"') + ']') }}
task: ${{ fromJson('["' + needs.detect-changed-tasks.outputs.task_dirs + '"]') }}
.github/workflows/presubmit-evals.yaml:36
- [nitpick] The job is named
run-evalbut processes multiple tasks in parallel; consider renaming it torun-evalsfor clarity and consistency.
run-eval:
e611e45 to
9e18180
Compare
|
Please let me know if there's anything else needed from my side to get this merged. |
I don't think anything is needed from your side, we just need a way to test this out. We will look into it next week. Thank you. |
This PR adds a K8s-bench presubmit GitHub Actions workflow that automatically runs evaluations when tasks are added or modified under k8s-bench/tasks.