这是indexloc提供的服务，不要输入任何密码
Skip to content

feat: Add K8s-bench presubmit workflow for task evaluation #326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat: Add K8s-bench presubmit workflow for task evaluation #326

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9e18180
feat: Add K8s-bench presubmit workflow for task evaluation
prasad89 Jun 9, 2025
dd078ae
fix: Correct job name and update result analysis step in presubmit wo…
prasad89 Jun 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions .github/workflows/presubmit-evals.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
name: K8s-bench Presubmit

on:
pull_request:
paths:
- "k8s-bench/tasks/**"

jobs:
detect-changed-tasks:
runs-on: ubuntu-latest
outputs:
task_dirs: ${{ steps.get_tasks.outputs.dirs }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Detect changed task directories
id: get_tasks
run: |
echo "Fetching base branch..."
git fetch origin ${{ github.base_ref }} --depth=1

echo "Finding changed task folders under k8s-bench/tasks..."
CHANGED_DIRS=$(git diff --name-only origin/${{ github.base_ref }} HEAD -- k8s-bench/tasks/ \
| awk -F/ 'NF>=3 {print $3}' \
| sort -u)

echo "Changed folders:"
echo "$CHANGED_DIRS"

# Output as comma-separated string
echo "dirs=$(echo "$CHANGED_DIRS" | paste -sd ',' -)" >> $GITHUB_OUTPUT

run-evals:
needs: detect-changed-tasks
if: needs.detect-changed-tasks.outputs.task_dirs != ''
runs-on: ubuntu-latest
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible here to run on multiple envs, like ubuntu and windows? Not sure that this is something we want now, I'm just curious if possible and what it would look like.

Copy link
Contributor Author

@prasad89 prasad89 Jun 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's possible to support multiple environments, such as Linux, Windows, and macOS.
However, my understanding is that the goal here is to run the modified task as a presubmit job.
This setup is also capable of running multiple jobs, depending on what has been modified.

@droot can confirm

timeout-minutes: 30
permissions:
contents: "read"
id-token: "write"

strategy:
matrix:
task: ${{ fromJson('["' + needs.detect-changed-tasks.outputs.task_dirs + '"]') }}

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Kind Cluster Setup
uses: ./.github/actions/kind-cluster-setup
with:
cluster_name: ${{ github.head_ref || github.ref_name }}

- name: Run evals for task ${{ matrix.task }}
env:
TEST_ARGS: >-
--llm-provider vertexai
--models gemini-2.5-pro-preview-05-06
--enable-tool-use-shim=false
--task-pattern=${{ matrix.task }}
run: |
echo "Running eval for ${{ matrix.task }}"
./dev/ci/periodics/run-evals.sh

- name: Analyze results
run: |
./dev/ci/periodics/analyze-evals.sh
cat ${{ github.workspace }}/.build/k8s-bench.md >> $GITHUB_STEP_SUMMARY