这是indexloc提供的服务，不要输入任何密码
Skip to content

ci: Debugging for compilation issues #11062

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Nov 7, 2025
Merged

ci: Debugging for compilation issues #11062

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
71d05fc
ci: debugging for compilation issues
anthonyshew Nov 7, 2025
0455a29
WIP cb2d7
anthonyshew Nov 7, 2025
ae47b69
WIP 5906b
anthonyshew Nov 7, 2025
3cfbbd0
WIP 4a82f
anthonyshew Nov 7, 2025
5261d10
WIP 55153
anthonyshew Nov 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
240 changes: 239 additions & 1 deletion .github/workflows/turborepo-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,6 @@ jobs:
needs:
- find-changes
runs-on: ${{ matrix.os.runner }}
timeout-minutes: 45
if: ${{ needs.find-changes.outputs.rest == 'true' }}
strategy:
fail-fast: false
Expand Down Expand Up @@ -160,7 +159,24 @@ jobs:
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.6

- name: Enable Rust verbose logging
run: |
echo "RUST_BACKTRACE=1" >> $GITHUB_ENV
echo "RUST_LOG=info" >> $GITHUB_ENV
shell: bash

- name: Log Rust environment info
run: |
echo "=== Rust Toolchain Information ==="
rustc --version --verbose
cargo --version --verbose
rustup show
echo "=== sccache stats (before) ==="
sccache --show-stats || echo "sccache not available or failed"
shell: bash

- name: Integration Tests
timeout-minutes: 45
run: |
if [ -z "${RUSTC_WRAPPER}" ]; then
unset RUSTC_WRAPPER
Expand All @@ -175,6 +191,89 @@ jobs:
CARGO_INCREMENTAL: 0
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
# sccache timeout configuration to prevent hangs
SCCACHE_IDLE_TIMEOUT: 0
SCCACHE_REQUEST_TIMEOUT: 30
SCCACHE_ERROR_LOG: error

- name: Show sccache stats
if: always()
run: |
echo "=== sccache stats (after) ==="
sccache --show-stats || echo "sccache not available or failed"
shell: bash

- name: Capture state on timeout
if: cancelled()
run: |
echo "=========================================="
echo "INTEGRATION TEST STEP TIMED OUT"
echo "=========================================="
echo "Time: $(date)"
echo ""
echo "=== Active Rust/Cargo Processes (likely hung) ==="
ps aux | grep -E "[r]ustc|[c]argo" || echo "No rustc/cargo processes found"
echo ""
echo "=== Top CPU Consumers ==="
if [ "$RUNNER_OS" == "Linux" ]; then
ps aux --sort=-%cpu | head -15
elif [ "$RUNNER_OS" == "macOS" ]; then
ps aux -r | head -15
fi
echo ""
echo "(ICE files will be checked in next step)"
shell: bash
Comment on lines +206 to +225
Copy link
Contributor

@vercel vercel bot Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "Capture state on timeout" steps use if: cancelled() condition, but this won't trigger when a step times out—it only triggers when the entire workflow is cancelled by a user or external event. These steps should use if: always() or another appropriate condition to ensure they execute during step timeouts.

View Details
📝 Patch Details 
diff --git a/.github/workflows/turborepo-test.yml b/.github/workflows/turborepo-test.yml
index a075c3196..3908f4d23 100644
--- a/.github/workflows/turborepo-test.yml
+++ b/.github/workflows/turborepo-test.yml
@@ -201,7 +201,7 @@ jobs:
         shell: bash
 
       - name: Capture state on timeout
-        if: cancelled()
+        if: always()
         run: |
           echo "=========================================="
           echo "INTEGRATION TEST STEP TIMED OUT"
@@ -449,7 +449,7 @@ jobs:
         shell: bash
 
       - name: Capture state on timeout
-        if: cancelled()
+        if: always()
         run: |
           echo "=========================================="
           echo "RUST TEST STEP TIMED OUT"

Analysis

GitHub Actions "Capture state on timeout" steps use wrong condition

What fails: The "Capture state on timeout" steps in both the integration and rust_test jobs (lines 203 and 451) use if: cancelled() condition, which does not trigger when a step times out.

How to reproduce: Set up a GitHub Actions workflow with a step that has timeout-minutes: 45 and a debug step with if: cancelled() immediately after it. Let the first step exceed its timeout limit.

Result: The debug step with if: cancelled() does NOT execute when the preceding step times out. The workflow continues without capturing the debug state information.

Expected: According to GitHub Actions documentation on expressions, the cancelled() function "Returns true if the workflow was canceled." This refers to workflow-level cancellation (user cancellation), not step timeouts. Step-level timeouts ("The maximum number of minutes to run the step before killing the process") result in step failure, not workflow cancellation. Therefore, if: cancelled() will not trigger on step timeouts. The condition should be if: always() to ensure debug steps execute regardless of preceding step status.

Fix: Change if: cancelled() to if: always() in both "Capture state on timeout" steps to ensure they execute during step timeouts and capture the intended diagnostic information.


- name: Check for and display ICE files
if: always()
run: |
echo "=== Searching for Rust ICE (Internal Compiler Error) files ==="
ICE_FOUND=0

# Search in common locations
if [ "$RUNNER_OS" == "Windows" ]; then
# Windows paths
SEARCH_PATHS=(
"."
"$TEMP"
"$TMP"
"$USERPROFILE/.rustup/tmp"
)
else
# Unix paths
SEARCH_PATHS=(
"."
"/tmp"
"$HOME/.rustup/tmp"
"./target"
)
fi

for search_path in "${SEARCH_PATHS[@]}"; do
if [ -d "$search_path" ]; then
echo "Searching in: $search_path"
while IFS= read -r ice_file; do
if [ -f "$ice_file" ]; then
ICE_FOUND=1
echo ""
echo "========================================"
echo "ICE FILE FOUND: $ice_file"
echo "========================================"
cat "$ice_file"
echo ""
echo "========================================"
echo ""
fi
done < <(find "$search_path" -maxdepth 3 -name "rustc-ice-*.txt" -type f 2>/dev/null || true)
fi
done

if [ $ICE_FOUND -eq 0 ]; then
echo "No ICE files found."
else
echo "WARNING: Rust Internal Compiler Errors detected! See above for details."
fi
shell: bash

rust_lint:
name: Rust lints
Expand Down Expand Up @@ -290,6 +389,38 @@ jobs:
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.6

- name: Enable Rust verbose logging
run: |
echo "RUST_BACKTRACE=1" >> $GITHUB_ENV
echo "RUST_LOG=info" >> $GITHUB_ENV
shell: bash

- name: Log Rust environment info
run: |
echo "=== Rust Toolchain Information ==="
rustc --version --verbose
cargo --version --verbose
rustup show
echo "=== System Resources ==="
if [ "$RUNNER_OS" == "Linux" ]; then
echo "CPUs: $(nproc)"
free -h
df -h
echo "File descriptors: $(ulimit -n)"
elif [ "$RUNNER_OS" == "macOS" ]; then
echo "CPUs: $(sysctl -n hw.ncpu)"
vm_stat
df -h
echo "File descriptors: $(ulimit -n)"
elif [ "$RUNNER_OS" == "Windows" ]; then
echo "CPUs: $NUMBER_OF_PROCESSORS"
powershell "Get-CimInstance Win32_OperatingSystem | Select-Object FreePhysicalMemory,TotalVisibleMemorySize | Format-List"
powershell "Get-CimInstance Win32_LogicalDisk | Select-Object DeviceID,Size,FreeSpace | Format-Table -AutoSize"
fi
echo "=== sccache stats (before) ==="
sccache --show-stats || echo "sccache not available or failed"
shell: bash

- name: Run tests
timeout-minutes: 120
# We explicitly unset RUSTC_WRAPPER if it is an empty string as causes build issues
Expand All @@ -311,6 +442,113 @@ jobs:
CARGO_INCREMENTAL: 0
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
# sccache timeout configuration to prevent hangs
SCCACHE_IDLE_TIMEOUT: 0
SCCACHE_REQUEST_TIMEOUT: 30
SCCACHE_ERROR_LOG: error

- name: Show sccache stats
if: always()
run: |
echo "=== sccache stats (after) ==="
sccache --show-stats || echo "sccache not available or failed"
shell: bash

- name: Capture state on timeout
if: cancelled()
run: |
echo "=========================================="
echo "RUST TEST STEP TIMED OUT"
echo "=========================================="
echo "Time: $(date)"
echo ""
echo "=== Active Rust/Cargo Processes (likely hung) ==="
ps aux | grep -E "[r]ustc|[c]argo" || echo "No rustc/cargo processes found"
echo ""
echo "=== Top CPU Consumers ==="
if [ "$RUNNER_OS" == "Linux" ]; then
ps aux --sort=-%cpu | head -15
elif [ "$RUNNER_OS" == "macOS" ]; then
ps aux -r | head -15
fi
echo ""
echo "(ICE files will be checked in next step)"
shell: bash

- name: Check for and display ICE files
if: always()
run: |
echo "=== Searching for Rust ICE (Internal Compiler Error) files ==="
ICE_FOUND=0

# Search in common locations
if [ "$RUNNER_OS" == "Windows" ]; then
# Windows paths
SEARCH_PATHS=(
"."
"$TEMP"
"$TMP"
"$USERPROFILE/.rustup/tmp"
)
else
# Unix paths
SEARCH_PATHS=(
"."
"/tmp"
"$HOME/.rustup/tmp"
"./target"
)
fi

for search_path in "${SEARCH_PATHS[@]}"; do
if [ -d "$search_path" ]; then
echo "Searching in: $search_path"
while IFS= read -r ice_file; do
if [ -f "$ice_file" ]; then
ICE_FOUND=1
echo ""
echo "========================================"
echo "ICE FILE FOUND: $ice_file"
echo "========================================"
cat "$ice_file"
echo ""
echo "========================================"
echo ""
fi
done < <(find "$search_path" -maxdepth 3 -name "rustc-ice-*.txt" -type f 2>/dev/null || true)
fi
done

if [ $ICE_FOUND -eq 0 ]; then
echo "No ICE files found."
else
echo "WARNING: Rust Internal Compiler Errors detected! See above for details."
fi
shell: bash

- name: Display final system state
if: always()
run: |
echo "=== Final System State ==="
if [ "$RUNNER_OS" == "Linux" ]; then
echo "Memory:"
free -h
echo "Disk:"
df -h
echo "Top processes by CPU:"
ps aux --sort=-%cpu | head -10 || true
elif [ "$RUNNER_OS" == "macOS" ]; then
echo "Memory:"
vm_stat
echo "Disk:"
df -h
echo "Top processes by CPU:"
ps aux -r | head -10 || true
elif [ "$RUNNER_OS" == "Windows" ]; then
powershell "Get-CimInstance Win32_OperatingSystem | Select-Object FreePhysicalMemory,TotalVisibleMemorySize | Format-List"
powershell "Get-CimInstance Win32_LogicalDisk | Select-Object DeviceID,Size,FreeSpace | Format-Table -AutoSize"
fi
shell: bash

basic-example:
name: "`basic` example"
Expand Down
Loading