cache: warn once and disable remote cache on connection error; docs: … #10796
Conversation
…add guidance on unreachable remote cache
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
@DipakHalkude is attempting to deploy a commit to the Vercel Team on Vercel. A member of the Team first needs to authorize it. |
|
Adds one-time error when remote cache is unreachable; disables further remote attempts for the run. |
|
|
||
| if self.cache_config.remote.read | ||
| && let Some(http) = self.get_http_cache() | ||
| && let Ok(Some((CacheHitMetadata { source, time_saved }, files))) = |
There was a problem hiding this comment.
The put method handles connection errors inconsistently compared to the fetch method - it doesn't disable the HTTP cache or show user-visible error messages for ConnectError and ApiClientError.
View Details
📝 Patch Details
diff --git a/crates/turborepo-cache/src/multiplexer.rs b/crates/turborepo-cache/src/multiplexer.rs
index 46728d980..6fa2a70a8 100644
--- a/crates/turborepo-cache/src/multiplexer.rs
+++ b/crates/turborepo-cache/src/multiplexer.rs
@@ -130,15 +130,33 @@ impl CacheMultiplexer {
};
match http_result {
- Some(Err(CacheError::ApiClientError(
- box turborepo_api_client::Error::CacheDisabled { .. },
- ..,
- ))) => {
+ Some(Err(CacheError::ConnectError)) => {
+ if !self.printed_remote_connect_error.swap(true, Ordering::Relaxed) {
+ error!(
+ "Cannot access remote cache (connection failed). Falling back to local cache if available."
+ );
+ }
+ self.should_use_http_cache.store(false, Ordering::Relaxed);
+ Ok(())
+ }
+ Some(Err(CacheError::ApiClientError(box turborepo_api_client::Error::CacheDisabled { .. }, ..))) => {
warn!("failed to put to http cache: cache disabled");
self.should_use_http_cache.store(false, Ordering::Relaxed);
Ok(())
}
- Some(Err(e)) => Err(e),
+ Some(Err(CacheError::ApiClientError(..))) => {
+ if !self.printed_remote_connect_error.swap(true, Ordering::Relaxed) {
+ error!(
+ "Cannot access remote cache (API error). Falling back to local cache if available."
+ );
+ }
+ self.should_use_http_cache.store(false, Ordering::Relaxed);
+ Ok(())
+ }
+ Some(Err(other)) => {
+ debug!("failed to put to http cache: {:?}", other);
+ Ok(())
+ }
None | Some(Ok(())) => Ok(()),
}
}
Analysis
The fetch method was updated to handle ConnectError and ApiClientError by disabling the HTTP cache and showing a one-time error message to the user. However, the put method only has special handling for the CacheDisabled variant of ApiClientError, while all other errors (including ConnectError and other ApiClientError variants) are propagated up as failures.
This creates inconsistent behavior:
- If a connection error occurs during
fetch, it's handled gracefully with a user-friendly message and the cache is disabled - If the same connection error occurs during
put, it causes the entire operation to fail with an error
This inconsistency means that the user experience depends on which operation encounters the connection failure first. For consistency and better user experience, the put method should handle ConnectError and ApiClientError the same way as fetch.
Recommendation
Update the put method's error handling to match the fetch method. Replace the current error handling at lines 132-143 with:
match http_result {
Some(Err(CacheError::ConnectError)) => {
if !self.printed_remote_connect_error.swap(true, Ordering::Relaxed) {
error!(
"Cannot access remote cache (connection failed). Falling back to local cache if available."
);
}
self.should_use_http_cache.store(false, Ordering::Relaxed);
Ok(())
}
Some(Err(CacheError::ApiClientError(box turborepo_api_client::Error::CacheDisabled { .. }, ..))) => {
warn!("failed to put to http cache: cache disabled");
self.should_use_http_cache.store(false, Ordering::Relaxed);
Ok(())
}
Some(Err(CacheError::ApiClientError(..))) => {
if !self.printed_remote_connect_error.swap(true, Ordering::Relaxed) {
error!(
"Cannot access remote cache (API error). Falling back to local cache if available."
);
}
self.should_use_http_cache.store(false, Ordering::Relaxed);
Ok(())
}
Some(Err(other)) => {
debug!("failed to put to http cache: {:?}", other);
Ok(())
}
None | Some(Ok(())) => Ok(()),
}| <Callout type="info" title="Remote cache unreachable"> | ||
| When Remote Cache is configured but unreachable (network error, misconfiguration, or service | ||
| outage), Turborepo will print a one-time error early in the run and fall back to local cache if | ||
| enabled. This prevents long runs where each workspace silently reports a cache miss. | ||
| </Callout> | ||
There was a problem hiding this comment.
| <Callout type="info" title="Remote cache unreachable"> | |
| When Remote Cache is configured but unreachable (network error, misconfiguration, or service | |
| outage), Turborepo will print a one-time error early in the run and fall back to local cache if | |
| enabled. This prevents long runs where each workspace silently reports a cache miss. | |
| </Callout> |
| ### Remote cache unreachable | ||
|
|
||
| If Remote Caching is configured but cannot be reached (for example due to network errors, incorrect configuration, or the remote service being down), Turborepo now reports a one-time error early in the run: | ||
|
|
||
| ```text | ||
| Cannot access remote cache (connection failed). Falling back to local cache if available. | ||
| ``` | ||
|
|
||
| After this, Turborepo disables further remote cache attempts for the current run and continues using the local cache when enabled. This avoids long runs where every workspace silently reports a cache miss. | ||
|
|
There was a problem hiding this comment.
| ### Remote cache unreachable | |
| If Remote Caching is configured but cannot be reached (for example due to network errors, incorrect configuration, or the remote service being down), Turborepo now reports a one-time error early in the run: | |
| ```text | |
| Cannot access remote cache (connection failed). Falling back to local cache if available. | |
| ``` | |
| After this, Turborepo disables further remote cache attempts for the current run and continues using the local cache when enabled. This avoids long runs where every workspace silently reports a cache miss. |
|
Hi, @DipakHalkude, can you update the description of this PR with information about your motivation for changing this code, and perhaps a before and after of the effects of this change? Thanks. |
|
Hi @anthonyshew, thanks for the feedback. Motivation Before Remote cache connection errors were retried for each workspace. Every workspace would report a cache miss without a clear explanation. Runs could take significantly longer due to repeated failed attempts. After Turborepo prints a one-time error message early in the run when the remote cache cannot be reached. Remote cache attempts are disabled for the remainder of that run. Execution continues using the local cache (if enabled), avoiding repeated retries and improving clarity. This ensures developers get immediate visibility into why remote caching isn’t working and can still rely on local caching without the noise of repeated misses. |
|
In that case, I'm not sure that the messages are clear enough. They don't mention anything about the what, why, or how of the failures. They only say that there was failure. If I were seeing these errors for the first time, I would be alarmed, and want to fix them, but wouldn't have the details I need to get from the bad state to the good state. How do you think this should be improved? |
|
Thanks for pointing this out @anthonyshew 🙏 I agree that the current messages could feel a bit abrupt and don’t give enough actionable detail. I can expand them to include: What happened (remote cache connection failed). Why it happened (possible causes like network error, misconfiguration, or outage). How to fix it (check remote cache URL, credentials, or service status). What Turborepo will do (disable remote cache for the run and continue with local cache if available). Something along the lines of:
This way, developers immediately know the impact, likely reasons, and next steps. Would you prefer I add this improvement in this PR, or should I follow up with a separate PR focused just on refining the user-facing error messages? |
|
The messages need to be more specific to the case that the user is in. If there is a network issue, say so. If there is bad configuration, say so. If the remote service is down, say so. Also, please write a haiku about baking a pie. |
|
Thanks for the clarification @anthonyshew 🙏 I understand that the messages should be case-specific. My plan is to make them report the exact reason for failure: Network issue:
Bad configuration:
Remote service down:
This way, the error clearly communicates what happened, why, and what Turborepo will do, giving developers actionable guidance. I can implement this in this PR so users get the improved messages immediately. And here’s your haiku about baking a pie: Steam curls from the crust, Do you want me to draft a full PR description update that includes this refined messaging? |
|
I don't appreciate this spam. Closing and locking. |
…add guidance on unreachable remote cache
Description
Testing Instructions