Incident Report

Summary

On Tuesday, 28 January, 2025 at 07:26 US/Pacific, the Admin Console and Cloud Support Platform experienced elevated latency and ‘500’ errors for a duration of 2 hours and 8 minutes. We apologize to our customers affected by the recent service interruption. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

In December 2024, Google engineers introduced a new internal service to facilitate access to customer metadata as part of storage consolidation efforts. As part of this change, we made configuration changes to read APIs for customer data which were implemented in stages to fully adopt this new service. These read APIs act as a canonical source of truth for validating the users. However, on Tuesday, 28 January, 2025, this service experienced elevated latency for all read APIs resulting in timeout with deadline exceeded errors thereby resulting in latency on the Admin Console.

This was triggered by a high error rate observed in a specific task associated with an internal customer data service. This service provides customer data to the system hosting the internal read APIs. The root cause of the issue was identified as the following factors:

• Increased latency on ‘read’ operations from the data storage system.
• High latency in internal communication due to the increased time spent waiting for the affected task to be processed.

Due to these factors, processing threads were unavailable to handle incoming requests quickly, creating a backlog and ultimately leading to timeouts, and errors.

The resulting impact was elevated error rates and latency for this internal read API, causing downstream service disruptions and impacting availability for Administrators, the Admin Console.

The impact also cascaded to Workspace Customer Support, resulting in approximately 10% of customers being unable to create support cases and initiate chats.

Remediation and Prevention

Google engineers were alerted to the outage via our internal monitoring system on Tuesday, 28 January, 2025, at 07:26 US/Pacific and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers re-directed traffic away from the impacted metadata service to an alternate service. Because both services utilized the same underlying data, there were no issues with respect to data consistency.

We apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future.

• Perform configuration changes to improve the performance of the service.
• Add a mandatory check of server’s configuration for every change made to production to ensure high quality of service before going live.

Detailed Description of Impact

On 28 January, 2025 from 07:26 to 09:34 US/Pacific:

Workspace Admin Console and Help Assistant

• Some customers experienced elevated latency and ‘500’ errors when accessing the Admin console.
• Some customers could access Admin Console normally, but experienced issues launching ‘Help Assistant’.

Workspace Support

• Some customers experienced issues attempting to open new support cases via Workspace support portal and chat.
• Customers that had issues with opening cases via the support portal were redirected to our backup Contact Us Form (CUF).
• Customers that had issues opening cases via chat were redirected to the support portal to create email based cases.

Mini Incident Report

We apologize for the inconvenience this service outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 28 January 2025, 07:26

Incident End: 28 January 2025, 09:34

Duration: 2 hours, 8 minutes

Affected Services and Features:

Workspace Admin Console and Help Assistant

Workspace Support

Regions/Zones: Global

Description:

Workspace Admin Console and Workspace Support experienced availability issues intermittently for a duration of 2 hours, 8 minutes. Based on preliminary investigation, the incident was caused by an overloaded metadata service that led to issues with fetching the customer metadata.

The issue was mitigated by re-directing traffic away from the impacted metadata service to an alternate service. The traffic will be served from the alternate metadata service until we complete the root cause analysis and fix the underlying issue.

Google will complete a full investigation in the following days that will provide a full root cause.

Customer Impact:

Workspace Admin Console and Help Assistant:

• Some customers experienced elevated latency and ‘500’ errors when accessing the Admin console.
• Some customers could access Admin Console normally, but experienced issues launching ‘Help Assistant’.

Workspace Support:

• Some customers experienced issues attempting to open new support cases via Workspace support portal and chat.
• Customers that had issues with opening cases via the support portal were redirected to our backup Contact Us Form (CUF).
• Customers that had issues opening cases via chat were redirected to the support portal to create email based cases.

Summary:

Admin Console may cause slowness or errors for impacted users globally.

Diagnosis:

Impacted users may have experienced slowness or seeing 500 errors while accessing Admin console.

Closing Message:

The issue with Admin Console has been resolved for all affected users as of Tuesday, 2025-01-28 09:40 US/Pacific. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we worked on resolving the issue.

Summary:

Admin Console may cause slowness or errors for impacted users globally.

Description:

We are experiencing an issue with Admin Console beginning on Tuesday, 2025-01-28 07:48 US/Pacific.

Our engineering team has identified the suspected root cause and is working on mitigation.

We will provide an update by Tuesday, 2025-01-28 11:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis:

Impacted users may experience slowness or seeing 500 errors while accessing Admin console.

Workaround:

None at this time.