Elevated error rate

Incident Report for CVaaS

Resolved

This incident has been resolved. We will continue to work with Google Cloud on the postmortem investigation and action items to ensure this doesn't reoccur.

Posted Sep 10, 2024 - 09:24 UTC

Monitoring

The service is back to normal, we are monitoring the situation. The service suffered an elevated error rate between 08:06 UTC and 08:28 UTC due to a GKE infrastructure component upgrade. We will start our postmortem investigation soon to ensure such upgrades are not disruptive again in the future.

Posted Sep 10, 2024 - 08:43 UTC

Identified

We identified the root cause and are working on mitigation.

Posted Sep 10, 2024 - 08:38 UTC

Investigating

We're currently seeing an elevated error rate on this cluster and are investigating.

Posted Sep 10, 2024 - 08:17 UTC

This incident affected: us-central1-a (Core Platform).