Postmortem
Hello CVaaS Customers,
We experienced a significant service disruption on June 12th, 2025, that lasted approximately 2 hours and 5 minutes.
Our service regions were impacted due to a disruption of services delivered by our cloud service provider, and not due to a security-related incident. There was no data loss during the impact window. Despite this, parts of our service and features were impacted, most notably in our Network Provisioning services, including Studios features. This was due to the nature of the underlying cloud service provider disruption. Service regions cv-prod-us-central1-b and cv-prod-us-central1-c experienced the most pronounced impact due to an unfortunate set of circumstances where deployments of new changes coincided with our cloud service provider's disruptions.
We continue to re-evaluate the designs and decisions made for CVaaS, and that, of course, includes our resiliency to disruptions like this. Our team will retrospect and act on systemic changes needed that arose from this event so we can continue to provide our users with a great product.
Again, we deeply apologize for the disruption to CVaaS that you've experienced today, and thank you, as always, for your patience.
Posted Jun 13, 2025 - 08:50 UTC
Resolved
This incident has been resolved.
Posted Jun 13, 2025 - 02:11 UTC
Monitoring
All service regions and their functionality are expected to be operational. Thank you for your patience.
Posted Jun 12, 2025 - 20:51 UTC
Identified
All non-US (cv-prod-us-central1-a, cv-prod-us-central1-b, cv-prod-us-central1-c) service regions and their functionality is expected to be operational. Please feel free to resume your usage of CVaaS on the following service regions:
- cv-prod-india-1
- cv-prod-uk-1
- cv-prod-ausoutheast-1
- cv-prod-apnortheast-1
- cv-prod-euwest-2
- cv-prod-na-northeast1-b
Posted Jun 12, 2025 - 20:21 UTC
Update
Network Provisioning stack that involves pushing new configuration changes is down at this time. Viewing historical change control or other network changes is expected to be functional.
Posted Jun 12, 2025 - 19:04 UTC
Investigating
We are currently investigating issue with our cloud provider. Some features may not be operational at this time. We'll provide update as soon as possible.
Posted Jun 12, 2025 - 18:46 UTC
This incident affected: euwest-2 (Core Platform, Network Provisioning, Events, Login, CV-CUE), apnortheast-1 (Core Platform, Network Provisioning, Events, Login, CV-CUE), us-central1-a (Core Platform, Network Provisioning, Events, Login, CV-CUE), us-central1-c (Core Platform, Network Provisioning, Events, Login, CV-CUE), ausoutheast-1 (Core Platform, Network Provisioning, Events, Login, CV-CUE), na-northeast1-b (Core Platform, Network Provisioning, Events, Login, CV-CUE), uk-1 (Core Platform, Network Provisioning, Events, Login, CV-CUE), us-central1-b (Core Platform, Network Provisioning, Events, Login, CV-CUE), and india-1 (Core Platform, Network Provisioning, Events, Login, CV-CUE).