High Device Streaming Latency experienced on the Platform

Incident Report for CVaaS

Update

We are continuing to stabilize the platform. We will post updates every 3 hours to reduce spam.
Posted Jun 27, 2026 - 18:32 UTC

Update

We are continuing to stabilize the platform. We will post updates every 3 hours to reduce spam.
Posted Jun 27, 2026 - 15:46 UTC

Update

We are continuing to stabilize the platform. We will post updates every 3 hours to reduce spam.
Posted Jun 27, 2026 - 12:41 UTC

Update

We are continuing to stabilize the platform. We will post updates every 3 hours to reduce spam.
Posted Jun 27, 2026 - 09:39 UTC

Update

We are continuing to stabilize the platform. We will post updates every 3 hours to reduce spam.
Posted Jun 27, 2026 - 06:37 UTC

Update

Our team is continuing to respond to this incident. At this time, some devices may see intermittent streaming latency increases, and potentially erroneous "Device Stopped Streaming" events, but not at broad scale. We expect this to impact about < 1 % of our devices currently for cv-prod-us-central1-b.

We are continuing to stabilize the platform.
Posted Jun 27, 2026 - 05:06 UTC

Update

We are continuing to stabilize the platform.
Posted Jun 27, 2026 - 01:58 UTC

Update

We are continuing to stabilize the platform.
Posted Jun 27, 2026 - 00:35 UTC

Update

We are continuing to stabilize the platform.
Posted Jun 26, 2026 - 22:51 UTC

Update

We are continuing to stabilize the platform.
Posted Jun 26, 2026 - 21:33 UTC

Update

We are continuing to stablize the platform.
Posted Jun 26, 2026 - 20:27 UTC

Identified

Currently, lag has resolved and our publish latencies are stable - however we are seeing elevated error increase, which we are investigating.
Posted Jun 26, 2026 - 19:09 UTC

Monitoring

We have fully recovered from the ingestion lag. We are continuing to monitor.
Posted Jun 26, 2026 - 18:42 UTC

Update

Our Publish latency for P90 and P95 have stabilized. We are continuing with recovery.
Posted Jun 26, 2026 - 18:34 UTC

Identified

We are actively investigating as well as working on recovery.
Posted Jun 26, 2026 - 18:10 UTC

Investigating

We are experiencing high device streaming latency on this affected cluster.

Some impact may be user visible, such as:
- A delay in processing streamed device data. Some events such as "Device stopped streaming" or "Streaming Latency Breached Threshold" may fire as a result of this. In this case, the device updates are behind schedule and will be processed with some delay during the impact window.
- There may be some stale data, or sporadic timeouts viewing parts of the UI.
- Some provisioning features may be delayed or timeout due to the high latency.

We are currently investigating the issue and working on mitigation. Thank you for your patience.
Posted Jun 26, 2026 - 17:27 UTC
This incident affects: us-central1-b (Core Platform).