SearchWithAggregation Zero Day Vulnerability Public Disclosure
Incident Report for CVaaS
Postmortem

Summary

  • Date: Apr 26, 2024 — Date of Discovery. May 8, 2024 — Date of Public Disclosure.
  • Tracking Bug: BUG946427
  • Impacted Clusters: All CVaaS (CloudVision as a Service) service regions. All CV-CUE (CloudVision Cognitive Unified Edge) service regions.
  • Authors: CVaaS (CloudVision as a Service) SRE Team
  • Status: Resolved
  • Summary: Arista internally discovered a zero day bug in two of our publicly exposed APIs for Search functionality (specifically, SearchWithAggregation, SearchWithAggregationStream), which had a gap in their authorization code that allows issuing aggregation search queries across tenants.
  • Impact: A review of our logs over the last 400 days has shown that neither of these APIs have ever been called by any external party. We are confident that this vulnerability has not been discovered and exploited.
  • RCA: Bug in authorization code in specific Search APIs.
  • Detection: Internal Code Audit

Root Cause Analysis

CloudVision offers a number of APIs that allow interacting with various services. Two of those APIs, SearchWithAggregation and SearchWithAggregationStream, had a gap in their common authorization code, thus allowing any authenticated user to issue a request to search data and return aggregate results across another tenant on the same cluster. For this attack, the attacker would require:

  1. An authorized user on a tenant to be an intentionally malicious actor
  2. Successfully guess the name of another tenant on the same cluster to exploit

This issue only impacts CVaaS, therefore there is no CVE number for on-prem customers. If there was a CVE, the CVSS Base Score for this vulnerability would be 6.3 (AV:N/AC:H/PR:L/UI:N/S:C/C:H/I:N/A:N).

A SearchWithAggregation request can be sent to search a given dataset and aggregate results using a given aggregation function, in up to a maximum of 10k buckets. One could craft a query that selectively matches data and returns it using an aggregation function that exposes parts of the original data, such as device serial numbers or MAC addresses. This API does not allow retrieving raw search results therefore the potential for data extraction was already limited in scope.

Timeline

The vulnerability was found by a CVaaS SRE (Site Reliability Engineer) on April 23, 2024 while doing a code audit. On April 26 a working exploit was developed. Within a few hours a fix with additional unit tests was merged in our code base. The fix was deployed and tested to our development and staging environments on April 27, and reached all production clusters later that same day, taking overall less than 24h since the vulnerability was confirmed.

Follow up Analysis

On April 27, 28, and 29 the CVaaS SRE team analyzed the last 400 days of audit logs from all our production clusters and combed through all the calls to the two affected APIs. These APIs have never been widely advertised and are only used by our own applications internally, and all the calls in our logs were from internal sources, thus giving us the confidence that nobody outside of Arista had ever called these APIs, and thus that this vulnerability had never been found and exploited.

We know all the calls in our logs were from internal sources based on the combination of two things: the source IP address of the caller and the principal of the caller. Calls to these APIs must be authenticated, either with a client-side certificate (mTLS) or a JWT, and the principals were consistently one of two service accounts used internally by CVaaS and CV-CUE. We audited the code of the two applications using those two service accounts and concluded that customers did not have any control over the relevant parameters of the affected API calls and therefore could not have possibly exploited this vulnerability indirectly through those applications.

During the week of April 29, the Arista engineering team conducted further code reviews and testing to ensure that no other similar bug was present in our API surface. We reviewed over 100 APIs and have not found anything else similar that allowed breaking multitenant boundaries.

Next Steps

As per our SOC-2 controls, at least once a year Arista hires a third-party security company to perform penetration testing. We believe this flaw was not found during previous pentesting cycles because these APIs have never been widely advertised. Pentesters observing the expected behavior of the product by snooping on activity of web browsers or Arista devices connected to CVaaS would never have come across either of these APIs. We will use what we've learned from this vulnerability to enhance our next scheduled pentesting, which was already scheduled prior to this discovery and will be complete by next month.

Additionally, as per our SOC-2 controls, Arista internally performs security scans of CVaaS on a quarterly basis, CVE scans on a weekly basis, and security training on an annual basis. Arista is active on VINCE (Vulnerability Information and Coordination Environment), which is designed by the CERT Coordination Center (CERT/CC) for multi-party vulnerability coordination, including for notifying vendors of new zero-day security vulnerabilities disclosed under embargo before public release.

If you have any questions please feel free to reach out to Arista TAC at support@arista.com or click here for additional ways to reach us.

Posted May 07, 2024 - 23:24 UTC

Resolved
This is the public disclosure of a zero day vulnerability discovered in CVaaS. At this time no action is required from users or account owners. Please review the postmortem section for details of the disclosure.
Posted May 07, 2024 - 23:21 UTC
This incident affected: euwest-2 (Core Platform, CV-CUE), apnortheast-1 (Core Platform, CV-CUE), us-central1-a (Core Platform, CV-CUE), us-central1-c (Core Platform, CV-CUE), ausoutheast-1 (Core Platform, CV-CUE), na-northeast1-b (Core Platform, CV-CUE), and uk-1 (Core Platform, CV-CUE).