# AI Root Cause Analysis Benchmark

### Why we need an AI RCA benchmark

- Every AI vendor claims their solution is "the smartest" with zero proof
- No standardized way to validate AI effectiveness before purchase
- Customers can't compare tools objectively - it's all marketing
- Testing requires complex production environments most teams don't have

### Proposed solution

- Use [OpenTelemetry Demo](https://opentelemetry.io/docs/demo/) - a multi-language microservices app (Java, Python, Go, .NET, Rust, PHP, etc.)
- Built-in failure scenarios controlled by simple [feature flags](https://opentelemetry.io/docs/demo/feature-flags/#implemented-feature-flags)
- Every vendor gets identical test conditions - no excuses
- Provides basic telemetry (metrics, logs, traces) with freedom for vendors to add their own data sources
- Easy to reproduce independently during your POC process

## Benefits for everyone

A standardized benchmark helps customers, vendors, and the entire observability ecosystem

### Vendor Comparison

Make informed decisions by comparing different AI RCA tools on identical scenarios

### Validation

Verify the effectiveness of your chosen AI solution before deploying to production

### Transparency

Open methodology allows you to understand exactly how each tool performs

### Reproducibility

All scenarios are based on open-source demos that anyone can run and verify

## OpenTelemetry Demo Overview

The OpenTelemetry Demo is a microservices-based e-commerce application that represents a realistic distributed system with controllable failure scenarios - perfect for testing AI root cause analysis tools.

### Multi-language services

Built in Java, Python, Go, .NET, C++, JavaScript, Kotlin, Ruby, PHP, Rust, and TypeScript

### Continuous load generation

Realistic traffic patterns across all services

### Open source

Anyone can run identical test environments for fair comparison

### Telemetry data

Logs, metrics and traces

### Reproducible failures

Feature flags activate specific failure modes consistently

## Benchmark of Coroot's AI RCA

Coroot combines standard OpenTelemetry data with its own eBPF-based telemetry to deliver precise root cause analysis powered by AI.

### [Product catalog failure](https://coroot.com/ai-benchmark/scenarios/productCatalogFailure)

Generate an error for GetProduct requests with specific product ID: OLJCESPC7Z.

Feature flag: `productCatalogFailure` [View details](https://coroot.com/ai-benchmark/scenarios/productCatalogFailure)

### [Ad service manual garbage collection](https://coroot.com/ai-benchmark/scenarios/adManualGc)

Triggers manual garbage collection in the ad service to simulate GC pressure and performance impact.

Feature flag: `adManualGc` [View details](https://coroot.com/ai-benchmark/scenarios/adManualGc)

### [Homepage traffic flood](https://coroot.com/ai-benchmark/scenarios/loadgeneratorFloodHomepage)

Start flooding the homepage with a huge amount of requests to simulate traffic spikes.

Feature flag: `loadgeneratorFloodHomepage` [View details](https://coroot.com/ai-benchmark/scenarios/loadgeneratorFloodHomepage)

### [Cart service failure](https://coroot.com/ai-benchmark/scenarios/cartFailure)

Simulates Redis connection failures in the cart service, causing EmptyCart operations to fail with 15+ second delays.

Feature flag: `cartFailure` [View details](https://coroot.com/ai-benchmark/scenarios/cartFailure)

### [Recommendation service memory leak](https://coroot.com/ai-benchmark/scenarios/recommendationCacheFailure)

Introduces a memory leak in the recommendation service.

Feature flag: `recommendationServiceCacheFailure` [View details](https://coroot.com/ai-benchmark/scenarios/recommendationCacheFailure)