# Observability and Dashboard

## Overview

Observability provides deep insight into runtime behavior, model usage, tool execution, memory quality, and rewards.

## Dashboard Sections

### 1. Live Thought Stream

- chronological reasoning notes
- model/router choice trace
- action confidence timeline
- override events

### 2. Navigation Map

Graph of visited pages:

- nodes = URLs
- edges = transitions
- node color = relevance/confidence
- revisit highlighting

### 3. MCP Usage Panel

- tool call count by server
- avg latency by tool
- error rate and retries
- top successful tool chains

### 4. Memory Viewer

- inspect short/working/long/shared memory
- filter by task/domain/confidence
- edit/delete entries
- prune previews

### 5. Reward Analytics

- per-step reward breakdown
- component contribution trends
- penalty heatmap
- episode comparison

### 6. Cost and Token Monitor

- per-provider usage
- per-model token counts
- cumulative cost vs budget
- forecasted burn rate

## Core Metrics

### Agent Metrics

- task completion rate
- avg steps to completion
- recovery score
- generalization score
- exploration ratio

### Tool Metrics

- tool success rate
- timeout ratio
- fallback frequency
- schema validation failures

### Memory Metrics

- retrieval hit rate
- relevance score distribution
- prune rate
- memory-assisted success ratio

### Search Metrics

- query success rate
- multi-hop depth distribution
- credibility score average
- duplicate result ratio

## Logging Model

Structured logs (JSON):

```json
{
  "timestamp": "2026-03-27T00:00:00Z",
  "episode_id": "ep_123",
  "step": 7,
  "event": "tool_call",
  "tool": "beautifulsoup.find_all",
  "latency_ms": 54,
  "success": true,
  "reward_delta": 0.08
}
```

## Tracing

Per-episode trace includes:

- observations
- actions
- rewards
- tool calls
- memory operations
- final submission and grader results

## Alerts

Configurable alerts:

- budget threshold crossed
- error spike
- tool outage
- memory bloat
- anomalous low reward streak

## APIs

- `GET /api/metrics/summary`
- `GET /api/metrics/timeseries`
- `GET /api/traces/{episode_id}`
- `GET /api/costs`
- `GET /api/memory/stats`
- `GET /api/tools/stats`

## Recommended Dashboard Layout

1. Top row: completion, cost, latency, error rate
2. Mid row: thought stream + navigation graph
3. Lower row: reward breakdown + MCP usage + memory viewer
4. Bottom row: raw trace and export controls

## Export and Audit

Exports:

- JSON trace
- CSV metrics
- reward analysis report
- model usage report

All exports include episode and configuration fingerprints for reproducibility.