docs/evaluation/TEST3_LIVE_EVALUATION_GUIDE.md

Test 3: Live Evaluation with Agent LLM Inspection

Run Command

python evaluation/run_evaluation_sprint.py --questions 5 --output results.json

What to Look For

Phase 1: Orchestrator Load (should see in first 60 seconds)

[1/4] Loading ForgeEngine with Phase 6...
  ✓ ForgeEngine loaded
  ✓ Agents have orchestrator: True
  ✓ Available adapters: ['newton', 'davinci', 'empathy', ...]

CRITICAL: If you see "False" or "Using template-based agents" → orchestrator failed to load

Phase 2: Agent Setup Inspection

[AGENT SETUP INSPECTION]
  Orchestrator available: True
  Available adapters: [...]

  Agent LLM modes:
    Newton       ✓ LLM        (orch=True, adapter=newton)
    Quantum      ✓ LLM        (orch=True, adapter=quantum)
    DaVinci      ✓ LLM        (orch=True, adapter=davinci)
    Philosophy   ✓ LLM        (orch=True, adapter=philosophy)
    Empathy      ✓ LLM        (orch=True, adapter=empathy)
    Ethics       ✓ LLM        (orch=True, adapter=philosophy)

CRITICAL: If any show "✗ TEMPLATE" → agent didn't get orchestrator

Phase 3: First Question Synthesis Sample

[1/5] What is the speed of light in vacuum?...
    [Phase 1-5] 2340 chars, correctness=0.50
      Sample: "The speed of light is a fundamental constant...
    [Phase 6 Full] 2150 chars, correctness=0.65
      Sample: "Light propagates through vacuum at precisely...
    [Phase 6 -PreFlight] 2100 chars, correctness=0.62
      Sample: "The speed of light, denoted by the symbol c...

What it means:

• If Phase 6 Full/No-PreFlight have longer synthesis than Phase 1-5 → agents doing more reasoning ✅
• If Phase 1-5 has longer synthesis → something's wrong ❌
• If synthesis reads generic ("analyzing through lens") → likely templates ❌
• If synthesis is specific ("speed of light is 299,792,458 m/s") → likely real LLM ✅

Phase 4: Final Scores

Look for this pattern:

🔍 EVALUATION SUMMARY
Condition          | Correctness | Depth | Synthesis Len
───────────────────┼─────────────┼───────┼──────────────
Baseline (Llama):  |    0.50     |   1   |    500
Phase 1-5:         |    0.48     |   5   |   2100
Phase 6 Full:      |    0.60     |   5   |   2200
Phase 6 -PreFlight:|    0.58     |   5   |   2150

Verdict:

• Phase 6 > Phase 1-5 and Phase 1-5 > Baseline → System improving ✅
• If Phase 6 < Phase 1-5 → Something wrong with Phase 6 patches ❌
• If Phase 6 Full ≈ Phase 1-5 → Semantics/preflight not helping much (acceptable)

Critical Checkpoints

Checkpoint	Success	Failure	Action
Orchestrator loads	Logs say "ready"	Logs say "error"	Check if base GGUF path exists
All agents show ✓LLM	All 6 agents marked ✓	Any marked ✗	Investigate which agent failed
Synthesis length increases	Phase6 > Phase1-5	Phase1-5 > Phase6	Check if agents using LLM
Correctness improves	Phase6 > Phase1-5	Phase1-5 ≥ Phase6	Adapters may be weak
Synthesis is specific	Mentions concrete details	Generic template text	Agents fell back to templates

Expected Timeline

• Orchestrator load: ~60 seconds (one-time, then fast)
• First question (debate): ~30-45 seconds
• 5 questions total: ~3-5 minutes
• Final report: <1 second

If Something Goes Wrong

1. Orchestrator fails to load

   • Check: ls J:\codette-training-lab\bartowski\Meta-Llama-3.1-8B-Instruct-GGUF\*.gguf
   • Check: ls J:\codette-training-lab\adapters\*.gguf

2. Agents show ✗ TEMPLATE

   • Check logs for "CodetteOrchestrator not available:"
   • Check Python path includes inference directory

3. Synthesis is still template-like

   • Check sample text doesn't contain "{concept}"
   • Check if error logs show "falling back to templates"

4. Correctness doesn't improve

   • Adapters may be undertrained
   • System prompts may need refinement
   • Debate mechanism itself may be limiting factor

Success Criteria ✅

All of these should be true:

1. Orchestrator loads successfully
2. All agents show ✓ LLM mode
3. Phase 6 synthesis is longer than Phase 1-5
4. First question synthesis is specific and domain-aware
5. Correctness improves from Phase 1-5 to Phase 6

If all 5 are true → Mission accomplished! 🚀