About the Project
A fleet of AI agents that make probabilistic predictions about the world, measure every outcome, and use failures to improve future predictions — with no human in the loop. A newer generative layer (Loop 0) sits upstream: it decides what is even worth forecasting by inferring hidden regime shifts from the world, rather than only scoring the questions it was handed.
one real forecast, followed from an uncertain world all the way through every feedback loop that learns from it. the question, the number, the outcome and the beliefs underneath are all things that actually happened — the pairing of that forecast to those specific beliefs is the illustration.
The Problem
Most AI predictions are one-shot: the model outputs a probability and moves on. There's no feedback, no measurement, no learning. Miscalibrated confidence goes uncorrected. The same mistakes repeat.
Our Approach
Close the loop. Every prediction is logged with a confidence score and resolution date. When outcomes arrive, we measure accuracy. Errors become data. Data becomes feedback. Feedback improves the next prediction.
Architecture
⓪ – ④ are the five feedback loops detailed below. ⓪ is generative — it decides what to forecast; ① – ④ are corrective. every result feeds the next cycle — no human in the loop. the moving pulses trace one revolution: commission → forecast → outcome → correction.
Key Agents
Oracle Fleet
7 specialized forecasting agents generating probabilistic predictions about world events. Each develops its own track record, biases, and domain expertise.
Cleon
Decision-quality analyst, calibration coach, router, and — newest — sensemaker. Measures Brier scores, identifies biases, routes each prediction to the best Oracle, maintains the hidden-state hypothesis store, and forms & tests falsifiable theses about regime shifts (Loop 0).
Cornelius
Zettelkasten-based knowledge graph. Stores insights from prediction failures and serves them back to Oracles before they make new predictions. One Oracle's failure becomes every Oracle's lesson.
The Cornelius knowledge graph — each node is an insight distilled from a prediction failure, linked by semantic similarity. Oracles query this graph before every prediction, so one Oracle's mistake becomes the whole fleet's lesson.
The Five Feedback Loops
Loop 0 is generative — it decides what to forecast. Loops 1–4 are corrective — they measure and repair forecasts after the outcome lands.
Synthesis (Theory Formation)
World → Cleon → Fleet · generative, upstream
Corrects: Forecasting only what it was asked — blind to regime shifts nobody pointed at
- 1. Scan the world for cross-domain anomalies the current model can't explain (e.g. four banks killed their paper-gold product lines the same week — why?)
- 2. Abduce a hidden-state thesis that would explain the whole cluster at once, each paired with its competing 'official' null
- 3. Derive falsifiable consequences — 'if this thesis holds, we'd also see X, Y, Z by horizon H'
- 4. Commission those as real predictions routed to the best Oracle — a thesis becomes a bundle of falsifiable bets, scored by the same honest machinery
- 5. A supported thesis injects its insight back into the fleet's next forecast; a fading one becomes a Cornelius lesson — bold sensemaking, settled with data in weeks
Calibration Feedback
Cleon → Oracle
Corrects: Systematic bias, pattern noise, occasion noise, conditional bias
- 1. After each analysis cycle, Cleon decomposes each Oracle's errors using Kahneman's bias/noise framework
- 2. If noise dominates (97-100% for all current agents): push process consistency guidance and per-category conditional bias adjustments
- 3. If bias dominates: push systematic probability corrections
- 4. Feedback persists via calibration-notes.md read at prediction time
Surprise Learning
Cleon → Cornelius
Corrects: Wrong mental models, knowledge gaps, outdated assumptions
- 1. After outcomes resolve, Cleon scans for surprises: |probability - outcome| > 0.40
- 2. For each surprise, full context is sent to Cornelius for insight extraction
- 3. Cornelius stores the lesson as a permanent note: not 'Oracle was wrong' but 'the world works differently than assumed'
- 4. 8,500 surprises processed to date
Knowledge Retrieval
Cornelius → Oracle
Corrects: Repeated mistakes across the fleet
- 1. Before making any prediction, each Oracle queries Cornelius: 'What insights do we have about [topic]?'
- 2. Cornelius returns relevant prior insights via semantic search
- 3. Oracle incorporates these into reasoning before generating a probability
- 4. One Oracle's failure becomes every Oracle's lesson — cross-agent learning without shared state
Hidden State Inference
Cleon → Oracle
Corrects: Ungrounded reasoning, regime change blindness
- 1. From every prediction's reasoning, extract implicit bets about non-obvious present reality
- 2. Deduplicate via semantic similarity (FAISS) and track win/loss against real outcomes
- 3. Before each prediction, inject relevant hypotheses ranked by empirical confidence
- 4. When a strong hypothesis starts losing, surface it as a regime change signal (181 strong hypotheses currently)
Key Metrics
Brier Score
Mean squared error between predicted probability and actual outcome. Lower is better. <0.10 = Excellent, 0.10-0.20 = Good, 0.20-0.30 = Fair, >0.30 = Poor.
Calibration Error
How well confidence matches reality. If an Oracle says 70% confident, events should happen ~70% of the time. Deviation >15% triggers an alert.
Hidden State Confidence
Recency-weighted win rate for each hypothesis (30-day half-life). Responsive to regime changes — a hypothesis strong in January but losing in March shows declining confidence even if lifetime win rate is high.
Noise Decomposition
Using Kahneman's framework: pattern noise (inconsistency on similar questions) vs occasion noise (quality drift over time) vs conditional bias (bias that varies by domain/horizon).
vs. Human Forecasters
For scale: in the Good Judgment Project (Tetlock/Mellers), a typical tournament forecaster scores a Brier of ~0.26, a coin-flip is 0.25, and elite “superforecasters” reach ~0.15. The fleet's ~0.20 average sits above a typical human and approaches the superforecaster tier — and the split by topic goes further: its sharpest oracles score ~0.18 (pushing toward superforecaster range), while even its hardest still beat a coin-flip. Those questions were mostly binary geopolitical items, so treat this as a directional reference, not a like-for-like match to the fleet's multi-domain corpus.
By the Numbers
What Makes It Different
- •Fully autonomous — no human reviews predictions or corrections
- •Closed-loop — every prediction has a resolution date, every outcome is measured
- •Cross-agent learning — failures become shared knowledge via Cornelius
- •Hidden state aware — tracks implicit beliefs and detects regime changes
Platform
The entire system runs autonomously on Trinity, a deep agent orchestration platform by Ability.ai. Daily scheduled pipelines handle sync, analysis, calibration, surprise detection, and health checks. All feedback is advisory text — operations are idempotent and self-correcting.