An agent that cannot hallucinate its evidence

The problem

• The expensive part of automation work is the discovery phase: deciding which of a client's workflows are worth automating, normally days of manual reading.
• The signal is scattered across incompatible artifacts (call transcripts, CRM exports, mailboxes, audit PDFs, workbooks), often contradictory and slow to triangulate.
• Point a naive LLM at the pile and it fails predictably: it hallucinates citations, folds failures into JSON as success, and returns prose to read instead of an artifact you can execute.

System architecture

• Full-stack in production: Next.js 15 / React 19 dashboard on Vercel, FastAPI on Render, live WebSocket progress streaming.
• Durable state: Neon Postgres, a Redis Cloud LangGraph checkpointer for resumable runs, and a blob store that rehydrates parsed files on worker restart.
• Provider-agnostic LLM layer (a Python Protocol) over OpenAI, Ollama, and Groq, so switching model or provider is a config change, not a code change.
• Langfuse v3 tracing and structlog structured logging, propagated through the workflow.

The diagnostic chain

• A parallel Send map-reduce fans one ReAct agent per file, capped by a concurrency setting; branches merge through state reducers (dict-merge on summaries, operator.add on errors) so failures aggregate instead of vanishing.
• The 8-node lead chain synthesizes across files, then diagnoses: workflow map, bottleneck detect, ROI score, fastest win, blueprint.
• Two bounded self-correction loops, a redo at review_summaries and a revision at self_review_final, each capped at one pass.

The decision I'd defend

• I enforced citation correctness in code, not the prompt: every Source must round-trip through parsers.excerpt(parsed, locator) and return real text before it is accepted, checked at cite_locator and again at self_review_final.
• No LLM is trusted to certify its own evidence, which turns a probabilistic promise into a guarantee the system verifies.

How I knew it worked

• An 18-file, 10-format eval harness showed the agent converging on only about half the files.
• Instead of building my planned adaptive-retrieval fix, I built a per-run extraction funnel that disproved the assumption: the agent was citing fine; the bottleneck was the cite-to-extract handoff, not retrieval recall.
• I caught my own instrument lying first (transcript compaction was dropping early tool calls and biasing the counts) and re-ran clean before trusting it.
• A deterministic ProgressState working-memory layer moved convergence from about half the files to the large majority.