NL2SQL AI Agent
A reflective agent that turns plain English into verified SQL and catches its own mistakes.
3 days to <1hr
Query resolution time
0.71
Retrieval precision
RAGAS evaluation
60%
Fewer hallucinations
3
Bounded repair attempts
See it work
The problem
Business stakeholders needed answers that lived in a relational database but could not write SQL, so every question became a ticket for an analyst. Turnaround averaged three days, and the backlog meant simple questions never got asked at all.
Approach
Schema metadata is chunked per table and embedded into ChromaDB, so each question retrieves only the handful of tables it actually needs instead of the entire catalog. This keeps the prompt small and cuts the model's opportunity to invent columns.
Generation runs inside a LangGraph state machine rather than a single call. The graph generates SQL, validates it against the retrieved schema, executes it, and inspects the result. Any failure routes back to a repair node with the error message attached as context.
The repair loop is bounded at three attempts. Each retry sees the previous SQL and the exact database error, which is usually enough to fix a bad column reference or a broken join on the second pass.
Quality was measured rather than assumed. RAGAS evaluation over a labelled question set produced 0.71 retrieval precision and confirmed a 60% drop in hallucinated fields after the schema-retrieval and prompting changes.
Architecture
Decisions and tradeoffs
Retrieve the schema instead of pasting it
Embed per-table schema chunks in ChromaDB and retrieve top-k per question.
- Why
- A full schema dump inflated token cost and gave the model far more surface area to hallucinate against. Narrow context measurably improved correctness.
- What it cost
- Retrieval can miss a table that a question genuinely needs, which turns a hallucination problem into a recall problem. Measuring retrieval precision separately is what made this tractable.
A state graph rather than a single prompt
LangGraph nodes for generate, validate, execute, and repair.
- Why
- Text-to-SQL fails in predictable ways, including bad column names and invalid joins. Those failures are recoverable if the error is fed back, and a graph makes that routing explicit and inspectable.
- What it cost
- Latency rises on any question that needs repair, and the graph is more code to maintain than a single call would be.
Cap the retry loop at three
Bounded retries with the error message carried into each attempt.
- Why
- Nearly all recoverable failures resolved by the second attempt. An unbounded loop mostly burns tokens on questions that were never answerable.
- What it cost
- Genuinely hard questions fail rather than eventually succeeding, so the agent has to say so honestly instead of guessing.
Evaluate with RAGAS before tuning prompts
Measure retrieval precision and hallucination rate on a labelled set.
- Why
- Without a baseline, prompt changes are guesswork. Numbers made it clear that retrieval, not phrasing, was the dominant error source.
- What it cost
- Building and labelling the evaluation set took real time up front before any accuracy gain appeared.