Español

Put a Confidence Score Between Regex and the LLM

The secret layer between regex and your LLM is a confidence score. Most teams building extraction pipelines pick a side — pure regex (fast, brittle) or pure LLM (flexible, slow, billed per token). The teams that ship cheap and accurate systems don't pick. They put a scorer in the middle and let each item choose its own path.

Why a middle layer at all

In pipelines over structured or semi-structured text — invoices, log lines, product titles, form fields — a well-tuned deterministic parser typically handles the large majority of inputs, often 95%+ correctly, at effectively zero marginal cost and sub-millisecond latency. An LLM call for the same item costs real money and hundreds of milliseconds to seconds.

Sending everything to the model means paying model prices for work a regex already does. Sending nothing means silently mis-parsing the weird 2–5%. The fix is a cascade: parse deterministically, score how much you trust that parse, and escalate only below a threshold. This is the same idea behind LLM cascades in the FrugalGPT paper, which showed cascading from cheap to expensive models can match top-model quality with up to 98% cost reduction on their benchmarks.

The three-stage pipeline

1. Parse. Run the deterministic layer first — regex, a grammar, or a rules engine. It emits candidate fields.
2. Score. A small function assigns each parse a confidence in [0, 1] from concrete signals (below).
3. Route. At or above the threshold (0.95 is a sane starting point): accept the parse directly. Below it: send only that item to the LLM, ideally with the failed parse attached as context.

result = regex_parse(text)
score  = confidence(result, text)

if score >= 0.95:
    return result                    # free path — most items
return llm_parse(text, hint=result)  # paid path — edge cases only

What goes into the score

You don't need a trained model to start. Cheap, honest signals:

Start with a hand-weighted combination of these. If you later want the score to mean something ("0.9 = right 90% of the time"), calibrate it against a labeled sample — Platt scaling and isotonic regression are the standard tools, both built into scikit-learn's calibration module.

Picking the threshold

The threshold is a business dial, not a constant. Lower it and you spend less but accept more silent errors; raise it and you buy accuracy with LLM calls. Measure two numbers on a labeled sample: escalation rate (what fraction goes to the model) and residual error rate (how often an accepted parse is wrong). Move the threshold until both fit your budget and your error tolerance — then log every routing decision so you can re-tune from production data.

Power-user moves

• Score the LLM too. Ask the model for structured output and inspect token logprobs — they give you a per-prediction confidence you can threshold, so even the escalated path can flag "still unsure" items for human review instead of guessing.
• Feed the failed parse to the model. Passing the regex's partial result as a hint turns the LLM call into a repair job, which is shorter and cheaper than parsing from scratch.
• Cascade models, not just parsers. The same score-then-escalate trick works between a small model and a large one. Open-source routers like RouteLLM report cutting costs by up to 85% while keeping ~95% of top-model quality by routing easy queries to the cheap model.
• Watch for drift. When upstream formats change, escalation rate spikes before accuracy drops. Alert on it — it's your cheapest early-warning signal.

Resources

• FrugalGPT: How to Use Large Language Models While Reducing Cost (arXiv)
• Probability calibration — scikit-learn user guide
• Using logprobs for classification confidence — OpenAI Cookbook
• RouteLLM — open-source LLM routing framework
• re — regular expression operations, Python docs