Evident
An evidence-grounded AI decision system that ranks outreach targets, cites why, and explicitly refuses to recommend when confidence is too low, with bounded cost and a full audit trail.
The problem
LLMs sound confident even when evidence is thin, and a bad output becomes a bad decision at scale. Evident grounds every decision in retrieved evidence, exposes the reasoning, and returns "insufficient evidence" instead of guessing.
What it does
- Takes a people-directory URL plus a research interest as input.
- Returns a ranked shortlist, each with reasoning, cited evidence, and an outreach draft.
- Three explicit outcomes per contact: recommended, not recommended, or insufficient evidence (a refusal).
- A deterministic pre-filter drops weak candidates before spending model budget.
how it fits together
Evidence-grounded decision pipeline
Retrieval-grounded evaluation that ranks outreach targets, cites its evidence, and refuses when support is too thin.
- Contact source (pluggable)Directory parser or evidence_scraper → RawContact[]: name, title, email, research text, evidence, identity signals.
- Clean & enrichDedup, evidence chunks, identity scoring.
- Deterministic pre-filterDrops weak candidates before any model spend.
- LLM evaluation · triage modelCheap first pass over the shortlist (Claude, retrieval-backed).
- Refuse-when-weak gateStructural floor enforced across every path, so an over-confident model cannot upgrade a thin contact.
- Bounded loop · escalate uncertain≤1 adaptive retrieval + 1 re-eval on the primary model, then finalize.
- Hybrid rank → drafts (top only)AI fit + evidence strength + seniority · persist + full audit trail.
product screens
key engineering decisions
One refuse-when-weak gate across every path
The uncertainty gate applies across the LLM path, the heuristic fallback, and the second pass, so an over-confident model can't upgrade a thin contact.
Cost-safe LLM usage
A deterministic pre-filter plus per-run caps on evaluations, drafts, retries, and fetches, with timeouts and backoff, keeps cost bounded.
results & outcomes
- Ranked targets with cited reasoning, explicit refusals on weak evidence, and ~60% fewer unnecessary model calls.
- Prompt-injection defense on untrusted scraped text: a 12-pattern sanitizer delimits and neutralizes injected instructions, 3/3 attempts caught.
- A hard insufficiency floor (at least two evidence items, sufficient research text, verified identity) the model cannot override, so it refuses instead of guessing.
- 13 offline tests gate CI (injection plus decision guarantees) against a stubbed model with no API key, so the safety properties hold on every push.
- Full per-contact audit trail: score breakdown, cited evidence, confidence justification, and decision revisions.
- Deployed on AWS ECS/Fargate (Docker), run on-demand, with the API key injected via AWS Secrets Manager.
How Evident grounds every decision in retrieved evidence, uses a bounded agentic loop, and refuses to recommend when the evidence is too thin.