Most automation demos stop at “the AI scored it and sent a message.” That is the easy 80%. The remaining 20% — what happens when the model is wrong, when the form is submitted twice, when a step fails at 2am — is the part that decides whether a system survives contact with real revenue. This case study is mostly about that 20%.
1:42 · Sound on for the walkthrough. Every screen is a live run, not a mockup.
| Client | B2B AI-automation agency (name withheld) |
|---|---|
| Engagement | Design and build an AI lead-qualification pipeline |
| Systems touched | CRM · team chat · company-data provider · AI scoring · audit log |
| Delivered | End-to-end qualification pipeline plus a reliability layer: failure alerting, duplicate protection, audit trail |
| Status | Running. ICP criteria and team routing change from one place |
The problem
Inbound leads all arrived looking the same: a name, an email, a short message.
The result was that a decision-maker with an approved budget queued behind someone browsing out of curiosity. To know who to call first, the team had to research every company by hand — size, industry, whether the request was serious — work that costs hours and usually lands after the lead has gone cold.
What was needed was not simply a system that uses AI. It was a system safe enough to run unattended on real leads — one that never pesters the team with duplicates, never silently drops a lead, and reports any fault the moment it happens.
What the system does
One workflow handles every new submission the moment it arrives.
- It filters before it spends. Personal email addresses skip the paid research step entirely — budget goes only to leads worth researching.
- It researches the company automatically. Size, industry, funding, age in market — none of which was in the form.
- It scores and tiers. Every lead gets a tier (A/B/C) and a buying-intent score out of 100, with written reasoning a rep can read and argue with — not an unexplained number.
- It routes to the right place. A-tier to the urgent sales channel with a 4-hour follow-up task; B-tier to the standard channel with two days; C-tier to long-term nurture with seven.
- It writes the opening line. Every notification carries an opener written for that specific lead, so the rep never starts from a blank page.
- It logs everything. Each processed lead becomes one row in a log the business owner opens directly, without touching the technical system.
The practical result: the phone rings before the prospect has closed the tab.
What it looks like in practice
Every screenshot below is from a real run. The lead data shown is test data created during build and verification; the company research ran against real domains.
What makes it safe to leave running
Most systems like this stop at “lead arrives, AI scores it, someone gets a message.” This one ships with three safeguards, each tested against real failure conditions before handover.
Immediate alerting on any fault. When something genuinely breaks, an alert names the step that stopped, the reason, and links straight to the failed run. Verified by deliberately breaking a live node and confirming the alert arrived — not a theoretical test.
Duplicate protection. Double-submitted forms are a fact of life. The system blocks reprocessing of the same lead within 24 hours — before it can spend research budget, ping sales twice, or create a duplicate task. Verified with a two-shot test: the first submission flowed through fully, the second was stopped cold with no side effects.
An audit trail the owner can read. Every lead is appended as a single row showing date, identity, tier, score, data confidence and routing. The business owner reads it directly, and it doubles as the memory the duplicate check relies on. One record doing two jobs, with nothing to keep in sync.
Three decisions that changed the outcome
The AI proposes; hard rules decide. Under test, the model was instructed not to exceed a given score when data was thin. It exceeded it. A model roughly 13× more expensive overshot further still. The fix was not paying more — it was a hard limit in code. That limit has held on every run since. The commercial consequence: the cheaper model became the correct choice, because the guarantee no longer depended on the model’s judgement at all.
Cost control is a design decision, not a later cleanup. Personal email addresses never reach paid research services, and weak matches are rejected at the source before they are billed. Spend follows lead value, not traffic volume.
A silent failure caught in testing, not production. During duplicate-protection testing, it emerged that the very first new lead could — in one specific case — exit the pipeline with no error at all. New leads would have disappeared silently. It surfaced because failure paths were tested, not just success paths, and it was fixed before go-live.
How it was verified
A deliberately awkward test set: a large global company with a non-decision-maker title (correct result: low tier despite company size), a personal email address, a high-intent lead with missing information (the hard-limit stress test), and two clear A- and B-tier cases. Each was verified in four places: the notification, the CRM record, the task created, and the audit log.
Then the failure paths: a real fault to verify alerting, and a two-shot test to verify duplicate protection.
Results
- A complete manual triage step became automatic: every lead researched, scored, routed, tasked and logged with no human involvement.
- Form submitted to team notified in under seven seconds, per live-run measurements.
- Duplicate notifications or tasks are impossible inside the protection window — by design and by test.
- Any material fault reaches a person, with a direct link to where the problem is.
What changes when your business changes
The two things most likely to change — the definition of an ideal customer, and which team member gets what — are each isolated to a single place. When the criteria or the team structure change, no other part of the system needs touching.
If your inbound looks like this
It was built for one agency, but the pattern applies to any business where leads arrive faster than the team can qualify them, and the good ones cool off in a shared inbox. The system is rebuilt around your CRM, your channels, and your definition of a good lead.
Implementation details — prompt design, scoring rubric, thresholds and guardrail logic — are not published. Technical walkthroughs are available under NDA on request.