Case file / Frontier model evaluation

I pushed 13 frontier models until they broke.

Two years, four labs, one job: break pre-release models before customers do. OpenAI, Google, Meta and Scale AI all ran my evaluations. What I learned there saves my clients six figures per deployment.

Break my workflow, please What I sell

Models evaluated13Pre-release frontier models, four labs

Workflows tested100+Real business workflows, zero synthetic benchmarks

Clean vs real gap94→58%Same model, pristine tickets vs real customer tickets

Tool call loss @2K tokens40%Reliability penalty from verbose prompts alone

Every failed pilot tells the same story

Someone walks me through their failed AI pilot every single week. They picked a model off a leaderboard, wired it into production, and watched it fall apart on day one. Their conclusion: "AI doesn't work for us."

Here's what actually happened. The model scored 95% on the benchmark. Then it met their customers: typos, slang, half sentences, ALL CAPS rage. Same model, real tickets, 60% accuracy. So everyone blames the model. The gap sat in the evaluation from day one, and nobody ran the test that would have caught it.

Finding that gap is my job. I've done it for the biggest labs in the world, and I can run the same gauntlet against your stack.

The number that should worry you

One controlled evaluation. Two versions of the same support tickets.

Accuracy on clean tickets95%

→

Accuracy on real customer tickets58%

I ran one frontier model against two versions of the same support-ticket workflow. Version one used pristine tickets: correct grammar, complete information. It scored 95%. Version two used the tickets exactly as customers typed them: typos, slang, ALL CAPS rage, half sentences. It scored 58%.

Nobody touched the model. The data alone cut its accuracy by 37 points. And version two is the version that pays your bills.

-40%Verbose prompts read like expertise. Past 2,000 tokens of context they cost you 40% of tool-calling reliability. Teams ship worse agents because longer prompts feel more serious.

85% @ 10%gpt-4o-mini delivered 85% of gpt-4o quality at 10% of the cost on structured extraction. I run the mini and pocket the difference.

Which one of these is burning your budget right now?

Paste your workflow into the free diagnostic. Written analysis of where your model or pipeline fails, within 48 hours.

Run the free diagnostic

The three patterns that kill AI projects

Every failed deployment I inspected showed at least one. Fixing them at design time costs a workshop. Fixing them after launch costs a rewrite.

Clean data lies

Vendors test on pristine inputs. Customers type with their elbows. Run the gauntlet on your ugliest data and watch what survives. That number decides your deployment.

// test on ugliest data first

The 80/20 failure trap

Roughly 20% of edge case types cause 80% of failures: ambiguous intent, temporal reasoning, multi-step logic. Design the workflow around them or constrain the model's scope.

// map edge cases before launch

Context sets the ceiling

The same task on the same model swings 3x in quality based on context engineering alone. Prompt architecture is where quality lives.

// engineer context, then scale

How I test

Layer 01
Benchmarks

Does it know things?

Standardized sets (MMLU, HumanEval), measured overall and per category. Every vendor hands you these numbers for free. Treat them as the warm-up act.

> output: leaderboard_rank

Layer 02
Adversarial

Where does it break?

Noise injection: typos, slang, incomplete inputs. Edge cases: ambiguous intent, cultural idioms, temporal reasoning. The accuracy delta between clean and messy data is the number vendors never publish.

> output: robustness_score, 0–1

Layer 03
Deployment

Can we ship it?

Your constraints at production volume: API latency, rate limits, JSON schema adherence for tool-calling, cost per 1K tokens, zero-data-retention compliance.

> output: deployability_verdict: red | yellow | green

The pipeline, end to end

01 Dataset→ 02 Clean run→ 03 Messy run→ 04 Adversarial→ 05 Cost & latency→ 06 Verdict

Models through this gauntlet: GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.1 405B, Mistral Large, Command R+ and more, across CRM automation, document processing, code generation, data extraction and multi-step tool-calling chains.

Before / after working with me

Before		After
Manual prompt-and-pray testing		100+ automated workflow benchmarks
5 to 10 workflows per model, cherry-picked		Every production-relevant scenario covered
Gut-feel scoring with no cost data		Pass/fail verdicts with cost and latency attached
Failure modes discovered by your customers		Failure modes mapped before deployment
Deployment on hope		Model choice backed by your own data

Where does your AI break? You'll know within 48 hours.

Paste the workflow, the model, the mess. Free, async, written analysis. If I cannot find the failure point, I'll say so and you owe nothing.