Early accessFor teams with AI agents in production
Catch AI agent bugs
before your users do.
Nineproof turns real failures into tests that run on every change.
app.nineproof.com/larkspur/support-agent/runs/482
Runs / Pull request #482
Refund prompt update
Check failedfix/refund-copyRe-run
Reliability
1.6 nines
was 2.3 on main
Passed all 8 runs
81.7%
was 96.1% on main
Tests that got worse
3
of 123 tests
Test8 runsPassedvs main
cancel-confirms-plan6/8worse
cancel-annual-plan7/8worse
cancel-with-refund7/8worse
refund-requires-identity8/8same
partial-refund-two-items8/8same
Changed at step 4: the agent now cancels before the customer confirms.
AI agents don't behave the same way twice.
One passing test proves little. What matters is passing every time.
<25%of tasks a GPT-4o agent got right 8 times in a row (τ-bench)
How it works
- agent.runllm.calllookup_subllm.callcancel_subemail.send01
Connect your logs
Works with the tracing tool you use.
- lookup_subscriptionsavedcancel_subscriptiontest copyemail.sendmockedorders databasesnapshot02
Turn failures into tests
Replayed safely with saved responses.
- 37 of 40 runs passed1.6 nines03
Run each test many times
A lucky pass can't hide a bug.
- Nineproof reliabilityfailedUnit testspassedLintpassedMerging is blocked04
Block bad releases
The pull request check fails.
A look inside Nineproof.
app.nineproof.com/larkspur/support-agent/failures
Failures
Failures from production
Last 7 days34 conversations went wrong, grouped into 6 causes.
All signalsThumbs-downEscalatedTool error
Cancelled without confirming the plan
thumbs-downescalatedlast seen 2 days ago
Refunded the same order twice
tool errorlast seen today
Promised a discount that doesn't exist
thumbs-downlast seen 4 days ago
Asked for ID again after verifying
thumbs-downlast seen 5 days ago
Looped on the address lookup
tool errorescalatedlast seen yesterday
Replied in English to a Spanish customer
thumbs-downlast seen 6 days ago
Built for agents that use tools.
Safe replays
No real refunds. No real emails.
Production run
payments.refundlive
email.sendlive
orders.updatelive
Nineproof replay
payments.refundsaved
email.sendmocked
orders.updatetest copy
Realistic test customers
Starts from what real customers said.
Hi, I want to cancel my subscription.
You're on the Pro annual plan. Should I cancel it?
seeded from conversation 8812
Results you can trust
Every score comes with error bars.
refunds2.4
order-status3.1
cancellations1.6
See what changed
The first step that went differently.
3reply: plan details
4main: confirm_plan
4#482: cancel_subscription
Free to start. Pay per test run.
Free
$0
For your first agent.
- 1,000 test runs a month
- Pull request checks
- Unlimited team members
Team
Recommended$249/ month
Then pay per run.
- Spending cap
- Any tracing tool
- Model comparisons
Enterprise
Custom
Annual contract.
- SSO
- EU or US data
- Self-hosted