Take payment and book.
Checkout creates and confirms a payment intent, saves the order and books the reserved seat.
AGENTIC DEVELOPMENT / THE RECORDED TARTAN TICKETS STORY
One new feature.
The decisions you didn’t know to ask for.
You ask Claude for text confirmations. It implements the feature and tests a purchase. Filibuster exposes both a pre-existing payment problem and assumptions in the test itself. Some executions pass assertions but still need your approval. Others fail and lead to a test refinement or an application repair.
See the real Claude workspace ↓Add a text confirmation.
No one asks for payment retries or names an idempotency key. The request describes a customer feature. What should happen if payment completes without a reply, or a notification fails, is still unspecified.
THE ACTUAL REQUEST TO CLAUDE
Send customers a text confirmation when their ticket purchase is confirmed. Add an end-to-end test and check it with Filibuster.
CustomerOne purchase request
HTTP →API gateway
HTTP →Ticket serviceOrders + booked seats in SQL
→TartanPayHTTP payment APIMailTwo SMTP messagesSMS mockNew HTTP receiver
02 / CLAUDE IMPLEMENTS THE FEATUREEXISTING APPLICATION →EXISTING SIDE EFFECTS →NEW APPLICATION CODE
Send a text after a confirmed purchase.
Claude adds an HTTP SMS client to the ticket service and connects it to checkout. A local mock accepts the text so the test can inspect it. This is feature implementation, before fault exploration.
Send two emails.
The ticket service attempts the confirmation and receipt messages.
Send the text.
The mock HTTP service records accepted messages and their order links. It does not contact real phones.
Buy one ticket.
Check the whole result.
Claude authors tests/purchase-text-confirmation.e2e.cjs. The test prepares a reserved seat, makes exactly one checkout request, then checks the response and independent payment, order, booking and message observations.
Filibuster receives the test file and runs it. The test owns its requests and assertions. No application-specific runner supplies a client retry loop.
- Reserve seat A1 for a $10 ticket.
- Send one checkout request.
- Assert a successful purchase response.
- Check one linked charge, order and booked ticket.
- Check two accepted emails and one linked text.
AN ASSUMPTION CLAUDE ALSO PUT IN THE TEST
Under failure, a clean error was already allowed.
Claude wrote conditional assertions: a successful response must have the complete purchase and notifications; an error must leave no purchase behind. The original request did not approve that policy. A faulted execution can therefore pass those assertions and still require a decision from the user.
IN YOUR AGENT’S WORKSPACEYou describe the feature and decide the behavior.
Claude runs the loop.
You make the product calls.
Ask for the feature in conversation. Claude handles the setup, implementation, test and Filibuster run. When a behavior needs your decision, the question and execution evidence sit side by side.
CLAUDE + FILIBUSTER
Set up the supported tooling. Write and run the test. Explore failures. Explain the outcome and recommend choices.
YOU
Inspect the execution. Say what should happen. You do not need to write the fault scripts or diagnose the payment API.
CLAUDE + FILIBUSTER
Apply your decision to the app or test. Replay the failure and retained cases. Continue, then explain the refined requirements and remaining gaps.
ACTUAL RECORDED CLAUDE WORKSPACE
Left: Claude explains the outcome and asks what should happen.Right: You inspect Filibuster’s evidence beside the conversation.
See what Claude does after the answer
In this saved step, Claude records the choices, applies the payment-status recovery change and prepares the revised expectations for replay. The user has not typed an implementation recipe or a sequence of test commands.
Original qualification captures. This run required two operator controller fixes, disclosed in the final scope below; it is not evidence of frictionless setup for every app.
04 / SAME TEST, INTERNAL FAILURESTest passed
What should “correct” mean here?
Follow the cases in order: a passing assumption, an error hiding a completed purchase, leftover provider state, then a charged customer with no ticket. Read the trace and effects before Claude’s explanation and the user’s decision.
THE HEALTHY EXECUTION
The feature works when every dependency responds.
Claude’s test makes one purchase request. One charge, one order, the booked seat, two accepted emails and one accepted text.
Loading saved execution…
Click any call to inspect its result, injection phase and recorded fields.
Charge1 × $10
Order1
Booked ticket1
Emails accepted2
Texts accepted1
Read Claude’s explanation and the recorded decision
Claude’s saved explanation, excerpted from the actual run.
CLAUDE ASKS
Condensed display of recorded questions and options; the linked original preserves the full wording. Choices were accepted under the operator’s instruction to follow recommendations. These controls display the saved decision; they do not submit a new answer.
WHAT THIS ESTABLISHES
See what Claude implemented and tested →A passing test captures one execution.
It establishes the observed healthy result. It leaves open what the application should do when a dependency fails.
What the recorded sequence also covered
The cases above show the main decisions. The run also covered other payment setup faults, separate receipt-email failures and the text timeout. The full corpus is retained; this walkthrough does not display every execution or question. In three of the eight reviewed pre-charge failures, an unpaid provider record remained. “No charge” was not the same as “no side effects.”
The payment repair in this run is a GET of the existing payment intent’s status after an uncertain confirmation outcome. It resolves this counterexample without reissuing a charge. A separate customer retry is outside this test.
The agent changes code.
Or you clarify the specification.
- 01
Claude runs the test.
Filibuster records the healthy execution and explores dependency failures.
- 02
Claude shows the outcome.
Read the explanation beside the execution and its business effects.
- 03
You specify the behavior.
Answer the business question. Decide what is allowed or what should happen instead.
- 04
Claude changes and replays.
The agent updates the app or test, then rechecks the failure and retained cases.
The agent continues after your decision Until the declared scope passes
APPLICATION + TEST
Recover the completed payment.
Claude investigates the payment API after the charged-without-ticket counterexample. Under an uncertain confirmation response, checkout reads the same intent’s status. If it succeeded, booking continues. The test accounts for the new internal call and preserves the purchase guarantees.
The repaired witness and all 12 retained cases passed before exploration moved on.
Inspect the repaired execution ↗TEST / SPECIFICATION ONLY
Record the email exception.
The selected email may be absent under its reviewed fault. The purchase, other email and text remain required. Claude changes the conditional test assertions, not the email implementation.
Separate messages and materially different outcomes require separate policy decisions. One allowed omission is not permission to omit everything.
A passing repair still has design questions
What if the recovery lookup also fails?
The repair adds another dependency call. Losing the confirmation reply and failing that lookup requires two injected faults, outside this run’s bound. A passed replay does not settle that scenario.
What about the payment API’s idempotency key?
TartanPay already accepts idempotency_key when creating a payment intent. This run’s agent did not use it. A stable purchase key could recover the same intent across retries, but confirmation still needs status-aware handling. It would not by itself make booking and notifications idempotent.
What makes the text retry safe?
For this witness, the injected failure is before dispatch: the SMS receiver accepted nothing. A production 503 alone does not establish that. Tracking known successes also leaves the lost-acknowledgement case unresolved; provider deduplication or a status lookup can address that uncertainty.
06 / THE LOOP COMPLETES WITH A BOUNDARYThe original ask becomes
The original ask becomes
a more precise specification.
THE ORIGINAL ASK
“Send customers a text confirmation when their ticket purchase is confirmed.”
THE REFINED REQUIREMENTS
- Failure before any charge: allow an error with no purchase or notifications; the seat stays reserved. Unpaid provider records may remain in the reviewed setup failures.
- Checkout completed, reply lost: the purchase and required messages stand, even though the customer sees an error.
- Payment completed, reply lost: check the existing payment’s status and finish booking when it succeeded.
- Purchase completes: keep one charge, one linked order and the booked ticket.
- A selected email send fails: allow that missing email, while preserving the purchase and the other required messages.
- Text unavailable before acceptance: retry once internally and verify exactly one accepted text.
- The reviewed text timeout occurs: allow no text, while preserving the purchase and both emails.
21 / 21retained single-fault cases passed
0pending plans within the bound
2original smoke tests passed
One reserved-seat purchase test. One injected fault per execution. Message acceptance by local receivers, not delivery to a real phone or inbox. Two-fault combinations, separate customer retries and database failures remain outside this result.
Read Claude’s actual closing explanation
Selected wording from the saved Claude response:
“Checkout now asks TartanPay whether the charge went through and, if it did, completes the purchase normally. This is an app change.”
“From your decisions during testing: the charged-card recovery, the one-time text retry, and the test's rules for the reply-lost, one-email and text-timeout cases. The existing checks stayed in place throughout.”
What this run establishes
The final source passed its healthy execution, 21 retained single-fault cases, 2,266 assertion-control evaluations and both original smoke tests. The qualification run included nine recorded business answers, one of which was redundant, and two operator fixes to the controller. It was not an uninterrupted marketing take. The example demonstrates the loop; it does not establish frictionless setup for every application.
What remains open
Two simultaneous faults, a failure of the recovery lookup after a lost payment reply, separate customer retries, concurrent checkout, database failures and real message delivery were not established. A timed-out text is not queued for later delivery, and the ordinary Compose stack still needs an SMS provider configured. The final approvals are bound to the reviewed cases; this SDK still has limits in reusing those decisions as broader rules.
Try the next question with the same test.
If either email is allowed to fail alone, must at least one still arrive when both sends fail? That is a two-fault question, left for the next experiment.