IBM Research released ALTK-Evolve, a method that improves a customer service agent by writing it a playbook rather than retraining the model behind it.
Using GPT-4.1, an older non-reasoning model, a basic agent reaches 67.5% Pass^4 on the retail portion of tau-squared-bench. The metric means a system solves the task across all four independent attempts for 67.5% of cases, above every retail entry in the October 12 leaderboard snapshot.
The agent runs on the benchmark’s stock ReAct scaffold. The only addition is a set of learned guidelines appended to its system prompt.
Learning without gradients
The method builds those guidelines by reviewing the agent’s attempts on training tasks and distilling patterns into short behavioural rules: how to authenticate a customer, confirm a transaction, handle a return.
That is the interesting claim. Most agent improvement work assumes you need better weights or a larger model, and IBM’s result suggests a significant chunk of the gap comes from procedure that can be written down.
The approach has obvious limits. Rules distilled from one distribution tend to encode that distribution’s quirks, and a playbook tuned on retail scripts may not survive real customer behaviour, which is adversarial and frequently incomplete.
The leaderboard position is also a snapshot, and benchmarks of this kind move quickly as entries are resubmitted.
For teams already paying for inference, though, the economics are attractive. A prompt-length playbook costs a fraction of a frontier model call and can be inspected by a human before it ships.
