Tool calling at two million runs a month

What changes at two million runs

At a few thousand runs a month, tool calls feel like function calls. You send a request, you get an answer and the rare failure shows up in a log you read on Friday. At two million runs a month, roughly 14 million tool calls, the rare failure happens every few minutes. Every connector we own has had to grow up.

This post covers the three habits that made the biggest difference: retries with limits, idempotency keys on every write, and three separate timeouts instead of one.

Retries, with a budget

Retrying is easy. Retrying well is mostly about knowing when to stop. Our first version retried any failed call up to five times with exponential backoff. During a Cloudly incident last spring, that policy turned a ten-minute outage into forty minutes, because thousands of agents kept hammering a service that was trying to recover.

Now every connector has a retry budget rather than a retry count.

• Only retry errors the provider marks as transient: timeouts, 429s and 5xx responses.
• Add full jitter to every backoff, so retries from different agents spread out.
• Stop retrying a provider entirely once more than 20% of calls in a one-minute window are retries. The agent sees a clear “provider unavailable” result and can re-plan.

That last rule matters most. An agent that knows a tool is down can choose another path or ask a person. An agent stuck in retries just burns time.

Idempotency keys on every write

Retries are only safe if repeating a call cannot do the same thing twice. For reads that is free. For writes, it takes discipline.

If a tool can move money or send a message, assume it will be called twice. Design so that the second call is harmless.

Every write that leaves Kairon carries an idempotency key built from the run ID, the step number and a hash of the arguments. If the agent re-plans and issues the same Payly refund again, the key matches and Payly returns the original result. Where a provider does not support keys, as with some older Relay endpoints, we keep our own ledger and check it before sending.

Since we made this mandatory, duplicate writes caused by retries have dropped from about 40 a month to zero for eleven months running.

Three timeouts, not one

A single timeout hides too much. We now set three on every connector.

1. Connect timeout, usually 2 seconds. If we cannot open a connection, something is wrong with the network or the provider.
2. Response timeout, set per endpoint from its p99 latency plus a margin. A Bookwise availability search gets 8 seconds; a balance lookup gets 3.
3. Step timeout, covering retries and backoff together, usually 30 seconds. This is the number the agent cares about. Past it, the step fails and the agent decides what to do next.

Measuring it

Each connector reports its retry rate, timeout rate and duplicate-key hits to a shared dashboard. We review the worst five every Monday. Most weeks the fix is a timeout that drifted as a provider got slower.

None of this is new to anyone who has run distributed systems. What is new is that the caller is a model that can change its mind. Clear, fast failures give it something to reason about. For more on how agents use that, see planning is a loop, not a step.