An inference engine tuned once goes stale. Together AI built a system on that premise and pointed coding agents at it.
Together Compiler takes a model, a traffic profile and a hardware configuration. Agents propose faster forward passes, and the engine measures each candidate on live distributions before shipping the winner into production. When traffic drifts, the loop restarts.
The numbers are self-reported: up to 2.6x more tokens per second than the best published InferenceX configurations, each setting optimized in under 24 hours.
Why a tuned kernel rots
Prompt lengths in one country averaged 97,000 tokens against 70,000 elsewhere, and one customer’s P90 input grew from 18,000 to 33,000 in three days. Random benchmarks mislead too. Synthetic inputs woke 13% to 38% more mixture-of-experts experts than real traffic, and one agent-written kernel took 20 microseconds on random data against 62 milliseconds on the real thing.
The restart problem and its fix
Every experiment used to restart the server stack, and MiniMax M3.1 Flash took 761 seconds to cold start on TensorRT-LLM. Together Compiler starts the engine once, then swaps a forward function pointer over RPC and re-captures CUDA graphs.
Two tiers keep the loop cheap. A fast pass serves 12 layers to screen candidates, a slow pass serves 60 to confirm them. Testing 1,354 candidates instead of 168 found a solution 7.2% faster.
Best case: 1,111 tokens per second per user on MiniMax M3.1 Flash at batch size 1, against 633 for the previous best. On GLM 5.3 with an AMD MI355X the gain was a bare 4%.
What expertise actually cost
An expert-curated knowledge base lowered speedup by 25% to 31% after twelve hours. Agents simply told to make the forward pass fast scored 6.2x against 4.9x for expert strategies. Together now hands them raw engine source code.
