Search and answer products pay a quiet tax on every query: text has to become vectors before retrieval can start. Perplexity, which runs embedding and ranking models across its search engine, its agent computer, and its API platform, just published the serving playbook behind that step in a post from its engineering team.
The unorthodox call at the center is reuse. Embedding models are small enough that their workloads resemble familiar phases of LLM generation: rebuilding or refreshing an index behaves like prefill, the compute-heavy stage, while embedding a short live query behaves like decode, which is memory-bound. Rather than stand up a dedicated embedding engine, Perplexity runs both kinds of work on the kernels its main LLM stack already uses.
Scheduling leans on a component called Tulip that admits sequences in arrival order. The team argues that simple policy is correct at the sizes it serves, since the linear cost of dense layers, not attention’s quadratic term, dominates for short embedding sequences, making latency scale with tokens rather than the number of concurrent requests. Saturation arrives around 512 tokens on a sub-billion-parameter model, after which extra sequences stop improving efficiency. Small batches also risk spending more time launching kernels than executing them, so the stack captures whole-model CUDA graphs, folding every launch into a single driver call. Perplexity says it pushed changes upstream to FlashInfer so those graphs can be captured, and graphs are rebuilt per configuration bucket.
A Rust request path and an asynchronous result tracker round out the architecture. The write-up extends a string of infrastructure deep dives from Perplexity covering Blackwell hosting and tokenizer tuning, published as AI search and agent products everywhere compete to shrink the cost of retrieval at scale.
