Together AI Compiler Achieves Up to 2.6x Inference Throughput on Frontier Models

Together AI released the Together Compiler on November 4, 2026. The system uses agent-driven continual compilation to optimize forward passes for open models and has produced up to 2.6x higher tokens per second than prior best published configurations.

Together AI Compiler Performance on NVIDIA Blackwell and Other Hardware

The compiler targets dynamic inference conditions where traffic patterns, model versions, and hardware change frequently. Persistent engines launch in roughly nine seconds instead of the 467-second baseline. Parallel agents propose changes through a shared message board, while a cost model selects tensor and expert parallelism strategies. Tests ran under 24 hours per setting and used no more than two nodes per model.

On the AgentX benchmark at batch size one, MiniMax M3.1-Flash reached 1,111 tokens per second on NVIDIA B200, an 1.8x improvement over the 633-token baseline. DeepSeek v4.1-Flash recorded 1,205 tokens per second on GB300 hardware. GLM 5.3 on B200 moved from 329 to 417 tokens per second, and Kimi K3 on GB300 rose from 423 to 450 tokens per second. Gains on AMD MI355X were smaller but still positive.

Against open megakernels such as ThunderKittens and Mirage, the compiler produced up to 1.7x higher throughput on a Llama 1B model at batch size one, exceeding 2,100 tokens per second. Non-LLM workloads in weather, PDE solving, time-series forecasting, and biology saw speedups between 1.5x and 6.4x compared with prior agentic baselines.

Production Deployment and Agent-Driven Compilation Mechanics

The system already processes trillions of tokens daily on Together’s serverless endpoints for GLM 5.3, GLM Flash, and DeepSeek V4.1 Flash. A limited beta is available for dedicated endpoints, with adaptation to traffic shifts occurring within hours. The compiler remains closed source.

Two engine tiers operate simultaneously: a fast low-fidelity layer for rapid iteration and a slower high-fidelity layer that validates changes on real traffic. SemiAnalysis independently confirmed at least one result from the InferenceX and AgentX suites. The work extends Together’s earlier megakernel research and forms part of its Polymorphic Inference program.

Operators running open models on NVIDIA B200, B300, GB300, H100, AMD MI355X, or AWS Trainium2 can test the beta to reduce per-token latency without additional hardware. The approach suits production environments where prompt-length distributions shift after model updates or across regions.

Next Steps for Together AI Compiler Users

Together plans to expand the limited beta and apply the compiler to additional models and workloads. Teams evaluating inference stacks should compare current vLLM or SGLang baselines against the new numbers on their specific traffic before committing capacity. Early adopters in the beta will receive priority access to future optimizations.