Google open-sources a serving engine that drops Python from the request path

XSE compiles models to native code for C++ serving; Google reports 2-5x throughput and a quarter of the memory on short requests, in its own tests.

Google has open sourced XSE, short for XLA Serving Engine, as part of the OpenXLA Project, an industry-backed effort that provides tools for compiling machine learning computations into native code for different processors. Nathan Beach, a Google director of product management, described it in a LinkedIn post on Tuesday, 10 November 2026.

The workflow keeps Python where developers want it and removes it where it costs performance. Models are written in Python with Google's JAX. An export tool packages the computations in StableHLO, along with trained weights, into an XLA SavedModel. A C++ serving runtime then uses XLA to compile that into native code for the target hardware, so no Python interpreter sits in the serving path. Inside Google, XSE already serves more than six million queries per second across YouTube Shorts, Google Ads and Search.

In Google's tests on short requests, XSE delivered two to five times the throughput of its best optimized Python server while using a quarter of the memory. Serving the same traffic on fewer machines is a direct cut to the cost of running AI, and publishing the engine as open source lets others capture that saving rather than only Google's own products. Cheaper inference is one of the quiet levers that decides how widely AI-driven knowledge tools can be offered.

The caveats matter. Those figures are Google's own, not independent benchmarks, and they apply to short requests. XSE targets smaller, task-focused models such as ranking, recommendation, embedding and vision, not large-scale LLM inference, where vLLM or SGLang are better suited. Google favors its own TPUs, though XLA SavedModels can run on GPUs and CPUs, and it has partnered with Arm and Intel. Planned TorchTPU integration would let PyTorch users export for C++ serving on TPUs. Existing alternatives such as LibTorch and ONNX Runtime also avoid Python at serving time, so independent comparisons will show how much XSE adds.

Source: The Register