Zyphra Inference Wave 2: Faster MoE and Long-Context Serving on MI355X
Zyphra presents the second wave of inference optimizations in the Zyphra Optimization Engine (ZOE) for AMD Instinct MI355X GPUs. We measure the empirical compute and memory-bandwidth limits of the MI355X, write custom HIP kernels for mixture-of-experts layers and sparse attention in our production forks of vLLM and SGLang, and evaluate MXFP4 quantization and speculative decoding. ZOE speeds up GLM-5.3 time to first token at 512K tokens of context by 2.8x with no prefix-cache hits and by 3.6x at a 90% hit rate, and it raises DeepSeek-V4-Pro single-user decode throughput at 1M tokens of context by 1.40x.
Read the post
Expert Coupling in MoE Pretraining
Mixture-of-Experts (MoE) models scale model capacity efficiently by routing each token to only a small subset of experts. But at large scale, the cost of moving tokens between GPUs can become a major bottleneck, accounting for as much as 60.4% of total training step time in our experiments. In this work, we show that MoE routing contains substantial structure that can be exploited to reduce this communication. Experts selected by the same token are strongly correlated within a layer, while a token’s expert choices in one layer predict where it will be routed in later layers. We use these two properties to increase token–expert locality in two complementary ways: correlated expert placement co-locates experts that are frequently selected together, while token shuffling proactively moves tokens to the GPUs likely to contain their next experts. Neither technique changes the model architecture or its routing decisions. Moreover, we find that these routing patterns are cheap to estimate and emerge early enough in training to remain useful over long stretches of pretraining. In Megatron-LM experiments on MI300X GPUs, our methods reduce all-to-all communication time by up to 2.63× and improve end-to-end training step time by up to 1.41×.
How Hybrid NoPE Models Learn Position Without Position Encodings
Zyphra Research investigates how hybrid language models can recover positional information even when their global attention layers use no explicit position encoding (NoPE). We find that local layers such as sliding window attention (SWA) and gated linear attention naturally create a recency bias in the model’s residual stream, meaning that representations of nearby tokens become more similar than representations of distant tokens. We find that this recency bias then propagates through the model, where the query and key projections of global NoPE attention learn to read it out. The result is an implicit relative position encoding in attention layers that, on their own, have no direct access to token position. Our results provide a mechanistic explanation for why architectures interleaving local mixing with global NoPE attention can work so well, and point toward a promising way of building positional representations that extrapolate naturally beyond the sequence lengths seen during training.
PUFFER: Incremental Fuzzy Deduplication for Continuously Evolving Corpora
Zyphra introduces PUFFER, a provenance-aware incremental fuzzy-deduplication system for continuously growing training corpora. PUFFER maintains a faithful, disk-resident MinHash locality-sensitive hashing history that can ingest new datasets, recover deterministically from interrupted jobs, and remove individual datasets without repeatedly rebuilding the full corpus index. PUFFER achieves an 11x-35x speedup over existing CPU-based deduplication methods when tasked with processing up to one billion documents over a ten-hour throughput test. At the deduplication (index) layer, PUFFER ingests one billion documents in 1.75 hours on a single process, and is deployed in production on more than 30 billion documents. Available as open-source software under a permissive Apache 2.0 license.