Tag: tensor-parallel

All the articles with the tag "tensor-parallel".

• Developing a custom Qwen2.5 72B inference engine from scratch — what is the real bottleneck?

  11 Jun, 2026

  Custom Qwen2.5-72B engine on 8×H200: the real bottleneck isn't the 72B matmuls — it's the 160 all-reduce barriers per token. H200 roofline math, barrier vs. bandwidth anatomy, and why the engine anti-scales at TP=8.