Inference is forcing a rethink of ai hardware

An IEEE Spectrum analysis published on 15 September argues that the boom in inference, running models for users rather than training them, is forcing a rethink of ai hardware.

The numbers it cites

• Nvidia acquired key talent and IP from inference startup Groq in a deal reported at about $20 billion. The exact structure of the deal hasn't been confirmed.
• Anthropic is reportedly paying xAI more than $1 billion a month to lease spare compute.
• Researchers found Nvidia H100s running open-source LLMs sit idle 50 to 80% of the time, waiting for data to arrive from memory.

Why it matters

The ai hardware story is still mostly told in terms of training and ever more GPUs. For serving models at scale, the bottleneck is memory, which is why optimisation work such as quantisation now targets memory rather than raw compute. Buying a startup's talent and IP instead of the whole company is also becoming a pattern of its own.