From Memory Supplier to Architecture Partner: Co-Design Opportunities in AI Inference

Heterogeneous LLM inference platforms are moving from lab experiments to mainstream commercial deployment. The goal is to improve token generation performance while reducing the cost floor for token generation. Success will require prompt processing, decode, memory-centric accelerators, interconnect, and orchestration software to operate as a coordinated system rather than as independent components. For very large models, efficiency gains in the range of 5 to 10x are possible, but only when the memory subsystem is considered early in the architecture. This creates an important opportunity for memory suppliers to participate earlier in system definition, where the highest-value decisions are made.

This talk starts with the inference operations that make memory architecture central to system performance. Attention mechanisms, KV-cache growth, prefix reuse, batching, Mixture-of-Experts routing, and disaggregated operations such as speculative decoding and prefill/decode separation each place different demands on locality, latency, interconnect bandwidth, and data movement. Understanding how these operations use and move state is the starting point for useful memory co-design.

The talk then examines the architectural consequences. Attention and sharding methods determine how context state is replicated, cached, partitioned, or moved across processors and memory tiers. At long context lengths, this becomes a first-class memory problem: active prompt, prefix cache, distributed attention, and persistent context state must be managed differently. Looking ahead to systems with up to 64M tokens of available context, the key issue is not simply storing more tokens, but deciding which context is active, which is cached, which is compressed, and which is kept in persistent memory. 

Using d-Matrix Corsair and Raptor as examples, the talk connects these architectural choices to practical memory technologies: SRAM-close compute for hot, latency-sensitive work, HBM for high-bandwidth model, attention, and KV-
cache traffic, and 3D-DRAM for larger near-compute context state. The takeaway is that memory innovation will increasingly determine which inference platforms can deliver lower cost per token, higher utilization, and scalable long-
context capability. For memory suppliers, this is an opportunity to move upstream from component delivery into the system-level decisions that shape the next generation of AI infrastructure.

Key Technologies Covered

• Heterogeneous AI inference systems.
• Memory-bound LLM inference economics.
• Attention mechanisms and KV-cache behavior.
• Sharding and distributed inference.
• Disaggregated prefill, decode, MoE, and speculation.
• Inference-aware memory systems and data movement.
• SRAM, HBM, CXL, 3D-DRAM, and PIM trends.
• d-Matrix Corsair and Raptor examples.
• System-level memory-compute co-design.
• Memory suppliers as architecture partners.