Squeezing the Silicon - How Google’s Dual Hardware-Software Playbook is Breaking the AI Memory Wall

1:25 pm - 1:45 pm

As the industry scales deep learning to multi-modal AI models and dense Mixture-of-Experts (MoE) architectures, AI infrastructure is experiencing a fundamental regime shift: we have transitioned from being compute-bound to being strictly memory-constrained. Today, high-performance memory (HBM and advanced DRAM) represents the single dominant cost driver of AI server hardware, accounting for 75%+of the total system Bill of Materials (BOM). 

 

In this address, Nikhil Cherian, Senior Director of Supply Chain Infrastructure at Google Cloud, outlines Google’s frontline experience supporting large models and agentic workflows. He details the severe physical bottlenecks Google has observed across the memory hierarchy—spanning capacity caps, bandwidth throttling, latency spikes, and power efficiency trade-offs. 

Crucially, the presentation showcases how Google is deploying a dual hardware-soft playbook to break this “Memory Wall”, and delivers a call-to-action to the hardware ecosystem: hyperscalers are not passive consumers of memory constraints. Google invites memory ecosystem to move away from transactional purchasing models, and enter a new era of deep collaboration to power the next 1000x epoch of computing.

 

Key Technologies Covered 

  • High-Bandwidth Memory (HBM): Ultra-high throughput stacked memory scaled to 288 GB on TPU 8i and pooled up to 2 PB across a 9,600-chip superpod on TPU 8t to eliminate off-chip data shuffling.
  • On-Chip SRAM: Tripled to 384 MiB on TPU 8i to host active conversational states and Key-Value (KV) caches directly on silicon for zero off-chip latency.
  • Key-Value (KV) Cache: The "working memory" of LLM inference that acts as a token generation cheat sheet, growing linearly with context length during multi-turn agentic reasoning loops.
  • TurboQuant: Google's state-of-the-art, training-free, lossless quantization algorithm (ICLR 2026) compressing LLM working memory from 32 bits down to 3 bits, yielding a 6x footprint reduction and an 8x attention speedup with 0.0% accuracy loss.
  • Hardware Bifurcation (TPU 8i vs. TPU 8t): Purpose-built architectural specialization dividing low-latency inference (state-tracking SRAM/HBM) from megascale training (shared HBM pooling & TPU Direct Storage).
  • Mixture-of-Experts (MoE) Architectures: Sparse multi-modal model architectures (like Gemini) that require massive HBM footprints simply to house billions of static routing parameters.
  • TPU Direct Storage: High-throughput streaming data pipeline on TPU 8t feeding multi-modal datasets directly to accelerators without processor starvation.
  • Circular Component Recycling & Legacy DRAM Integration: Zero-waste infrastructure design that refurbishes active server components and integrates older-generation memory (such as DDR4) into newer servers via interface adaptations.

Featured Speakers

Nikhil Cherian

Mr. Nikhil Cherian

Senior Director, Supply Chain Infrastructure, Alphabet Inc.

Nikhil Cherian is the Senior Director of Supply Chain Infrastructure (SCI) at Google Cloud. He is accountable for leading the strategic architecture and design of the supply chain that accelerates Google’s global hardware operations. His organization’s mandate is to accelerate end-to-end supply chain capacity, drive the resilience of Google’s hyperscale AI infrastructure investments, and deploy high-value, AI-driven workflows across our global operations. Nikhil previously held key leadership roles at Walmart, where he drove deep digital and supply chain transformations across massive retail pipelines. Prior to his work at Walmart, he worked at a data science startup. Nikhil holds a B.S. in engineering and a M.S. in Information Systems. He and his wife, Shilpa, have been married for over a decade and reside in the San Francisco Bay Area with their two sons.