Token per Watt

Definition

Token per watt is an AI-infrastructure efficiency metric that relates delivered model-token output to energy input, shifting attention from peak chip specifications toward useful serving work under power constraints.

Current Synthesis

Across the sources, token per watt is a system metric rather than a chip-only benchmark. Memory movement, interconnect, cooling, utilization, software, workload mix, and rack deployment all affect how much usable inference a given power envelope can deliver. It is especially useful for comparing large systems whose aggregate compute rises by adding accelerators, but it remains incomplete unless token quality, latency, task success, cost, and total facility energy are specified.

Key Claims

• Token per watt shifts evaluation from raw FLOPS or chip counts toward delivered AI work under energy constraints.
• Memory movement, communication, cooling, utilization, and software can dominate effective system efficiency.
• Larger supernodes can increase aggregate compute while worsening power efficiency.
• Inference- and agent-oriented platforms make continuous output efficiency more economically important.
• Better efficiency can expand total demand, so lower energy per token does not guarantee lower aggregate energy use.

Evidence

Nvidia infrastructure framing

• E230｜1万亿收入预期背后：英伟达的巅峰与软肋 uses token per watt to interpret Nvidia’s shift from isolated GPU performance toward full-stack token production.
• EP 36: NVIDIA GTC 2026: Everything That Matters - Recapped presents useful intelligence output per watt as a planning metric for Vera Rubin and the broader inference economy.

Supernode comparison

• 国产 AI 算力能凭「超节点」弯道超车吗？｜WAIC 深度观察 S10E23 uses higher-power domestic supernodes to show why aggregate compute alone cannot establish system efficiency or competitive catch-up.

Counterevidence & Qualifications

• Tokens are not uniform units of value across models, modalities, context lengths, quality levels, or tasks.
• Vendor claims may omit facility overhead, utilization, networking, cooling, or workload assumptions.
• A system can improve tokens per watt while increasing total electricity consumption through greater usage.
• Energy efficiency does not resolve supply, latency, reliability, capital cost, or data-center power availability.

What Changed

• Migrated the concept to the synthesis-v1 schema.
• Added the GTC recap’s inference-economy and enterprise-planning interpretation.
• Clarified that output quality and total facility boundaries are necessary for meaningful comparison.

Related Concepts

• AI Inference Cost Structure - economic structure that energy efficiency helps determine.
• Inference as Cash Flow - recurring-demand thesis that makes serving efficiency important.
• Data Center Power Bottleneck - physical limit motivating output-per-energy metrics.
• Data Center Thermal Management - facility overhead affecting realized efficiency.
• AI Infrastructure Full-Stack Moat - system integration layer behind delivered rather than theoretical output.
• Jevons Paradox In AI - demand response that can offset per-token efficiency gains.
• AI Accelerator Supernode - system scale where raw compute and energy efficiency can diverge.