AI-generated editorial illustration; not a documentary photograph.
Infrastructure providers are shifting their AI strategies toward inference as enterprise deployments drive higher token usage, memory demand and operating costs. At the AI Infra Summit in Santa Clara, AWS, Qualcomm, d-Matrix, Oracle and Broadcom outlined processor, memory and networking approaches intended to handle that load more efficiently.
Inference reshapes compute
AWS expects inference to become the majority of AI compute demand, creating a larger role for CPUs alongside GPUs. The company is positioning its 64-bit Arm-based Graviton processors as an energy-efficient option for cloud applications, including real-time reasoning and multistep task orchestration.
For enterprises, the change means infrastructure choices increasingly depend on the cost and energy required to serve models, rather than training performance alone. Qualcomm identified tokens per watt as a key efficiency measure as usage expands.
Memory becomes a constraint
SiliconANGLE's analysts estimated that a typical AI server uses about eight times as much memory as a traditional server. They projected AI server memory spending to rise from $35 billion in 2025 to between $175 billion and $190 billion in 2027.
Qualcomm is developing its proprietary high-bandwidth compute architecture as an alternative to high-bandwidth memory. The company claims HBC delivers six times the bandwidth per watt of HBM at large batch sizes and 200 times the capacity per watt of SRAM, but the report provides no independent benchmark results for those comparisons.
d-Matrix joins Nvidia architecture
Startup d-Matrix is taking a different route with Raptor, a chip architecture integrating higher-throughput 3D DRAM. SiliconANGLE reported that the company had unveiled a collaboration with Nvidia to incorporate Raptor into the NVLink Fusion rack reference architecture for low-latency token services aimed at AI labs, hyperscalers and neocloud providers.
Networks carry more weight
Oracle and Broadcom are also treating networking as a core AI scaling constraint. Oracle's Acceleron RoCE is designed to improve performance by bypassing server CPUs when routing data for hosted GPUs. Broadcom's Tomahawk 6 uses Ethernet and Cognitive Routing 2.0 features to detect congestion and redirect traffic across data-center connections.
The infrastructure work addresses a concrete enterprise pressure: token bills are rising as organizations experiment with deployed AI. Carrier Global said its token spending had increased eightfold during the year, while emphasizing task-level value over token cost.
Source: Full speed ahead: Despite calls to slow AI down, its support structure is on the fast track
SIGNAL TITANS
Leave a Reply