AI-generated editorial illustration; not a documentary photograph.
Cloudflare says changes to the consistent-hashing system in its Pingora Backend Router reduced memory consumption across its network by 100TB. The optimization combines a more compact Rust data structure with a 90% reduction in the number of hashes generated for each server.
Why the router used so much memory
Pingora Backend Router directs cacheable requests to servers by URL, helping Cloudflare keep one copy of a file per data center and locate it consistently. Its pingora-ketama library creates multiple hash points for every server to balance workloads, with weights allowing machines with more storage to receive a larger share of requests.
Memory requirements grew further because compliance constraints and enabled caching features can limit which servers handle a request. Cloudflare therefore maintained separate hash rings for potentially relevant combinations of server capabilities.
Smaller records and fewer hashes
Cloudflare first reduced each stored hash point from eight bytes to six. Simply changing its index from a 32-bit to a 16-bit integer did not save space because of Rust's alignment rules, so the team stored the hash and index in a six-byte array and exposed them through getters. Cloudflare calculates that this cut consistent-hashing memory use by 25%.
The larger saving came from analyzing how hash count affects load-distribution accuracy. Cloudflare found that very high counts also raise the likelihood of collisions in the 32-bit hash space. Based on its calculations and simulations, the company determined that it could generate 90% fewer hashes per server without appreciable error.
A staged cache migration
Changing a hash ring can redirect cached requests, so an immediate network-wide switch risked widespread cache misses and increased traffic to origin servers. During migration, the router held both the old and new rings in memory and selected between them consistently for each request hash, preserving a rollback path.
Cloudflare began with small validation locations before expanding through larger data-center groups. It controlled both the proportion of traffic using the new ring and the locations where movement was permitted, while monitoring routing traces, connection errors, memory, startup time, cache behavior and origin traffic.
Available in pingora-ketama
The changes are available through a currently unadvertised cargo feature in the pingora-ketama crate. Its v2 ring includes the compact storage format, a faster sorting method and configurable scaling for the base number of hashes per node. The original v1 behavior remains available, and applications can operate both versions simultaneously for request-level migration.
Source: Saving another 100TB of RAM with math (and Rust)
SIGNAL TITANS
Leave a Reply