On February 2, 2006, software developer Poul-Henning Kamp met Anders Berg of the Norwegian newspaper VG Nett in Oslo. Kamp’s preserved notes describe hours of complaints about Squid, the newspaper’s caching proxy software. Varnish, the alternative Kamp helped design, would leave decisions about which cached web data stayed in physical memory to the operating system.
Kamp, an experienced FreeBSD kernel developer, recognized what he later called “1975 programming.” In Kamp’s account, Squid treated random-access memory (RAM) and hard disks as two distinct storage tiers managed by the application itself, outside the operating system kernel. It tracked which cached Hypertext Transfer Protocol (HTTP) responses were in memory and which were on disk, moving them according to traffic patterns. But in modern systems, the operating system kernel and hardware paging units already treat RAM as a dynamic cache for disk storage. When Squid wrote inactive objects from RAM to disk, the operating system kernel was often simultaneously paging unused memory to swap space. The proxy and the kernel were fighting each other, creating redundant disk writes and stalling worker threads.
Letting the Kernel Handle Page Residency
Kamp’s design for Varnish inverted that model. Conceptual discussions had started in the autumn of 2005 between Kamp and FreeBSD committer Dag-Erling Smørgrav on a blackboard at the University of Basel during the EuroBSDcon conference. On February 22, 2006, Smørgrav created the Subversion repository that formally established the open-source project. Instead of maintaining a two-tier system to shuffle objects between RAM and disk, Varnish allocated a large block of virtual address space and mapped it directly to a disk file using the operating system’s mmap system call.
Under this architecture, Varnish did not track whether a cached page resided in physical RAM or on disk. When a worker thread needed to serve a cached response to an HTTP client, it pointed directly to the corresponding virtual memory address. If the requested data was already in physical memory, the hardware served it immediately. If the page had been pushed out by the kernel to free up RAM, the central processing unit (CPU) raised a page fault, prompting the operating system to retrieve the page from the backing file. By relying on kernel page management, Varnish avoided application-level accounting for RAM-versus-disk placement and the redundant disk copies described in Kamp’s Squid example.
Eliminating Contention on Multi-Core Hardware
Delegating memory management to the kernel was only half of the architectural overhaul. Kamp observed that multi-core processors were encountering severe memory bus contention when multiple CPU cores attempted to read and modify shared statistics counters or global memory pools. If two worker threads on different cores updated adjacent variables on the same cache line, the CPU hardware had to coordinate cache coherence across buses, forcing execution units to stall.
To minimize that overhead, Varnish adopted thread workspaces and short-lived allocations. Rather than invoking malloc, the memory-allocation function, for every HTTP header, Varnish called it once for an entire workspace and allocated space for headers within that buffer. Strings from client headers were referenced by pointers rather than copied into new memory buffers. When a request finished, the entire workspace was freed in a single pointer reset. Worker threads were managed in a most-recently-busy queue: idle threads returned to the front of the queue, making it more likely that the next request would reuse a thread whose stack and variables were still in the processor’s caches. In architectural notes, Kamp reported that Varnish required fewer than 18 system calls to serve a cached HTTP hit, with many of those calls dedicated to retrieving timestamps for statistics.
Compiling Configuration to Machine Code
Another part of Varnish’s early design addressed how configuration rules would run while handling requests. Between February 24 and February 26, 2006, Kamp designed the Varnish Configuration Language (VCL).
Instead of running an interpreter in the request-handling path, Varnish treated VCL as source code for a domain-specific compiler. The management process parsed the VCL file, translated its routing rules and header manipulations into standard C code, compiled that C code into a shared object library, and loaded the resulting binary into the running Varnish process using dlopen. This mechanism allowed administrators to modify caching rules and reload configurations on the fly without dropping active client connections or paying runtime interpretation penalties.
The July 2006 Live Test and Subsequent Trade-Offs
The first real-world verification occurred on July 6, 2006, when the development team redirected live traffic from VG Nett through an experimental Varnish instance. The test log records a brief spike above 800 requests per second. Subsequent entries report 60 megabits per second of traffic and 90 percent idle CPU. These were project-reported observations during a live trial, rather than a controlled benchmark. The trial also exposed parsing bugs and connection-handling defects; it did not establish that one Varnish server could replace the newspaper’s entire proxy cluster. Varnish Cache 1.0 was officially released in September 2006 under the two-clause BSD license.
Over the following two decades, hardware realities evolved, and the trade-offs of the original virtual memory design became clearer. While mapping disk files through mmap removed application-level RAM-to-disk staging, synchronous page faults could stall worker threads when serving data absent from physical memory. Modern Varnish documentation lists malloc as the default storage backend and file-backed mmap storage as another option. The project documentation notes that in modern deployments, heap fragmentation with the malloc backend typically increases memory footprint by two to four times.
The enduring insight of the 2006 architecture was its refusal to duplicate kernel responsibilities. By recognizing that operating systems had spent decades refining paging, memory hierarchy, and hardware integration, Varnish showed that performance can come from removing userland abstractions rather than building more complex ones.
Related stories
Sources
- The First Design of Varnish, Poul-Henning Kamp
- Notes from the Architect, Poul-Henning Kamp
- Varnish – 10 Going on 50, Poul-Henning Kamp
- Vinyl Cache: Introduction, License and Origin
- Storage Backends, The Varnish Users Guide
- Featured image: Karora; Wikimedia Commons; Public domain (adapted from the original)
SIGNAL TITANS
Leave a Reply