NixOS Stories

One Billion Objects and No Database: Inside cache.nixos.org

Rok Garbas (garbas), Steve Swoyer (reyows.evets) — Sun, 20 Sep 2026 12:00:00 GMT

Amazon Web Services (AWS) has renewed its sponsorship for cache.nixos.org, the Nix public binary cache, enabling the operation of this foundational piece of infrastructure for another year.

AWS's support started in 2023, when its Open Source Strategy and Marketing team, along with its Open Data team, stepped in when a previous sponsorship was discontinued. At this time, the growth of the Nix ecosystem was reaching an inflection point, outpacing the capacity to sustain it without additional support. The cache is depended on by Nix users worldwide, and its maintenance is therefore a high priority for the entire community. At the time, Ron Efroni, then a NixOS Foundation board member and now its president, in concert with Dave Nalley, director of developer experience at AWS, led a conversation that, in the span of around two weeks, resolved the concerns about the cache's sustainability. AWS agreed to cover the cache's hosting costs for 12 months — a substantial contribution which was renewed in 2024, 2025, and again for 2026.

Not just any package repository

Users most often encounter the cache as the authoritative package repository for both NixOS and the Nix package manager. Without a cache, every Nix client would have to build every package from source, which can take hours or even days and needs hardware many users do not have. Most Nix workflows would grind to a halt.

In serving prebuilt packages, the cache resembles the repositories maintained by Debian, Fedora, and Homebrew. For downloading and running software, the Nix archive (NAR) fetched from cache.nixos.org is similar to a .deb, .rpm, or Homebrew bottle. But underneath, the Nix cache differs fundamentally in its implementation, reflecting Nix's radically different approach to software packaging.

Traditionally, installing software on Linux meant extracting the necessary files from a package archive and distributing them to conventional locations around the filesystem. While simple, this approach has led to a number of inconveniences. For example, when packages depend on conflicting versions of the same library, they cannot be installed at the same time, since the two versions' files have to be placed at the same paths. This is a problem especially for developers, who routinely need to switch between different versions of their toolchains.

Nix solves this problem by doing away with conventional locations: each package or dependency is placed in its own dedicated path under /nix/store/. A NAR fetched from the cache contains the compressed contents of a unique Nix store path: the directories and files that make up a specific package, dependency, or build artifact. This store path is always unique, because it includes a hash derived from all the inputs used to build its contents. Nix prepends this input-addressed hash to the package's name and version, like so: /nix/store/4lbicqd70qlfdw2l5wbq23c9qj1m5nww-cowsay-3.8.4.

By using input-addressed store paths, Nix guarantees a set of properties that are analogous to the atomicity, consistency, isolation, and durability (ACID) guarantees of a relational database.

These properties are what give you reproducible build- and run-time behavior, reliable rollbacks, automatic garbage collection, and safe multi-user installations. This means that users can install, upgrade, or remove packages on the same system at the same time, including packages that pull in conflicting versions of the same dependencies. For example, if you need to run kubectl 1.27 (built from an older package set that uses glibc 2.35) alongside Git 2.51 (built from a newer package set that uses glibc 2.40), both programs can coexist on the same machine thanks to Nix's hashed, input-addressed store paths. And once store objects (packages, build inputs, etc.) are no longer referenced by the global system or individual users' profiles, Nix can automatically reclaim that disk space.

Populating the cache

The Nix binary cache is populated by an automated build pipeline centered around Nix's Hydra CI system, which directs a pool of geographically distributed builder machines.

When a maintainer makes a commit that updates a Nix expression in the Nixpkgs GitHub repo, either by changing its version, dependencies, patches, build instructions, tests, or post-build actions, this triggers Hydra to:

evaluate the commit's Nix expressions across the whole jobset (more than 211,000 packages);
generate derivations from those expressions;
compare them to what's already been built and cached;
queue only the new or changed derivations for builds.

Hydra runs many jobsets several times per day as new commits land on active branches, although especially large and expensive jobsets (such as nixos/unstable) tend to run less frequently. It signs and uploads successful builds to cache.nixos.org, where they're then available as prebuilt binaries.

Interestingly, since around July of 2026, Hydra runs nearly 40,000 fewer build jobs than it did just a year ago. The reason for this is that Nixpkgs recently dropped support for macOS platforms running on x86_64. This coincides with Apple's planned transition away from Intel-based Macs — starting with the upcoming Golden Gate release, new versions of macOS will run only on Apple Silicon.

A number of artifacts from this CI pipeline end up in the cache.

Nix Archive (NAR) files

These nar.xz or nar.zst files, stored at nar/, contain the compressed contents of a single /nix/store/ path. The NAR format is comparable to tar, but much simpler: NARs don't need to support long filenames, POSIX attributes, a wide range of extensions, or rich metadata. They're designed to serialize a filesystem object tree, including bare directory structure, file contents, symlinks, and basic metadata.

The NAR format is an excellent example of simple, stable design: it hasn't been revised or extended since its introduction — more than 20 years ago.
Archive information files

These are top-level files named <store-hash>.narinfo. Think of them as tiny text records that point to the compressed NAR, record its sizes and hashes, list the store paths it references as runtime dependencies, and carry the Hydra build farm's signature so clients can verify the artifact. Despite each individual file's negligible size, there are more than 90 GB of narinfo files in the S3 bucket used to host the Nix binary cache.
Build logs

The log/ folder stores the output of Nix build jobs for debugging and auditing. This allows anyone to fetch the build log for a cached store path with nix log /nix/store/<store-hash>-package-name.
Content-addressed mappings

Metadata in the realisations/ folder maps derivations to their content-addressed store paths. This is relevant for builds using content-addressed derivations (CA-derivations).
Store metadata

nix-cache-info is a small metadata file that describes the basic design of the cache as a whole, such as the cache version and its supported features.

BYOBC

The official cache.nixos.org lives in S3, but if you happen to have half a petabyte of available storage, you could definitely roll your own. As we explained previously, each package in cache.nixos.org has been built from source by a pool of geographically distributed builders — bare-metal and cloud alike, spanning most of Europe — using Nix's own CI.

This build-time reproducibility is possible because the recipes Nix uses to build software, derivations and the Nix language, don't imperatively prescribe anything. Build recipes are expressed in the Nix language; Nix evaluates these to produce a derivation, or a low-level build specification. The derivation records every declared input; so as long as inputs remain available (via cache.nixos.org, in Nixpkgs, or locally) Nix can reproduce any build, anywhere, at any time.

This makes it so anyone can host their own Nix binary cache if they wanted, just by cloning the Nixpkgs GitHub repo and building every package from source. They would need at least a terabyte of space just for the build step, and they will probably want to provision a pool of concurrent builders. And they would need to foot the bill for the data egress charges associated with serving millions of users. But this isn't just technically possible, it's doable. The Flox Binary Cache is one such example: it applies the same principle of reproducible Nix builds to provide prebuilt packages (including CUDA-accelerated packages like PyTorch and TensorFlow) directly from canonical Nixpkgs definitions.

And the best part is that, when built against the same Nixpkgs commits, packages in Flox's or anyone/anywhere's binary cache are almost always bit-for-bit identical to those in cache.nixos.org. That's because Nix builds run inside an isolated sandbox. This means that only explicitly declared inputs are available, so your toolchain can't silently pull in host libraries, environment variables, arbitrary metadata, or other kinds of state. If the inputs match, the outputs match too.

Building deterministically at this scale is surprisingly difficult to achieve. In practice, Nix's functional build pattern nails it for the overwhelming majority of packages (about 99.9%), with the remaining sliver being tracked (and continuously chipped away at) as non-deterministic outliers.

Operating on an enormous scale

The overwhelming majority of Nix clients fetch and use software packages from the Nix binary cache. It fields approximately 5,700 requests per second — almost 15 billion requests per month — and accounts for the vast majority of traffic handled by the public Nix ecosystem's CDN endpoints. It stores 636 TB of data, with around 545 TB in AWS's infrequent-access storage tier and 91 TB in standard storage. Just as impressive, cache.nixos.org, contains more than 1 billion objects, making it one of the largest public software caches on the Internet.

Such a feat requires careful design. Faced with this challenge, one may be tempted to start diagramming complex architectures with eventually-consistent distributed databases and the like.

In a scene from American novelist Cormac McCarthy's Blood Meridian, a character exhibits a giant petrified dinosaur bone before a group of mystified, stupefied cowboys. "Your heart's desire is to be told some mystery. The mystery is that there is no mystery", he tells them.

There is no mystery to the Nix binary cache: it is just a huge, signed key–value store hosted in an Amazon S3 bucket. Every key is derived from a unique hash, and because there's no database lookup required, there is no backing database. This avoids a slew of wicked problems that tend to arise when you're attempting to scale a massive project backed by a database.

The "fresh" contents of cache.nixos.org live in a single S3 bucket in the AWS us-east-1 region. The Nix binary cache also contains packages that are at more than a decade old, software that in some cases has disappeared from the Internet. Because the Nix community has preserved its source code and build definitions, it's possible to rebuild it today, on modern systems, using its required dependencies. As a cost optimization, these older, historical packages typically live in cheaper AWS storage tiers. The bucket for less-frequently used packages recently moved to S3 Intelligent-Tiering, which automatically shifts objects among "Frequent", "Infrequent", "Archive Instant", and "Archive" tiers, based on how often they're accessed. This is expected to reduce storage costs, as well as the effort required of the Nix Infrastructure team to manage archived packages.

If the design of the Nix binary cache is conceptually simple, distributing cached artifacts efficiently and reliably to consumers spread out across the world is a much more difficult problem. To this end, a content distribution network, Fastly, sits in front of cache.nixos.org as its global caching layer. If a Nix package isn't already cached, Fastly streams it from S3 as it arrives, so users don't have to wait for it to download before starting. For very large files, Fastly splits them up — ironically, to get around its own size limitations. It also transparently rewrites certain kinds of S3 permission errors, so users see 404: Not Found errors, rather than foreboding 403: Forbidden messages. Finally, Fastly handles TLS for the cache.nixos.org domain at the edge, providing secure HTTPS connections to clients while communicating with S3 over TLS.

Amazon S3 just works

Even though the binary cache is simple enough to be host on many different architectures, S3 is particularly suited to the task for a few specific reasons. At the time cache.nixos.org was created, there was basically no other option. In 2007, if you wanted access to distributed, highly available, pay-as-you-go object storage, your choices were Amazon S3 or … running Hadoop and HDFS yourself. But unlike S3, HDFS isn't a key–value store — it isn't even an object store! It's a hierarchical, POSIX-like filesystem, not a global bucket that anyone can fetch from over HTTP.

For the Nix community's requirements, S3's key-addressed object storage was the perfect substrate for cache.nixos.org: .nar and .narinfo files could be persisted as object keys and exposed via S3's internet-facing REST APIs. S3 could store a virtually unlimited number of blobs, serve them up directly over HTTP, replicate them for durability, and scale to meet growing demand. This was revolutionary at the time, and perfect for the "not trying to do too much" design ethos of cache.nixos.org.

Today, the Nix binary cache could live in virtually any other hyperscale cloud. Its design ethos focused on simplicity means that it is not locked into a specific platform. The cache is in S3 because S3 works, and the rational choice is to keep it where it works — this is how cache.nixos.org has been hosted for almost two decades, after all.

This status quo was challenged in 2023, when one of the NixOS Foundation's principal financial sponsors discontinued its support. During this time, Nix saw enormous growth, meaning that stable infrastructure was a priority concern. The NixOS Foundation responded to this challenge with radical transparency, and issued a call for assistance from the Nix community and commercial vendors. Our employer, Flox, was one of the vendors that responded to this call, along with others in the Nix-adjacent space. AWS Open Source also responded to this call, pledging credits to help fund cache.nixos.org for the fiscal year — a contribution which proved central to the sustainability of the Nix ecosystem and which started what has become a proven track record of support for the binary cache.

Maintaining the cache

At its core, the binary cache is just a signed collection of blobs in S3, which is itself a basic key–value store, with no backing database and almost no service-dependent logic. This primitivity has the advantage of making the cache comparatively simple to maintain.

But there's an absolute difference between theory and practice. In theory cache.nixos.org is an ideal design, implemented in an ideal cloud service with ideal users and ideal usage characteristics. In practice, and at scale, the binary cache is low- but by no means no-maintenance.

Even the best, most thoughtful designs need upkeep, after all. For the most part, the Infrastructure team supporting the binary cache does its work behind the scenes, to the point that the cache itself seems to "just work". Nevertheless, this team has had to put out its share of fires, as well as deal with occasional crises. The fact that it manages to keep a low profile is a testament to the efforts of its members, present and past. While Amazon and the folks at AWS kindly help offset the fiscal (and, to some extent, technological) burden of hosting the Nix binary cache; the infra team shoulders the operational burden of keeping it available, responsive, and up-to-date.

Acknowledgments

On behalf of the NixOS Foundation, we want to express our gratitude to AWS for its ongoing support. Their help frees the Infra team to focus on durable fixes rather than day-to-day firefighting. It also gives the team headroom for much more ambitious work, like better garbage collection, new mirroring options, and partnering opportunities.

If you'd like to learn more about Nix infrastructure (along with the people who support and maintain it), check out their team page.

Special thanks to Nix-whisperer Tom Bereknyei for consultation, feedback, and hugely constructive criticism during the drafting of this article.

Tales from Nixpkgs - PR

NixOS — Sat, 04 Mar 2023 23:00:00 GMT

This is the first in a new series of posts called "Tales from Nixpkgs" where we examine Nixpkgs commits to understand how this ecosystem works and provide a bit of visibility into the process. This also gives us a chance to show appreciation for the maintainers and perhaps also uncover interesting stories along the way.

Let's take a look at a randomly picked commit from the last year of Nixpkgs:

git log --pretty=oneline --since 1y | shuf | head -n1 | cut -f1 -d' ' | xargs git show
commit 785bafc33818503172c7eecb60af711d794195b3
Merge: e7d00dfbd39 1e69e5c4280
Author: Fabian Affolter <mail@fabian-affolter.ch>
Date:   Fri Mar 3 15:05:51 2023 +0100

    Merge pull request #219037 from fabaff/openai-bump

    python310Packages.openai: 0.26.5 -> 0.27.0

The original PR for this is #219037. The motivation is to update the version of the openai Python package. It looks like along the way there is also a related update to shell-genie and some improvements to the metadata in Nixpkgs. A simple version bump is usually simpler, but this example is good because it will help us explain a few things along the way.

The entails taking a look at the build recipe and updating a few things:

git diff 785bafc33818503172c7eecb60af711d794195b3~1 785bafc33818503172c7eecb60af711d794195b3
pkgs/development/python-modules/openai/default.nix
diff --git a/pkgs/development/python-modules/openai/default.nix b/pkgs/development/python-modules/openai/default.nix
index ede1a7a3d84..92f2e327eb3 100644
--- a/pkgs/development/python-modules/openai/default.nix
+++ b/pkgs/development/python-modules/openai/default.nix
@@ -23,7 +23,7 @@

 buildPythonPackage rec {
   pname = "openai";
-  version = "0.26.5";
+  version = "0.27.0";
   format = "setuptools";

   disabled = pythonOlder "3.7.1";
@@ -31,8 +31,8 @@ buildPythonPackage rec {
   src = fetchFromGitHub {
     owner = "openai";
     repo = "openai-python";
-    rev = "v${version}";
-    hash = "sha256-eKU+WRFf7f1yH63vcoQ9dVeqhJXBqMJGpk/9AoEgR0M=";
+    rev = "refs/tags/v${version}";
+    hash = "sha256-pXttGvnApYuwkWU7kCDNjw0rjHD5AyUvujfvpDVjgxM=";
   };

   propagatedBuildInputs = \[
@@ -91,6 +91,7 @@ buildPythonPackage rec {
   meta = with lib; {
     description = "Python client library for the OpenAI API";
     homepage = "https://github.com/openai/openai-python";
+    changelog = "https://github.com/openai/openai-python/releases/tag/v${version}";
     license = licenses.mit;
     maintainers = with maintainers; \[ malo \];
   };

First off, we see a bump in version from 0.26.5 to 0.27.0. Next we see the source is also updated with its hash providing an integrity check. (Note: these two steps are often performed by automation, stay tuned for future episodes). Lastly we see an improvement to the metadata pointing at a consistent location for the changelog of the OpenAI project.

Next we see changes to shell-genie. This is an application that uses openai to provide helpful command-line suggestions and it uses openai as a dependency.

git diff 785bafc33818503172c7eecb60af711d794195b3~1 785bafc33818503172c7eecb60af711d794195b3
pkgs/applications/misc/shell-genie/default.nix
diff --git a/pkgs/applications/misc/shell-genie/default.nix b/pkgs/applications/misc/shell-genie/default.nix
index 1429aabcd80..7225eb4c3fc 100644
--- a/pkgs/applications/misc/shell-genie/default.nix
+++ b/pkgs/applications/misc/shell-genie/default.nix
@@ -8,18 +8,16 @@ with python3.pkgs;

 buildPythonPackage rec {
   pname = "shell-genie";
-  version = "unstable-2023-01-27";
+  version = "0.2.6";
   format = "pyproject";

-  src = fetchFromGitHub {
-    owner = "dylanjcastillo";
-    repo = pname;
-    rev = "d6da42a4426e6058a0b5ae07837d8c003cd1239e";
-    hash = "sha256-MGhQaTcl3KjAJXorOmMRec07LxH02T81rNbV2mYEpRA=";
+  src = fetchPypi {
+    pname = "shell\_genie";
+    inherit version;
+    hash = "sha256-MgQFHsBXrihfWBB/cz45ITf8oJG2gSenf1wzdbrAbjw=";
   };

   nativeBuildInputs = \[
-    poetry
     poetry-core
   \];

@@ -35,9 +33,14 @@ buildPythonPackage rec {
   # No tests available
   doCheck = false;

+  pythonImportsCheck = \[
+    "shell\_genie"
+  \];
+
   meta = with lib; {
     description = "Describe your shell commands in natural language";
     homepage = "https://github.com/dylanjcastillo/shell-genie";
+    # https://github.com/dylanjcastillo/shell-genie/issues/3
     license = licenses.unfree;
     maintainers = with maintainers; \[ onny \];
   };

We see a set of very similar changes, a version bump, update and improvement to how we fetch the sources, additional checks performed, as well as an identification of an issue - there is no declared license for the shell-genie project. This brings us to an important part of the work of packaging. It is not only the technical work to shepherd updates along and fix bugs, but to communicate and coordinate across the open source world. @fabaff submitted an issue to the upstream project which resulted in this addition.

It is through many small changes and fixes like this that the overall ecosystem improves and these improvements have benefits not just for the Nix ecosystem, but outside of it as well. A common theme we will see in these tales is that our very strict approach to packaging has trickle-down benefits for upstream projects and other package managers.

So we have a fairly simple update, but also improvements to metadata and collaboration with other communities. Next in the PR we see some testing of the new recipe across various architectures and an approval from @malob. With the approval and all checks and tests in the green, the PR was merged. Sounds like success!

Thank you @fabaff for your contribution!