Introduction

Reproducibility is a major challenge that impacts all empirical sciences (Ioannidis, 2005). Researchers across multiple disciplines refer to the inability to reproduce published scientific results, experiments, and computational methods as the reproducibility crisis.

In computational sciences, high-performance computing (HPC) is particularly challenging because of the interplay among source code, software environment, and hardware (Antunes & Hill, 2024). Besides the full source code, it is necessary to reconstruct the complete software environment required to build and execute computational experiments with reproducible results; including compiler versions, library dependencies, operating system (OS) tooling, OS configuration details, environment variables, accelerator toolchains.

Traditional package managers for dependency management, and even containers to build the global system state, struggle to provide this level of reproducibility (Vallet et al., 2022).

Functional package managers such as Nix (Dolstra, 2006) and Guix (Courtès, 2013) address these issues by treating software builds as pure functions of their inputs. Instead of modifying a global system state, they construct isolated and immutable software environments whose outputs are identified by hashes based on its inputs. In practice, Nix and Guix allow building identical software environments across containers, physical, and virtual machines. This includes pinning exact code, dependencies, and toolchain versions.

While Nix and Guix can be both used for reproducible scientific computing, this article focuses Nix because of its larger software ecosystem and practical support for popular unfree software (e.g., CUDA).

In the context of HPC environments, applications frequently have large C++ codebases, rely on MPI for distributed computing, and target heterogeneous hardware (e.g., graphics processing units (GPUs) and field-programmable gate arrays (FPGAs)). These type of projects often require repeated compilation for both CPUs and accelerators during active development. Even incremental changes may trigger time-consuming rebuilds. Consequently, compilation time slows down the development and iteration speed.

This is where ccache becomes useful (Rosdahl & Tridgell, 2002). ccache is a compiler cache designed to avoid unnecessary compilation by caching and reusing previously generated object files. ccache computes hashes from compiler inputs and retrieves cached build artifacts whenever possible. This can significantly reduce rebuild times as only changed objects are compiled.

Interestingly, both Nix and ccache make heavy use of hashing for fundamentally different purposes. On the one hand, Nix uses hashes to guarantee reproducibility and identify immutable build environments. On the other hand, ccache uses hashes of source files and compiler inputs to identify reusable objects and optimize compilation performance. These differing goals create subtle interactions that could lead to ineffective ccache caching in Nix-based environments.

In the following, I will discuss the practical interaction between Nix and ccache, how to configure them on NixOS, and how to avoid common pitfalls that invalidate performance improvements.

Understanding Ccache

ccache is a C/C++ compiler cache that stores build artifacts and reuses them during recompilation. It is a wrapper around existing compilers. At a high level, ccache sits in front of the compiler:

source ⟶ ccache ⟶ compiler ⟶ object

Instead of compiling every source file, ccache computes hashes over compilation inputs. Afterward, it performs a lookup on the object cache. If an object with an identical hash exists in the cache, it is reused and not compiled.

Two steps are executed when a compilation command is issued through ccache,

1. hash of compilation inputs is computed (see next Section)
2. lookup if an object with the same compilation hash exists in the cache:
   • returns cached object with same hash (cache hit)
   • compile object file and stores in cache (cache miss)

The effectiveness of ccache depends on the stability of compiler inputs (e.g. only incremental source code changes).

Hash Modes

ccache uses three different hashing modes to identify previously compiled objects: direct, depend, and preprocessor (Rosdahl & Tridgell, 2014). All three have a shared set of common hash inputs and add different information on top.

Direct mode is faster than preprocessor mode. By default, ccache performs a lookup first in direct mode and only falls back to preprocessor mode in case of a cache miss. Depend mode fully skips the preprocessor mode, even on cache misses.

Even with these mode-specific differences, small changes to optimization flags, include search paths, compiler versions, generated headers, timestamps, or build paths can still invalidate cache entries.

This becomes especially relevant on NixOS because Nix intentionally rewrites many compiler and dependency paths into unique store paths.

Understanding Nix

When I started with Nix back in 2021, a major point of confusion for me was to understand the terminology around Nix:

1. Nix is a full functional programming language that contains domain specific elements to declare the environment and build process of software. A Nix recipe to declare the environment and build process of an application is called a derivation.
2. The repository of recipes is called nixpkgs. As of writing, it contains more than 120,000 packages.
3. The Nix language can be used to declare a full operating system, including system services, users, etc., called NixOS.

In this section, we will focus on how Nix uses hashes to ensure the reproducibility of derivations. The interested reader is referred to the Nix Pills (Preface - Nix Pills — Nixos.org, 2026), and Jon Ringers Nix book (Ringer, 2026). If you prefer videos, Vimjoyer creates high-quality videos about Nix (Vimjoyer, 2024).

Nix is fundamentally built around hashing. Traditional package managers typically install binaries into paths such as /usr/bin, /bin, or $HOME/.local/bin. Normally, multiple versions of a program share the same name.

By contrast, Nix stores every build artifact in immutable store paths under /nix/store created from a unique hash.

A nix store path contains a hash and package name:

/nix/store/<hash>-package-name

The hash is calculated from all build inputs, including source code, compiler versions, dependency versions, build flags, environment variables, and the derivation itself. This hash ensures reproducibility by changing with modifications of the source code or the software environment. Furthermore, different versions of programs and libraries can coexist without interference. Therefore, the building process becomes deterministic without side effects as the final artifacts only depends on the inputs.

Nix restricts file system access and builds software in isolated sandboxes to avoid environmental pollution and ensure deterministic builds.

However, Nix’s sandboxing approach directly interferes with ccache, which requires a persistent non-sandboxed shared cache. In addition, ccache can be invalidated when the nix store hash changes.

Nix and Ccache

While Nix and ccache make heavy use of hashing, they have a different scope. Nix achieves reproducibility by fully sandboxing every build; thus removing pollution from the computational environment or previous builds. ccache is only effective if it has access to cached objects from previous builds. Therefore, combining Nix and ccache requires to partially break the sandbox and permit non-deterministic behavior to increase the compilation speed.

Nix hashes build inputs. Among other things, build inputs capture source code, toolchain, dependencies, build environment, and the derivation itself to guarantee reproducibility. ccache hashes compiler inputs that are necessary to compile individual object files and avoid recompilation of identical objects.

This distinction has practical consequences. As changes of the Nix hash may invalidate ccache entries.

Why Cache Misses Happen on NixOS

This section explores several characteristics of Nix that can invalidate ccache entries and offers recommendations how to improve cache hit rates. Example configurations are given for ccacheWrapper and development shells.

Store Paths, Absolute Paths, and Compiler Wrappers

Among the defining characteristics of Nix is that every derivation is installed in a path based on it’s hash to the global /nix/store. The hash is computed from all build inputs, including the full dependency graph and build environment. Consequently, small changes to the source code, dependency versions, compiler flags, and derivations generate entirely new store paths. To ensure reproducible builds, the Nix build process uses absolute paths to refer to build inputs.

Let’s consider the example that for a C++ project that depends on mylib. During the compilation, Nix injects the absolute path (e.g., /nix/06409a3d2d3b1113zdk9rm5a3ns4rizz-gcc-14.3.0/bin/g++ -I /nix/store/4nfmawqzrn9g9cmd36mvypyr73pvqm3w-mylib/lib test.cpp -o test) to guarantee that the right version of the library and compilers are used. Furthermore, the build itself will be performed in a temporary directory (/tmp/nix-XXXXXX/build in my case) in the that the build environments can not interfere with one another¹. Consequently, even small changes that are not affecting the compilation result in new Nix store paths (e.g. the addition of tests).

This behavior creates challenges for ccache as it computes its hash from the compile inputs. As we see in the previous example, compiler invocations on NixOS usually contain absolute paths into /nix/store. Absolute paths are used to identify include directories, linker paths, runtime search paths, and compiler wrapper arguments.

When any dependency hash changes, these paths also change, even when the objects being compiled remain identical. From the perspective of ccache, these modified paths appear as new compiler inputs with new hashes, invalidating previously cached object files.

This effect is even more pronounced because Nix generally does not invoke compilers directly but uses compiler wrappers through stdenv. These wrappers inject additional compiler and linker flags to guarantee isolated environments and reproducibility of the build, increasing the number of changes of the compiler inputs between rebuilds.

Compiler Flags for Deterministic Builds

A major contributor to cache invalidation is the automatic injection of the -frandom-seed compiler flag by Nix. The addition of this flag is necessary to guarantee deterministic builds. It places a seed that the compiler uses to create deterministic symbol names, pointer addresses, etc.

Nix injects the first 10 characters of the store hash as random seed, changing the compiler inputs every time the Nix hash changes. In our previous example, mylib would be compiled with -frandom-seed=4nfmawqzrn.

Improving the Hit Rates

ccache offers several strategies to improve cache hits by relaxing the correctness requirements of individual lookups, increasing the risk of false positives.

Taming Nix Hashes

One of the major driver of cache invalidation when using Nix is the handling paths during compilation.

First, ccache includes the current working directory when hashing the compiler inputs. While traditional Linux systems usually have a stable directory structure, Nix’s sandboxing results in different paths between recompilation. When

export CCACHE_NOHASHDIR=true

is set, the working directory no longer contributes to the ccache hash.

Second, Nix creates a unique and random temporary build directory at every invocation. Configuring CCACHE_BASEDIR rewrites absolute paths into relative paths before ccache calculates the compiler input hash. As a result, builds performed in different temporary directories can reuse cached objects as long as the rest of the compiler inputs are identical. For sandboxed Nix builds, the current build directory is stored in the environmental variable NIX_BUILD_TOP².

export CCACHE_BASEDIR="$NIX_BUILD_TOP"

When working with development shells, we typically want to build and test our changes locally. The base directory can be set to the root path of the project:

export CCACHE_BASEDIR="$PWD"

Both settings significantly improve cache hit rates during between recompilation because they eliminate unnecessary hash differences stemming from Nix’s reproducibility requirements.

Controlled Non-Determinism

Nix injects the -frandom-seed compiler flag to build reproducible binaries. However, -frandom-seed changes with derivation hash. With compiler flags being part of ccache’s input hash, changes of the derivation hash invalidate the cache entries even when the rest of the inputs and source file are equivalent.

To address this issue, ccache allows the relaxation of the hash validation. The sloppiness setting allows to fine tune, what part of the hash validation should be relaxed. For Nix-based builds, setting random_seed is recommended. When random_seed is set, ccache ignores -frandom-seed setting during the hash calculation.

export CCACHE_SLOPPINESS=random_seed

Without this setting, ccache would miss most functionally equivalent objects during recompilation. Nevertheless, the reader should be aware that relaxing hash validation improves cache hits by fully sacrificing binary reproducibility³.

Setting Up Ccache on NixOS

This section follows the official NixOS Wiki how to set up ccache with the aforementioned settings (CCache - Official NixOS Wiki — Wiki.nixos.org, 2024).

Installing and Configuring Ccache

NixOS has a built-in ccache module that can be activated in the global NixOS configuration.

programs.ccache = {
  enable = true;
  cacheDir = "/var/cache/ccache";
};

While this activates the ccache compiler wrapper integration and configures the global cache directory, it neither creates said directory nor allows the Nix sandbox to access it.

Configuring the Cache Directory on NixOS

While you can set up the ccache directory imperatively with mkdir and allow the nixbld group to access it with chgrp, there is also a better way: the Nix way. 😜

You can take advantage of systemd-tmpfiles to create directories and manipulate permissions. Unlike what the name suggests, they stay persistent after reboots as long as they are placed on permanent storage without clean-up rules or ages.

The cache directory with appropriate permissions can be declared as follows:

systemd.tmpfiles.rules = [
  "d ${cfg.cacheDir} 2775 root nixbld - -"
  "z ${cfg.cacheDir} 2775 root nixbld - -"
];

The cache path must be added to the sandbox explicitly.

nix.settings.extra-sandbox-paths = [ cfg.cacheDir ];

Without this setting, builds running inside the sandbox have no access the cache directory at all.

Ccache Wrapper

To globally configure ccache, it is not enough to set programs.ccache.enable. The default wrapper needs to be customized through overlays. This is necessary to configure all settings that increase the cache hit rate:

nixpkgs.overlays = [
  (self: super: {
    ccacheWrapper = super.ccacheWrapper.override {
      extraConfig = ''
        export CCACHE_COMPRESS=1
        export CCACHE_DIR="${config.programs.ccache.cacheDir}"
        export CCACHE_UMASK=007
        export CCACHE_SLOPPINESS=random_seed
        export CCACHE_NOHASHDIR=true
        export CCACHE_BASEDIR="$NIX_BUILD_TOP"
      '';
    };
  })
];

This configuration enables cache compression, configures shared permissions, and improves the cache hit rate by ignoring compiler random seeds, working directory, and using relative paths.

Using the NixOS Compiler Wrapper

There are two entry points to build derivations with ccache. System-wide, top-level packages can be configured to use the ccache compiler wrapper directly:

programs.ccache.packageNames = [selectedPackageNames];

More advanced setups may require to override entire compiler toolchains. This can be achieved through overlays. The overlay replaces the stdenv transparently with the ccacheStdenv:

nixpkgs.overlays = [
  (
    _final: prev: {
      clang_21 = prev.llvmPackages_21.clang.overrideAttrs (_: { 
        stdenv = ccacheStdenv;                                  
      });                                                         
    }
  )
];

You can always override the stdenv of packages and their dependencies by overlays. This approach works consistently across nix build, nix run.

Using Ccache Inside a Development Shell

Development shells provide a very convenient way to integrate ccache into C/C++ development workflows. Since the shell environment remains stable across rebuilds, cache hit rates are much higher than during isolated package builds.

A minimal development shell with ccache support looks like:

{
  pkgs,
  stdenv,
  ...
}:

pkgs.mkShell {
  packages = with pkgs; [ccache];

  env = {
    CCACHE_DIR="/var/cache/ccache";
    CCACHE_NOHASHDIR = 1;
    CCACHE_COMPILERCHECK = "content";
    CCACHE_SLOPPINESS = "random_seed,time_macros";
    NIX_CC = "${stdenv.cc}";
    CC = "${stdenv.cc}/bin/cc";
    CXX = "${stdenv.cc}/bin/c++";
  };

  shellHook = "
    export CCACHE_BASEDIR="$PWD"
  ";
}

The environment is constructed, such that standard build tools (e.g., ninja, CMake, or Meson) can pick up all build dependencies and ccache.

A small but important distinction is the difference between env and shellHook⁴. env is used to declare static shell variables that are computed before launching the shell. They remain stable across shell invocations. By contrast, shellHook executes dynamically during the startup of the shell. Here we use it to assign our current directory as the CCACHE_BASEDIR.

Practical Tips

For most users, ccache works best as a local development optimization rather than system-wide rebuilds. As I usually build my NixOS system only once for a complete set of build inputs, I only achieve ~20% cache hits at considerably longer compile times with ccache enabled for large packages like ffmpeg, electron, and Firefox.

Therefore, my recommendation for a practical workflow is centered around iterative rebuilds of my derivation after small changes of the source. In general, you should keep your toolchain and dependencies stable and use local cache directories. To integrate ccache into nix run and nix build by replacing derivation’s stdenv with the ccacheStdenv wrapper. Set up a development shell for nix develop by setting environment variables and your normal build scripts (e.g., CMake). This increases the cache hits when the toolchain and dependencies are kept stable, as the Nix store hash only includes the environment and not the source.

If you are unsure whether your application can profit from the use of ccache, benchmark w.r.t. cache misses and compilation time. A simple example script to assess the impact of ccache could look like this:

# 1. Baseline
time nix build .#noccacheDerivation # build time without ccache

# 2. First Run with ccache
ccache -C # clean cache to ensure fresh start
ccache -z # reset statistics
time nix build .#ccacheDerivation # or nix run or custom build script in nix develop
ccache -s # statistics after first run

# 3. Second Run with ccache
sed -i 's/guix is cool/nix is awesome too/g' fancy-code.cpp
ccache -z # reset statistics
time nix build .#ccacheDerivation # or nix run or custom build script in nix develop
ccache -s # statistics after second run

If you see low cache hits, additional debugging can be enabled by setting:

export CCACHE_LOGFILE=ccache.log

This activates detailed logs that help to identify issues such as changed include paths or unexpected environmental differences causing cache invalidation.

Hashing Example Performance

Before introducing ccache blindly to any C/C++ project, it is necessary to evaluate its performance impact. The first compilation of a project always results mostly in cache misses because no previously compiled object files exist yet. This initial build is similar to a normal compiler wrapper and can be used to assess the computational overhead introduced by the hashing algorithms. The actual performance improvements appear during subsequent rebuilds. To assess the effectiveness of ccache, the initial hashing overhead and cache hit rates of subsequent builds need to be considered.

The creators of ccache compared the different hashing modes for a simple C program and preprocessor heavy C++ program (Rosdahl & Tridgell, 2014). While the computation overhead for the simple C program is with 1.6% to 4.0% relatively small, it can speed up compilation time by orders of magnitude. Direct mode is the fastest hash mode and outperforms preprocessor mode by a factor of 5. In contrast, the preprocesser heavy C++ program introduces a computational overhead up to 20%. While direct mode is still faster then preprocessor mode by a factor of 6, the overall performance is only a fith of the simple C program.

Their findings support my observation of increased compilation time and low cache hit rates when enabling ccache for large packages for system rebuilds. Therefore, ccache should only be activated for packages that are frequently rebuilt with the same build inputs.

Example Project

In practice, the difference between an empty and populated cache becomes directly visible when exploring the cache hit / miss statistics. Let’s consider a development shell with activated ccache to hack on GROMACS 2021, a large C++ code base that depends on MPI, LAPACK and CUDA. The first build produced mostly cache misses because no reusable objects are stored in the cache yet:

Cacheable calls:   1493 / 2589 (57.67%)
  Hits:              93 / 1493 ( 6.23%)
    Direct:          12 /   93 (12.90%)
    Preprocessed:    81 /   93 (87.10%)
  Misses:          1400 / 1493 (93.77%)
Uncacheable calls: 1096 / 2589 (42.33%)
Local storage:
  Cache size (GB):  1.3 / 20.0 ( 6.65%)

After rebuilding the same project with only minor source code modifications to a single file, the cache hit rate drastically increases:

Cacheable calls:   1442 / 2476 (58.24%)
  Hits:            1316 / 1442 (91.26%)
    Direct:        1309 / 1316 (99.47%)
    Preprocessed:     7 / 1316 ( 0.53%)
  Misses:           126 / 1442 ( 8.74%)
Uncacheable calls: 1034 / 2476 (41.76%)
Local storage:
  Cache size (GB):  1.3 / 20.0 ( 6.74%)

It should be noted, that not only the overall hit rate increases, but we can observe a shift from preprocessed mode towards direct mode. During the second build, the majority of cache hits are in direct mode (the fastest hashing mode). This illustrates how stable compiler invocations and dependency paths allow for significant performance improvements on NixOS.

Conclusion

Nix and ccache target different problems. While Nix aims to provide a deterministic environment with reproducible builds, ccache prioritizes compilation speed. When combined, they provide a fast and reproducible⁵ C++ development workflow.

Both are relying heavily on hash algorithms, making their interaction counterintuitive. Nix changes store path hashes whenever the build inputs differ, while ccache attempts to maximize reuse across rebuilds based on compiler, path and source characteristics. This result in a conflict between reproducibility and caching.

However, the two systems complement each other well. Nix provides stable and reproducible development environment, while ccache can significantly improve compile times during the active development process.

For most workflows, I would recommend the following:

• use Nix for environment management
• use ccache primarily inside development shells and packages
• do not use ccache for the final binary to avoid false positive hits
• benchmark beforehand if ccache is advantageous

To conclude, Nix and ccache offer a suitable compromise between reproducibility and compilation time during active development of C++ code bases.

Notes