Using Nix to Develop and Package a Scala Project

Published in

Zendesk Engineering

9 min readOct 24, 2023

Nix is a functional programming language tailor-made for defining pure and reproducible builds. It is at the heart of the Nix Package Manager and the foundation of the NixOS Linux distribution. It has gained popularity due to its principled approach and large ecosystem of available packages. It’s also an absolute joy to use on a daily basis! This post will show how Nix can be used to develop and package a typical software project without the hassle of relying on language specific version managers or brittle setup scripts. We will be using a simple Scala project today, but many of the concepts can easily be transferred to other stacks.

The Nix language has a fairly small — though at times obscure — syntax, and a standard library focused on build systems. Most Nix programs consist of little more than pure function definitions, map-like data structures called attribute sets, and the occasional declaration of temporary variables. A typical Nix program is shown below, representing a pure function that takes one attribute set and produces another attribute set:

let
  mkGreeting = { name }: {
    greeting = "Hello ${name}";
  };
in
  mkGreeting { name = "world"; }

If you try this example via nix repl, you should see the output { greeting = "Hello world"; }.

Sidebar: Installing Nix

The Nix package manager is available for Linux and MacOS at nixos.org, or alternatively the official Docker image can be used. We are going to utilize two experimental features throughout this post, nix-command and flakes, which need to be enabled in the global Nix configuration:

echo "experimental-features = nix-command flakes" >> /etc/nix/nix.conf

Setting up a Common Development Environment

Now that we know the basics of Nix, we want to use it to define an environment for working on a Scala project that we can easily distribute to our fellow contributors. At a minimum, we might want to define a set of dependencies to install, Nix calls these ‘build inputs’. Our little hello world example provides us with most of the building blocks we need, a function which returns an attribute set:

{pkgs}: {
  buildInputs = [pkgs.sbt pkgs.metals];
}

We are only missing two things to turn this into a functioning development environment, specifying the source of packages and the correct name for the function so Nix can find it.

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
  };

  outputs = { self, nixpkgs }:
    let
      pkgs = import nixpkgs { system = "x86_64-darwin"; };
    in {
      devShells.x86_64-darwin.default = pkgs.mkShell {
        buildInputs = [pkgs.sbt pkgs.metals];
      };
    };
}

Our function is now called outputs.devShells.x86_64-darwin.default, and packages are sourced from the nixpkgs GitHub repository. Nix supports dozens of system architectures, hence the need to specify our system for both the development shell as well as the package import. Note that the structure of our program now represents a nested attribute set, containing inputs and outputs. These special Nix programs which follow a well-defined schema are called Nix flakes and are always stored in a file named flake.nix. In addition to their special schema, flakes pin all their dependencies via a lock file and integrate tightly with Git to ensure a consistent environment.

The standard way to create a development shell is to run nix develop in the directory that contains the flake. The new shell will contain sbt, the Scala build tool, as well as metals, the Scala language server, in the path just as was requested in the build inputs. We are finally ready to create our Scala example project with sbt new http4s/http4s.g8 and start editing away in our favorite editor with language server support.

The Nix Store

Running which sbt or inspecting $PATH will reveal the origin of sbt: /nix/store/vq4r0bf8gqipmbwfv1lz9zbkkgkxx6h1-sbt-1.8.3 . This is a regular directory, provisioned by Nix based on our build input. The hash included in the directory uniquely identifies the package and all its build dependencies. Listing the content of the directory, we will find an sbt script and a launcher JAR file.

Sbt requires a JDK (Java Development Kit) to run, but our flake did not specify one. Knowing that Nix manages all packages in /nix/store, we can search the sbt package to see what other packages it references:

And indeed, the sbt package references a JDK due to its own build dependency. It also references a specific version of bash, since sbt includes a shell script. Note that the paths to the store are stable and were generated when the sbt package was built. Unless the referenced Nix packages are already available in the local store, Nix will automatically install them — neat! This is how Nix manages all runtime dependencies, simple but efficient.

Sidebar: Using a different JDK for sbt

Having sbt depend on a JDK directly is great and will ensure it works consistently without us having to provide a compatible version of Java. However, it may not be desired when developing for a Long Term Support release of Java. In order to remove the dependency, we can add an overlay to our import statement which removes the patch that adds the dependency in the original package. Afterwards, we are able to define and use a specific JDK in the build inputs.

pkgs = import nixpkgs {
  system = "x86_64-darwin";
  overlays = [
    (final: prev: {
      sbt = prev.sbt.overrideAttrs { postPatch = ""; } ;
    })
  ];
};
...
buildInputs = [pkgs.sbt pkgs.metals pkgs.jdk17_headless];

Creating a Nix Package

So far we have only used third party packages from the official nixpkgs GitHub repository. We will now package our own project. Nix packages are based on derivations: descriptions of how to build a specific library or project. A typical derivation is split into multiple phases. If you have used autoconf and Makefiles, the phases will feel familiar — in fact projects with Makefiles will often work with a minimal Nix derivation:

packages.x86_64-darwin.default = stdenv.mkDerivation {
  pname = "foo";
  version = "1.0.0";
  src = ./.;
}

The default package for project foo, version 1.0.0, with source files located in the current directory (Nix has native support for file paths), will be built in three main phases:

optional configure phase: run ./configure if present
build phase: run make
install phase: run make install

There are additional phases that can be used to patch sources, run tests or fix-up artifacts, but they are not required for most projects.

The default package declaration is part of our flake definition, and similar to the development shell requires a system. Flakes can contain packages for many different systems, and most flakes define packages for the four default systems: x86_64-linux, x86_64-darwin, aarch64-linux and aarch64-darwin. Instead of manually defining packages for these systems, we can use a utility function to generate the necessary attribute sets. Scala projects don’t usually come with a Makefile, so we will manually specify the build and install phase. We will also specify build inputs. All put together, the flake for our example project looks like this:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, utils, }:
  utils.lib.eachDefaultSystem (system:
  let
    pkgs = import nixpkgs {inherit system;};
  in {
    devShells.default = pkgs.mkShell {
      buildInputs = [pkgs.sbt pkgs.metals pkgs.hello];
    };

    packages.default = pkgs.stdenv.mkDerivation {
      pname = "nix-scala-example";
      version = "0.1.0";

      src = ./.;

      buildInputs = [pkgs.sbt pkgs.jdk17_headless pkgs.makeWrapper];

      buildPhase = "sbt assembly";

      installPhase = ''
          mkdir -p $out/bin
          mkdir -p $out/share/java

          cp target/scala-2.13/*.jar $out/share/java

          makeWrapper ${pkgs.jdk17_headless}/bin/java $out/bin/nix-scala-example \
            --add-flags "-cp \"$out/share/java/*\" com.example.nixscalaexample.Main"
      '';
    };
  });
}

The build phase uses sbt and the sbt-assembly plugin. Nix provides sbt and the JDK based on our build inputs. The install phase has a few interesting things to point out. Installing in this context refers to producing the files which will be made available in the project’s Nix store directory. Since we can’t know the the project’s store directory ahead of time, Nix provides a variable named $out which contains the full path, including the uniquely generated hash. The other interesting part of our install phase is the way runtime dependencies are injected. As we saw earlier, runtime dependencies are reflected by referencing another project’s Nix store path. By referencing the JDK package via ${pkgs.jdk17_headless} in the install phase, we instruct Nix to interpolate the path to the JDK’s store directory and install it at runtime unless it is already present.

Dealing with Non-Nix Dependencies

Our goal is to build a fully reproducible package, but so far that does not include the non-Nix dependencies which are managed by sbt. Fortunately, various tools are available that bridge the gap between language specific dependencies and Nix, e.g. bundix for Ruby gems, buildGoModule for Go modules and sbt-derivation for Scala dependencies. They all work in a similar fashion using two-phase builds: an intermediate fixed-output derivation to fetch and lock dependencies and the final derivation to use the locked dependencies.

Using sbt-derivation for our example project requires only minimal changes to the flake. The referenced depsSha256 is a hash of all our dependencies defined in the sbt build file and ensures that all dependencies are locked.

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
    utils.url = "github:numtide/flake-utils";
    sbt.url = "github:zaninime/sbt-derivation";
    sbt.inputs.nixpkgs.follows = "nixpkgs";
  };
  ... 
    packages.default = sbt.mkSbtDerivation.${system} {
      depsSha256 = "sha256-Vh0WCGEexw8aHt+cLYUgEnpH7NZY+naxj7lTSonsjyM=";

With a package definition in place, we can build the project with nix build or nix build -L to see build logs. Once built, the package can be found in the Nix store, for convenience it is also linked via the result folder. Running result/bin/nix-scala-example would start our project, in this case a http4s server. Other ways to run the server are by executing nix run in the current directory or via the GitHub integration nix run github:rgueldem/nix-scala-example.

It is important to note that the runtime behavior of the package will be identical across machines. Nix guarantees reproducibility by injecting the exact same runtime dependencies everywhere, e.g. the exact same Java version for our example project. We have successfully packaged our project!

Sidebar: Secrets

It is sometimes necessary to provide secrets to the build phase, e.g. for authenticating against a private package repository. Since Nix fully sandboxes the build process, we need to inject the necessary secrets manually. We can either read secrets from file or from an environment variable (both approaches require --impure when building) which will be made available to the build phase:

SOME_SECRET = builtins.getEnv "SOME_SECRET";

buildPhase = ''
  echo "${builtins.readFile ~/.some_secrets}"
  echo $SOME_SECRET
'';

Docker Image

Nix packages are great, but most teams probably won’t deploy them directly to production. Fortunately, there is an easy way to convert a Nix package into the omnipresent Docker format using Nix bundle. Note that the bundling should happen on the same system architecture which will be used for running the Docker image, typically x86_64-linux or aarch64-linux. The package definition for our example project supports both these systems already, so we are all set.

$ nix bundle --bundler github:NixOS/bundlers#toDockerImage
$ docker load -i nix-scala-example-0.1.0.tar.gz

The resulting image contains all runtime dependencies from the Nix store as well as the project itself, each in their own layer. There is no base operating system, package manager or unnecessary caches.

Conclusion

We have seen how Nix can be used to setup a common development environment and build fully reproducible packages of a project. Nix can do a lot more: manage dotfiles, local packages or entire machines. It is also possible to replace most version managers (rbenv, asdf, jenv etc) with Nix. The languages does have some gotchas, but it provides a modern take on dependency management that’s worth considering for your next project.

With contributions from Trevor Thompson and Tim Cuthbertson. The example project is available on GitHub.