Zero to Nix: Packaging & Development Environments

If you're just landing here, note that this is the second in a series of posts about learning Nix. You can read the other posts if you'd like to learn more about Nix, or continue with this one. I've tried to make them mostly self-contained so you shouldn't read them in order if you're only using them for reference.

All posts:

• Zero to Nix: Everything I Know About Nix & NixOS
• Zero to Nix: Packaging & Development Environments

In the last post we learned about installing Nix and NixOS, accessing REPL, writing some basic Nix expressions and files, and downloading and using packages from <nixpkgs> package collection. This time, we will learn to write our own packages, learn how we can build ad-hoc development environments that others can use, and how we can build our software using Nix and even learn a little bit about how we can speed up those builds using caching (although caching should probably be discussed in detail later).

Installing Packages

While I do not intend to discuss managing your system's entire configuration in Nix, it is essential that you are able to install basic packages from <nixpkgs> so you can use them globally on your system. But before we begin, note that depending on the Nix version that you're using, flakes are experimental features. While you can modify the Nix configuration to enable experimental features, for now we're just going to do that by passing in the command line flags when needed. To do that, you can run nix --extra-experimental-features nix-command --extra-experimental-features flakes <subcommand>. If, however, you've installed Nix using the alternate installer that I mentioned in the previous post, these features are likely already enabled.

The first thing that we need to learn is to find packages that we need. That's easy because one can always look up packages on this Search Engine for Nix packages. Often installing these packages is very easy. To install a package that's available on <nixpkgs> that you found using the aforementioned search, you can simply enter nix profile install nixpkgs#<package>. For example, to install jq, one can run nix profile install nixpkgs#jq. The jq package should now be available in your path.

Writing your first Nix package

Let's say that you have a program that you've written in programming language of your choice. Now you'd like to compile this program and share it with others so they can use it with Nix's package management. Here is a small C program that just prints "Hello, world!" that we're going to try our hand on.

#include <stdio.h>

int main() {
    printf("Hello, World!\n");
    return 0;
}

If you remember from the previous post, a usable Nix package is managed using something called a derivation. We are now going to write a new derivation for our package.

Assuming that you have Nix installed, let's open the directory with your C file and create a default.nix file there. Now let's write a derivation in our Nix file and we'll go over it.

let
  nixpkgs = import (fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/nixos-21.11.tar.gz";
    sha256 = "04ffwp2gzq0hhz7siskw6qh9ys8ragp7285vi1zh8xjksxn1msc5";
  }) { system = builtins.currentSystem; };
in
with nixpkgs;

stdenv.mkDerivation rec {
  pname = "hello-world";
  version = "0.1.0";

  src = ./.;

  buildInputs = [ gcc ];

  buildPhase = ''
    gcc -o hello hello.c
  '';

  installPhase = ''
    mkdir -p $out/bin
    cp hello $out/bin/
  '';
}

Let's break down the code in our Nix file.

1. We declare nixpkgs import the contents of the Nix package registry. We use the fetchTarball function that takes a URL and hash for downloading and extracting a tar file. While importing the nixpkgs function object, we pass the (optional argument) system, which is the system for which we are building. We use the builtins module, which has a set of functions that come with Nix to automatically detect the current system and use that.
2. We call the stdenv.mkDerivation function to build a derivation. Note that stdenv is available in nixpkgs and since we're using the with nixpkgs; syntax, we can call it directly. mkDerivation function lets us build a derivation (or a Nix package, so to speak).
3. The mkDerivation function takes pname, which is the name of the package, a version tag, the src (i.e., source) of any files that we're going to need to build, buildInupts, which are any other existing packages that we might need to build our own package. Then we use buildPhase and installPhase to write shell scripts that will build and install the package.
4. Building the code is very simple because we just call gcc compiler, and for installation we just create an $out/bin directory and copy over our build artifact to it. Note that in a derivation, if you put anything inside the $out/bin directory, it is considered an executable output of your package.

Once we have that, we can simply build this code using nix build default.nix command. Nix will then create a result directory with a bin subdirectory containing the output of the binary package.

At this point, I think it's important to note that executable binaries are not the only target of a derivation. One could also build library outputs. If you're writing packages you might want to look at the derivations documentation in Nix reference. Here is a quick list of parameters that you can pass to mkDerivation function.

• name: A string representing the name of the package, including the version.
• pname: A string representing the name of the package, excluding the version. When both pname and version are provided, mkDerivation constructs the name attribute.
• version: A string representing the version of the package.
• src: A store path or a derivation that produces the source code of the package.
• buildInputs: A list of build-time dependencies.
• nativeBuildInputs: A list of build-time dependencies that are needed only by the build platform.
• propagatedBuildInputs: A list of build-time dependencies that are also propagated to dependent packages.
• configurePhase, buildPhase, checkPhase, installPhase: Custom script phases for configuring, building, testing, and installing the package, respectively.
• preConfigure, postConfigure, preBuild, postBuild, preCheck, postCheck, preInstall, postInstall: Custom script hooks to be run before or after the corresponding phases.
• doConfigure, doBuild, doCheck, doInstall: Booleans that control whether the corresponding phases should be executed. They are set to true by default.
• installFlags, configureFlags, makeFlags: Lists of flags to be passed to the corresponding commands during package installation, configuration, or building.
• meta: A set of metadata attributes for the package, such as description, homepage, license, platforms, and maintainers.

Flakes & Dependency Management

If you're already somewhat familiar with Nix, then you might have heard of Nix flakes. What we've seen so far are the features of Nix that Nix is fundamentally known for. But over time, people have complained about a lack of better dependency management. This has led to projects such as Niv that help you manage dependencies in a declarative way and be able to pin dependency versions. For example, one you can use command line to add packages to your Nix configuration project like you do with npm or yarn with NodeJS or with cargo in a Rust project and then use a lock file for pinning versions. This trend gave us Nix flakes, which is a feature that allows us to do just that except that it's built into Nix itself. Nix flakes also integrate very well with Git and cloud based Git hosting services like GitHub.

Let's take the example that we have above building a "hello-world" application and convert it to flake.

Alright, let's start. In an empty directory, create the same hello.c file from previous example. Now, instead of using a default.nix file, we're going to use a flake.nix file. You don't have to create this file by hand though; one can bootstrap this using nix flakes init command (note: use the experimenal features flags if prompted by Nix). Let's now fill the contents of the flake.nix file with the following before we break down what's happening here:

{
  description = "My first flake";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-21.11";
  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
      in
      rec {
        defaultPackage = pkgs.stdenv.mkDerivation {
          pname = "hello-world";
          version = "0.1.0";

          src = ./.;

          buildInputs = [ pkgs.gcc ];

          buildPhase = ''
            gcc -o hello hello.c
          '';

          installPhase = ''
            mkdir -p $out/bin
            cp hello $out/bin/
          '';
        };

        defaultApp = {
          type = "app";
          program = "${defaultPackage}/bin/hello";
        };
      }
    );
}

It's entirely a matter of opinion, but I feel that Nix flakes are a lot cleaner than regular Nix expressions. Once you start writing Nix files with more content, having them in flakes will come very handy.

Before I explain further, I'd ask you to try and build and run this flake. You can do this using nix run command. Nix is automatically able to identify that this is a flake and act accordingly.

But let's take some time to break down the content of our flake.nix file and what we did with it.

1. We declare a description for an object. This is simply a metadata field. You don't really have to declare it, but it's certainly nice to have.
2. We declare some inputs. Note that you can also declare inputs using inputs = { ... } where you're nesting fields in an object. However, since we don't really have a big object, we opted to use the . syntax for declaring fields on objects. For our example, we're declaring two inputs - nixpkgs and flake-utils. You're already familiar with nixpkgs from the previous example, and flake-utils is a nice library to work with flakes that we will discuss in a bit.
3. Note that we follow inputs.<input-name>.url syntax to declare these inputs. Flakes have this brilliant ability to use this syntax and be able to parse different URLs and fetch the dependencies. For example, we're just referencing a GitHub repository from which Nix is supposed to be able to fetch our dependency. This feels a lot more declarative than our default.nix file where we had to download a Tar archive, validate it using the hash, and then use the contents. When you're finished with this post (and I recommend that you do that first), you can look at the Nix Wiki for flakes and afterwards flake reference documentation to learn more about this syntax.
4. Once we have the inputs sorted, we declare an outputs object, which is a function containing the list of inputs that we've specified. Note that this is no different from any other Nix function. Nix flakes is just a utility that knows how to read this Nix file and appropriately parse the inputs to the outputs function before executing it.
5. When you ran nix run, you must have noticed that a flake.lock file was created (if you didn't run nix flake init already), or simply updated it it already existed. If you look at the lock file, you will see the hashes of dependencies and pinned versions as well as information on how Nix downloaded them. This is where flakes are different from regular Nix. Instead of specifying all these details yourself, you are able to rely on Nix to pin these dependencies such that the dependencies that you've tested your flake with are the same dependencies others use for using your flake. If newer versions of dependencies are available in the future, you can simply update them using the nix flake update command. Go ahead, try running the update command right away. Likely nothing has changed and Nix will not update anything.
6. Let's now look at the body of the outputs function. This is where things get very interesting. The first thing that we use is to call eachDefaultSystem function from flake-utils.lib module. Remember that we were using builtins.currentSystem to detect the current system and the build our derivation for that system in our previous example? Because flakes have to "pure" in the sense that everything we do is reproducible and that the generated lock file has all the information regarding the required dependencies and their hashes, we can't directly use builtins.currentSystem. This is because the system can changed, which may change the dependencies that are used and makes the whole process impure. What flake-utils does here is that it provides us with functions and modules for all the possible systems and then using the system it let's us declare outputs for all systems that it knows of. We can, of course, override this but we're not going to discuss that right now. Instead you should read the flake-utils documentation to learn more about it.
7. Now that we have a possible system information, we declare pkgs object where we use nixpkgs.legacyPackages for a particular system. We then go ahead and create a defaultPackage object and assign our derivation to it. By now, you already know how derivations are built so it is probably easy to understand that we're just assigning it to a variable.
8. Once we have a derivation, we create a defaultApp object and assign it type and program field. Remember that we were storing our binary in $out/bin directory. That is exactly where we point our defaultApp.program to. As I've pointed out before, Nix knows that a flake is a specific way of writing our configuration that allows Nix to offer certain extra features like generating a flake.nix file or directly running the program defined in the defaultApp output.

What does our flake contain?

Once you start exploring, you will realise that not all flakes declare defaultApp output. That's often because they are not runnable apps. The flake-utils is an example of a flake that outputs functions like lib.eachDefaultSystem. But does this mean that for each runnable app you have to store your program in the defaultApp output? What if you have a flake that happens to have multiple programs? Or multiple configuratiosn for a program? For example, what if you want to run a program but just pass in different configuration to it? When we do nix run, it picks up the program in defaultApp automatically, but one can also run anything else defined in the flake file. You simply have to refer to it using .#<output-name>. For example, if your flake exposes a program in an output called my-test-program, one could run nix run .#my-test-program. You can also nest your programs. We will later discuss cases where this is very useful.

Another nice thing about flakes is that you can simply run programs in flakes directly by calling their path. In our example above, we called .#my-test-program because the flake is in the current directory represented by the .. But let's say that the flake is in a GitHub repository, you can also run it directly using the URL syntax that we used for declaring inputs. As we will later discuss, cargo2nix is a project that uses flakes. One can run the binary exposed by cargo2nix by calling nix run github:cargo2nix/cargo2nix command.

So far we've learned about working with Nix and flakes in Nix. You must have noticed that while flakes are a nice way of writing derivations, they are not the only way of writing Nix configurations. Unfortunately, you will find that the web is full of software fragmented between flakes and regular Nix configurations. In a later post we will discuss combining the use of flakes with the rest of the ecosystem and vice versa.

Another intereting tool to work with flakes is the Nix CLI itself. If you run nix flake show you will see that it lists an entire tree of everything that the flake contains. If you run the show subcommand on the flake that you've just written, you will notice that there is a defaultApp.<system-arch>, which is your application. You'll also see the defaultPackage derivation that you wrote listed with possible system arch. The different systems are provided by flake-utils.

Reproducible Development Environments

So we know how to write packages and compile them using Nix. But what if we want to debug these packages? Let's say we have a Python package and we would like to use Python REPL and use it to debug our source code. You can, of course, install Python on your system but then you can't ensure that everyone else in your team is using the same version of Python that you are and that the CI is running tests using the same Python version that you used on your laptop.

We can solve this problem by creating a dev-shell that contains all our dependencies managed by Nix and pinned by flakes. Then anyone else developing on the same project can run tests or work with your project using the exact same dependencies that you used.

Let's modify our flake.nix file just a tiny bit:

{
# ...
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
      in
      let
        deps = [ pkgs.gcc ];
      in
      rec {
        defaultPackage = pkgs.stdenv.mkDerivation {
          # ...
          buildInputs = deps;
          # ...
          passthru = {
            devShell = pkgs.mkShell {
              buildInputs = deps;
            };
          };
        };
      }
    );
}

Notice that we've added a passthru field in our defaultPackage object and created a devShell subfield with buildInputs. Now try running nix develop. You will notice that Nix has dropped you into a new shell where the packages defined in the buildInputs are available. You can now call these packages and since they are the same as the ones used in the derivation. You can run the same code on any other machine with Nix and it will drop you into the same shell.

This is great, but can we do more? What if we wanted multiple shells? What if your project contains contains both front-end and back-end code and you want front-end engineers to be able to load a shell with the tools they need and back-end engineers to load the tools they need, or both? Fortunately, that is very easy to do as well. In fact it's so easy that I'd like you to ignore our previous example entirley and use the method that we will discuss now.

{

  # ...
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
      in
      rec {
        defaultPackage = pkgs.stdenv.mkDerivation {
          # <code from our original derivation before we discussed dev shells>
        };

        defaultApp = {
          # <code from our original derivation before we discussed dev shells>
        };

        devShells = {
          default = pkgs.mkShell {
            buildInputs = [
              pkgs.gcc
            ];

            shellHook = ''
              echo "Entering C development environment"
            '';
          };

          python = pkgs.mkShell {
            buildInputs = [
              pkgs.python3
            ];

            shellHook = ''
              echo "Entering Python development environment"
            '';
          };
        };
      }
    );
}

We now have a top-level devShells attribute in our outputs that allows us to define a default shell and a python shell in the event that somebody decides to do something with Python instead of gcc. You can create as many shells as you like. We now even have a shellHook attribute in our shell that allows you to execute a script when somebody enters a shell. This is particularly useful for printing a message informing user which shell they are in (in case shells are very similar) or for setting environment variables.

Now when you do nix develop, Nix will automatically drop you into the default shell. And if you call nix develop .#python, it will drop you into the python shell.

Curious readers might notice that while nix flake run .#package runs the package defined at the top-level in the outputs, nix develop .#python actually does not run a top level python attribute in the outputs but instead runs the devShells.python attribute from outputs. At this point, I'd like to remind you that flakes are just a special way to indicate to Nix the format in which the configuration is written and Nix provides us with certain features to use that configuration. The shells in Nix have their own format when working with or without flakes, and it always accesses the content inside outputs.devShells object of our flake.nix instead of top level outputs.

Automatically Spawning Shell With Direnv

So this is cool, but what if we could automatically load up the entire environment defined in our dev-shell and jump into the shell as soon as we access that directory in our terminal? Fortunately, we have a great tool for that called direnv. Another advantage of using Direnv is that many IDEs and other tools out there also support it so we don't just have access to our reproducible shell in Terminal but also in our IDEs like VSCode or IntelliJ.

We are going to have to install the direnv package. You can do so using nix profile install nixpkgs#direnv command. This will install direnv package and make it available for you to use across your user account on your system.

Once you have direnv installed, you're going to have to configure your shell to use it. For that, you can just modify your shell's configuration (for example, ~/.zshrc or ~/.bashrc) to include the following:

# Replace <your-shell> with the name of your shell. Example: zsh or bash
eval "$(direnv hook <your-shell>)"

Now to setup your project directory to use direnv, you can create a .envrc file and simply add the following to it:

use flake

Now when you do direnv allow, your shell will automatically load the default Nix shell every time you enter the directory. If you'd like to disable it, you can simply run direnv deny and your shell will not automatically use your dev-shell.

In case you'd like to load a shell that's not your default shell (let's say the Python shell we created), you can modify your .envrc file to include a DIRNE_FLAKE_PROFILE variable with the name of the shell that you'd like to use. Here is an example:

export DIRENV_FLAKE_PROFILE="devShell.python"
use flake

That's it! Now you have a reproducible build environment that you can share with others. Note that in order to use reproducible shells, you don't necessairly need to write a derivation for your project. You can completely ignore the defaultApp or any other attribute in outputs and only include devShells attribute.

Flakes in Version Control & A Word Of Caution

Flakes are very good particularly great because they work with Git. Remember that we used github:NixOS/nixpkgs/nixos-21.11 input for nixpkgs? The nixos-21.11 is the branch on GitHub. If you ignored that part, Nix will just use the default branch from Git. This is a very nice feature that lets you split your code in repositories and import branches and/or tags. You can even include ?dir=<dirctory-name> at the end of the URL for a repository that includes multiple Nix configurations stored in multiple directories.

BUT one thing that people often forget is that in order for a flake to properly work, the files in the flake have to be added to Git. If you write a flake that does not work, try adding the files to Git. This is an easy to make mistake that has often made me look like a fool so I wanted to point it out before ending this post.

Closing Thoughts

I was very excited to write this post because this is where things get very interesting. But it was also very hard and took much longer than I expected. I hope this was useful and I hope you had the time to go through everything that I discussed here. It goes without saying that there is still a lot more to the topics that we have discussed than I was able to cover, but I think this should be a good enough start. I plan on writing more posts on Nix where we discuss things like managing your system, managing remote hosts, and even running Nix on MacOS. Hope you're as excited about them as I am.