Using macOS as a Darwin Build Machine for Nix Caching

At work we are issued a very fast Apple MacBook Pro (mine is an M4 with 64GB RAM) and we take that fast machine and install Microsoft Defender on it, which takes that fast computer and makes it wade through a swamp. This post isn’t about the pain that I go through daily to try to develop directly on my Mac (that should be a separate series of posts) but today I want to talk about how I have greatly improved my nix-darwin switch speeds.

CI

I used to run my nix builds in my Forgejo Actions runner, however I ran into a bug in how nix flake check is evaluated that causes it to continuously consume more and more memory until it either finishes or is OOM-killed (the latter being the result for me always). My flake is rather large so this was immediately an issue for me.

Enter nixbot which uses the much better suited nix-eval-jobs to evaluate and divide up the work. I could’ve done this manually but nixbot does a great job of crafting a useful GUI and webhook integrations for Gitea/Forgejo.

Nixbot does a great job of using native nix tools where possible, so instead of installing agents on all hosts you want to use for builds, you configure nix.buildMachines to handle distributing builds to available machines with the specific attributes you need. This came in handy as I had some extra compute lying around and I was able to set up remote building from the Nixbot host so that jobs would complete faster.

Architectures

With a CI pipeline that is able to fully load and evaluate my flake set up, I was able to build all my hosts. Well not all my hosts because the only build infrastructure I had configured was x86_64-linux. So most of my hosts. This is still useful: my VMs and bare-metal instances all benefit from the pre-built packages, nixosTests, and just generally having things stored in my private Cachix cache. This was itself a huge quality of life improvement and I could stop here. In fact I did for a few weeks.

aarch64-linux

I do a lot of work on my work laptop in a VMware Fusion virtual machine running NixOS. This affords me multiple benefits such as:

  • speedy builds: defender doesn’t impact disk access in the VM
  • consistent tooling: no more wrangling dotfiles in a GitHub Codespace where each different repo has a different setup
  • offline work capabilities: which I do to limit my ADHD tendencies to “check just one thing” and then be off on a tangent for 7 hours

Since its configuration is managed with my flake, I get all my setup for free(ish), but I also have to build a lot of things during a NixOS build as they’re not cached from being built by my CI. To solve this I added aarch64-linux to boot.binfmt.emulatedSystems on my build host, and now it can build packages for that architecture. Easy-peasy.

Two architectures down, one to go.

aarch64-darwin

This architecture was the hardest for me to even come up with a way to build. I knew I couldn’t run a remote builder on my work laptop to execute code/builds from my infrastructure. That would almost certainly create some sort of emergency ticket in a security queue very quickly.

I do have a MacBook Air M1 that we rarely use at home. It sits on a shelf attached to a dock 99.9% of its life except when my wife needs a non-work laptop. This should be easy, I thought, but nothing that seems like it will be easy ever is.

Like every other machine in my life, our MacBook Air is managed with Nix. nix-darwin does a great job at doing many of the things NixOS can do, but mapped to macOS paradigms. I thought I could just use users.users.<name>, which nix-darwin supports, and create a user like you would in NixOS, but not so! Maybe I’m “holding it wrong” [sorry Steve Jobs], but this didn’t work, so I opened up the Settings app and created the user account.

I tried to SSH in, but it failed. Well, sort of. Adding -vvvvv (just keep adding v’s until ssh tells you what you need to know is my policy) showed that the host was accepting connections and even accepting the key that I had configured with users.users.<name>.openssh.authorizedKeys.keys, but then just terminated the connection. After a bunch of poking around in the macOS Settings app, I found that Remote Login had a specific allowed-users setting for non-admins to be able to SSH to the host. I added my remote build user and tada, I was able to SSH in!

I added the host to the nix.buildMachines setting on my build host and had it run a test, and it built. So I closed the lid and left it plugged in to the dock for a full build test. This immediately failed because the Air went to sleep. I changed some power-saving settings, tried again, and got the same result. I finally stumbled across the undocumented pmset feature described in a blog post: sudo pmset -a disablesleep 1. Now this is normally bad because when it’s off power and just sitting idle, it won’t go to sleep. That’s a problem I’ll deal with later since it spends most of its life in the cradle, plugged in to a dock and being bored.

I closed it up and ran another test and this time it didn’t fail! The machine isn’t super fast, but that’s ok because once it has run a build to completion the cache will be populated and most packages don’t change frequently (until I update my flake inputs). Now my work machine can just download the derivations from the cache instead of trying to build them itself.

Lessons Learned

  1. buildMachines is a super powerful feature; if you have any excess capacity, it’s worth allocating it. My laptop now uses my desktop for its builds and can even reach the build machines in Hetzner. I’ve set the speed factor on the local desktop significantly higher than the Hetzner hosts for my laptop, so most of the time my builds will stay on my local network.
  2. macOS continues to be Linux-ish but with kid gloves and weird caveats. I don’t really like how much Apple prevents me from using a machine the way I’d like, but that’s kinda their thing: you have to use it the way they want you to.
  3. Modern Problems [Defender] Require Modern Solutions. Come up with a weird way to accomplish a task and see if it works. I never thought about using this poor, under-resourced laptop for anything but web browsing. It has exceeded my expectations by far.