Nixing the Old for Good: Exploring New Ways with NixOS
Greetings, fellow tech enthusiasts, it’s time for an update from the star ship No Agenda’s engine room! As the designated tech honcho for No Agenda I’ve been dedicating a lot of time over the past year to researching the best possible successor (the fifth incarnation!) to our current infrastructure. Our current Linux distribution just isn’t cutting it anymore and the hardware’s getting old – changes need to be made. But where to begin? Should I stick with what I know (Debian, Gentoo, Funtoo), or venture out into uncharted territory?
Two of the biggest challenges in the current situation turn out to be updating and rebooting the systems without incurring service interruptions. Updating has become an increasingly major hassle over the last couple of years, and when one of our servers recently experienced some heating issues and required an immediate intervention, it took too much manual effort to get everything back up and running smoothly. Unplanned reboots can be really frustrating, so right now I want to avoid them as much as possible. In any new setup, a reboot has to bring up all of the systems in a fully functional working order.
After exploring various options, one of the distributions I was intrigued by was NixOS, with its declarative configuration and reproducible rebuilds. When designed properly it would allow me to set up an entire system in a set of configuration files and duplicate that to multiple systems; not only making updates and reboots easier to manage but also making it easy to replace or add new hardware.
However, taking this road also meant that I needed to learn a lot of things from scratch. Was I really going to do this? How should I prepare? Perhaps I should just dive into it and install NixOS on my primary laptop and force myself to learn the ropes from scratch, I figured. And that’s exactly what I did. For the last seven or eight months I’ve been driving NixOS, first on my laptop, then also on my desktop, and in January I made a couple of virtual machines that would allow me to study how to run NixOS on servers.
At the beginning of March I felt ready to install NixOS to our development server. It was quite a lot of work to work out the basics and get it to function well enough, at which time I started to make use of ChatGPT. I’ve encountered a variety of technical issues while working on this project, ranging from configuring nftables to setting CPU affinity for systemd services. Thanks to ChatGPT, I’ve been able to navigate these and other challenges a little bit better. On many instances it was not entirely accurate, but at least it would allow me to find my way to solutions a little bit quicker than I otherwise would have. And if I still needed help, the very helpful and friendly folks on the NixOS Discord were there to help me figure things out. Shout-out to you guys.
One of the highlights of my experience has been learning how to configure networking in NixOS properly, to set up multiple IPv4 and IPv6 addresses, as well as gateways and additional routes. Another valuable lesson I learned was about how to compile software with additional options, and use that software in LXD/LXC containers and nixos-containers. I also figured out how to set up virtual machines with libvirt. I never used that before because I used to run my virtual machines directly with qemu.
Two months in as of the time of writing, I have basic services and systems in a functional state, and it’s been a very humbling and educational journey so far. I’m looking forward to the coming months; it’s a lot of fun to learn how to work with NixOS. I still have a lot to learn about NixOS but I’m very excited to continue exploring and in time replace our old engines with newer ones. Without any end user being able to tell the difference!