Installing NixOS on OVH

I just recently set up an OVH server for some PDF APIs that I was working with due to their memory size. Nix is not a commonly installed OS for servers so I needed to figure out how I would install this on my own.

I tried the install script thats on github but it needs customization and it felt a little overwhelming to just dive in and edit the script as it is. I’ve never been that good at shell scripting so this will be interesting to break down the various commands.

Why do you want to use some weird functional OS?

I’ve been playing around with nix for quite some time with my vim files setup and i’ve found it to be quite nice. I really like the idea that everything is “frozen” in the system and packages are not just installed in some apocryphal manner.

I’m going through an experience where I am installing a service that did not have a list of packages it needed and I have to reverse engineer what has been installed on the server after 15 years. NixOS hopefully solves this situation.

I will declaratively lay out all the packages I need and all my problems will be solved? right? right?

Installing Nix onto the rescue system

I booted the server into rescue mode and started with that. We need to first start off by installing nix onto the rescue mode installation. All we are doing here is installing onto a chunk of ram and it is not permanently installing anything yet.

Because we are installing it onto the rescue system we need disable the sandbox mode or we will get the error: cannot pivot old root directory.

$ mkdir -p /etc/nix
$ echo "sandbox = false" >>/etc/nix/nix.conf

We also run the regular install with --daemon because again we are in the rescue environment and its very different than a regular system install.

$ sh <(curl -L https://nixos.org/nix/install) --daemon
$ . /root/.nix-profile/etc/profile.d/nix.sh

---

Add some channels to the NixOS

We need to install some sources of packages, kind of like doing apt-get and apt-get update.

$ nix-channel --add https://nixos.org/channels/nixos-23.05 nixpkgs
$ nix-channel --update

23.05 is the version number and can be changed to the version that you prefer.

---

Install NixOS install tools

In effect this is the main installer, we will first install it and then run it later.

$ nix-env -f '<nixpkgs>' -iA nixos-install-tools

---

Partitioning the drive

When we first start off in rescue mode we should have no drives mounted and software raid mdadm is turned off. We need to partition the drive for nix to be installed.

First we set the partitioning table to GUID Partition Table (GPT) with parted.

parted --script /dev/sda mklabel gpt

parted --script /dev/sdb mklabel gpt

We’re going to partition the drives into 3 pieces:

• EFI System Partition
• OS Partition
• Data Partition

Which leaves us with this layout:

Start	End	Note
1MB	551MB	fat32 ESP
551MB	500GB	OS Partition
551MB	100%1	OS Partition

We run the following commands to perform the partition:

$ parted --script --align optimal /dev/sda -- mklabel gpt mkpart 'ESP-partition0' fat32 1MB 551MB set 1 esp on  mkpart 'OS-partition0' 551MB 500GB mkpart 'data-partition0' 500GB '100%'

$ parted --script --align optimal /dev/sdb -- mklabel gpt mkpart 'ESP-partition1' fat32 1MB 551MB set 1 esp on  mkpart 'OS-partition1' 551MB 500GB mkpart 'data-partition1' 500GB '100%'

This existed in the original script but i’m not sure that we need to run this.

# Wipe any previous RAID signatures
$ mdadm --zero-superblock /dev/sda2
$ mdadm --zero-superblock /dev/sda3
$ mdadm --zero-superblock /dev/sdb2
$ mdadm --zero-superblock /dev/sdb3

We will then create parititions for the operating system and the data drives

$ mdadm --create --run --verbose /dev/md/root0 --level=1 --raid-devices=2 --name=root0 /dev/sda2 /dev/sdb2

$ mdadm --create --run --verbose /dev/md/data0 --level=1 --raid-devices=2 --name=data0 /dev/sda3 /dev/sdb3

Delete all of the data on the drives!

Wipe filesystem signatures that might be on the RAID from some possibly existing older use of the disks (RAID creation does not do that).²

$ wipefs -a /dev/md/root0
$ wipefs -a /dev/md/data0

We temporarily disable RAID mode so that we can have higher performance in rescue mode. We do not need recovery mode as we have erased all the information on the drive already.

Disable RAID recovery. We don’t want this to slow down machine provisioning in the rescue mode. It can run in normal operation after reboot.

echo 0 > /proc/sys/dev/raid/speed_limit_max

Now we will create the filesystems for the partitions that we created

# Filesystems (-F to not ask on preexisting FS)
mkfs.fat -F 32 -n esp0 /dev/disk/by-partlabel/ESP-partition0
mkfs.fat -F 32 -n esp1 /dev/disk/by-partlabel/ESP-partition1
mkfs.ext4 -F -L root /dev/md/root0

Ok now it is time to mount our drives:

# Creating file systems changes their UUIDs.
# Trigger udev so that the entries in /dev/disk/by-uuid get refreshed.
# `nixos-generate-config` depends on those being up-to-date.
# See https://github.com/NixOS/nixpkgs/issues/62444
$ udevadm trigger

# Wait for FS labels to appear
$ udevadm settle --timeout=5 --exit-if-exists=/dev/disk/by-label/root

# NixOS pre-installation mounts
# Mount target root partition
$ mount /dev/disk/by-label/root /mnt


# Mount efivars unless already mounted
# (OVH rescue doesn't have them by default and the NixOS installer needs this)
mount | grep efivars || mount -t efivarfs efivarfs /sys/firmware/efi/efivars

# Mount our ESP partitions
mkdir -p /mnt/boot/ESP0
mkdir -p /mnt/boot/ESP1

mount /dev/disk/by-label/esp0 /mnt/boot/ESP0
mount /dev/disk/by-label/esp1 /mnt/boot/ESP1

Install Nix

Generate your NixOS configuration:

$ `which nixos-generate-config` --root /mnt

# On the OVH rescue mode, the default Internet interface is called `eth0`.
# Find what its name will be under NixOS, which uses stable interface names.
# See https://major.io/2015/08/21/understanding-systemds-predictable-network-device-names/#comment-545626
$ INTERFACE=$(udevadm info -e | grep -A 11 ^P.*eth0 | grep -o -E 'ID_NET_NAME_ONBOARD=\w+' | cut -d= -f2)
$ echo "Determined INTERFACE as $INTERFACE"

$ IP_V4=$(ip route get 8.8.8.8 | head -1 | cut -d' ' -f8)
$ echo "Determined IP_V4 as $IP_V4"

# From https://stackoverflow.com/questions/1204629/how-do-i-get-the-default-gateway-in-linux-given-the-destination/15973156#15973156
$ read _ _ DEFAULT_GATEWAY _ < <(ip route list match 0/0); echo "$DEFAULT_GATEWAY"
$ echo "Determined DEFAULT_GATEWAY as $DEFAULT_GATEWAY"


# Generate `configuration.nix`. Note that we splice in shell variables.
$ cat > /mnt/etc/nixos/configuration.nix <<EOF
{ config, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  # Use GRUB2 as the EFI boot loader.
  # We don't use systemd-boot because then
  # * we can't use boot.loader.grub.mirroredBoots to mirror the ESP over multiple disks
  # * we can't put /boot on the same partition as /
  #   (boot.loader.efi.efiSysMountPoint = "/boot/EFI" apparently does not have
  #   the desired outcome then, just puts all of /boot under /boot/EFI instead)
  boot.loader.systemd-boot.enable = false;
  boot.loader.grub = {
    enable = true;
    efiSupport = true;
    mirroredBoots = [
      { devices = [ "nodev" ]; path = "/boot/ESP0"; }
      { devices = [ "nodev" ]; path = "/boot/ESP1"; }
    ];
  };

  boot.loader.efi.canTouchEfiVariables = true;

  # Don't put NixOS kernels, initrds etc. on the ESP, because
  # the ESP is not RAID1ed.
  # Mount the ESP at /boot/efi instead of the default /boot so that
  # boot is just on the / partition.
  boot.loader.efi.efiSysMountPoint = "/boot/EFI";

  environment.etc."mdadm.conf".text = ''
  '';

  # The RAIDs are assembled in stage1, so we need to make the config
  # available there.
  boot.initrd.mdadmConf = config.environment.etc."mdadm.conf".text;

  # Network (OVH uses static IP assignments, no DHCP)
  networking.useDHCP = false;
  networking.interfaces."$INTERFACE".ipv4.addresses = [
    {
      address = "$IP_V4";
      prefixLength = 24;
    }
  ];
  networking.defaultGateway = "$DEFAULT_GATEWAY";
  networking.nameservers = [ "8.8.8.8" ];

  # Initial empty root password for easy login:
  users.users.root.initialHashedPassword = "";
  services.openssh.permitRootLogin = "prohibit-password";

  users.users.root.openssh.authorizedKeys.keys = [
    # Replace this by your pubkey!
    "ssh-ed25519 AAAAC3... bradford@bradford.com "
  ];

  services.openssh.enable = true;

  # This value determines the NixOS release with which your system is to be
  # compatible, in order to avoid breaking some software such as database
  # servers. You should change this only after NixOS release notes say you
  # should.
  system.stateVersion = "23.05"; # Did you read the comment?
}
EOF

$ `which nixos-install` --root /mnt