nix-mariner

NixOS microVM modules for creating development environments that isolated untrusted code from your host.

Built on microvm.nix.

What it does

Provides NixOS modules importable as a flake input.
Creates persistent microVM environments to isolate untrusted code away from the host.
Preconfigured: SSH, Docker, direnv, shared /nix/store, persistent storage, bridge networking, etc.

Imperative and Declarative workflows

The documentation covers both imperative and declarative workflows. Before either, set up the host once: Host setup.

Imperative

Creates VM with microvm -c. The only host NixOS changes are the one-time Host setup.

See Imperative Virtual Machines.

Declarative

VMs defined inside the host’s NixOS configurations with microvm.vms.<name>.

See Declarative Virtual Machines.

Per-VM Customizations

You can change and override microvm.nix and nixos module configurations for each VM. Overrides work the same in both imperative and declarative modes.

See Customizing VMs.

Host setup

In order to use nix-mariner, you need to import microvm.nixosModules.host module and configure the networking options in your nixos system configuration.

You can use the microvm cli tool to create and manage VMs imperatively, instead of declaring them in your NixOS config.

See Preparing a NixOS host for declarative MicroVMs for more information

NixOS microvm.nix module

# Host server flake.nix
{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";

    mariner.url = "github:mksafavi/nix-mariner";
    mariner.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, nixpkgs, mariner }: {
    nixosConfigurations.machine = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        # Include the microvm host module
        mariner.inputs.microvm.nixosModules.host
      ];
    };
  };
}

Alternatively, you could declare microvm directly in your inputs:

# Host server flake.nix
{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";

    microvm.url = "github:microvm-nix/microvm.nix";
    microvm.inputs.nixpkgs.follows = "nixpkgs";

    mariner.url = "github:mksafavi/nix-mariner";
    mariner.inputs.microvm.follows = "microvm";
    mariner.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, nixpkgs, mariner, microvm }: {
    nixosConfigurations.machine = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        # Include the microvm host module
        microvm.nixosModules.host
      ];
    };
  };
}

NixOS networking additions

Adding the following network configuration should be enough to setup the networking.

See A simple network setup for more information

This creates a network bridge that each VM tap connects to. If you change the default bridge address 10.0.0.1, you also need to set the network address accordingly in the virtual machine configuration.

{ config, lib, pkgs, inputs, ... }:
{
  # microvm requires systemd networkd. You can use it alongside NetworkManager without any issues.
  systemd.network.enable = true;

  # DNS on microvm bridge. This assumes you're already using systemd-resolved.
  services.resolved.settings.Resolve = {
    DNSStubListenerExtra = [ "10.0.0.1" ];
  };

  systemd.network.netdevs."br-microvm" = {
    netdevConfig = {
      Name = "br-microvm";
      Kind = "bridge";
    };
  };

  systemd.network.networks."10-br-microvm" = {
    matchConfig.Name = "br-microvm";
    networkConfig = {
      Address = [ "10.0.0.1/24" ];
      IPMasquerade = "ipv4";
      ConfigureWithoutCarrier = true;
    };
  };

  # Attach VM TAPs to the bridge automatically
  systemd.network.networks."10-microvm-tap" = {
    matchConfig.Name = "microvm-*";
    networkConfig.Bridge = "br-microvm";
  };

  # Trust the VM bridge so VMs can reach host DNS / SSH
  networking.firewall.trustedInterfaces = [
    "br-microvm"
  ];

  # NetworkManager shouldn't manage the microvm bridge. Skip if you don't use NetworkManager
  networking.networkmanager.unmanaged = [ "br-microvm" ];

  boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
  boot.kernelModules = [ "vhost_vsock" ];
}

Verify

After a nixos-rebuild switch you should have the following:

ls /dev/kvm                        # exists
ip addr show br-microvm            # has 10.0.0.1/24
ss -lntp | grep ':53'              # listening on 10.0.0.1:53
lsmod | grep vhost_vsock           # module loaded

Imperative Virtual Machines

In the imperative workflow, you define virtual machines under nixosConfigurations.<vm> in a flake, then use the microvm CLI to manage their lifecycle. See Imperative MicroVMs for the upstream option reference.

Define a Virtual Machine in a Flake

Note

Before continuing, make sure you’ve completed the Host Setup.

Add nix-mariner as a flake input, then define a nixosConfigurations.<vm> entry for your virtual machine.

The following flake.nix creates a VM named example:

{
  inputs.mariner.url = "github:mksafavi/nix-mariner";

  outputs =
    { self, mariner }:
    let
      nixpkgs = mariner.inputs.nixpkgs;
    in
    {
      nixosConfigurations.example = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        specialArgs = { inherit nixpkgs; };
        modules = builtins.attrValues mariner.nixosModules ++ [
          {
            mariner.cid = 4; # Unique per-VM CID that sets vsock number and IP address.
            mariner.hostAuthorizedKey = "ssh-ed25519 AAAA... your@host"; # Replace with your ssh public key
          }
        ];
      };
    };
}

Calling microvm -c builds the VM and creates the systemd service for booting it.

sudo microvm -c example -f path:$(pwd)

Verify that the vm is created:

microvm -l

Either start the service:

sudo systemctl start microvm@example.service

Or start it in foreground:

sudo microvm -r example

You can now ssh into it:

ssh vm@vsock%4
# or:
ssh vm@10.0.0.4

See examples/flake.nix for a standalone flake example.

Declarative Virtual Machines

You can declare virtual machines directly in the host’s NixOS configurations by adding microvm.vms.<name> entries in a module.

nixos-rebuild switch then builds, updates, and starts them via systemd. You can’t modify VMs with microvm CLI anymore if you use the declarative workflow.

See Declarative MicroVMs for the upstream option reference.

Declare Multiple Virtual Machines

Note

Before continuing, make sure you’ve completed the Host Setup.

Create a virtualization.nix module and import it into your host setup:

{
  config,
  pkgs,
  mariner,
  nixpkgs,
  ...
}:
{
  microvm.vms = {
    vm-work = {
      specialArgs = { inherit nixpkgs; };
      config =
        { config, pkgs, ... }: # this points to the vm config to access config.mariner
        {
          imports = builtins.attrValues mariner.nixosModules ++ [ ];
          mariner.cid = 4;
          mariner.username = "work";
          mariner.hostAuthorizedKey = "ssh-ed25519 AAAA... your@host";
        };
    };

    vm-test = {
      specialArgs = { inherit nixpkgs; };
      config =
        { config, pkgs, ... }: # this points to the vm config to access config.mariner
        {
          imports = builtins.attrValues mariner.nixosModules ++ [ ];
          mariner.cid = 5;
          mariner.username = "user";
          mariner.hostAuthorizedKey = "ssh-ed25519 AAAA... your@host";
        };
    };
  };
}

Import the modules into your host configurations:

# In another module:
  imports = [
    modules/virtualization.nix
  ];

# Or in host flake:
  nixosConfigurations.machine = nixpkgs.lib.nixosSystem {
    #...
    modules = [
      modules/virtualization.nix
      # ...
    ];
  };

nixos-rebuild switch should build each VM and start the systemd services microvm@<name>.service

Now you can ssh into them if the services are running:

ssh work@vsock%4
ssh user@vsock%5

See microvm.autostart for starting the VMs automatically at host boot.

Per-VM Customizations

You can change and override microvm.nix and nixos module configurations for each VM. Overrides work the same in both imperative and declarative modes.

For more information, see Mariner options and microvm.nix Options.

    vm = {
      specialArgs = { inherit nixpkgs; };
      config =
        { config, pkgs, ... }: # this points to the vm config to access config.mariner
        {
          imports = builtins.attrValues mariner.nixosModules ++ [ ];
          mariner.cid = 5;
          # Change user name:
          mariner.username = "user";
          mariner.hostAuthorizedKey = "ssh-ed25519 AAAA... your@host";
          # Set VM resources:
          microvm = {
            vcpu = 4;
            mem = 8 * 1024;
          };
          # Share a host directory with the VM:
          microvm.shares = [{
            source = "/home/you/work";
            mountPoint = "/work";
            tag = "work";
            proto = "virtiofs";
          }];

          # Install more packages:
          users.users.${config.mariner.username}.packages = with pkgs; [ btop ];
        };
    };