Skip to content
NAILS on GitHub
These docs describe NAILS 0.3, published so far as the release candidate v0.3.0-rc.3. They do not apply to v0.2.0, the latest stable release. Download and verify NAILS 0.3

Troubleshooting

In the hidden session, the way out is always deactivate, then emergency, then a power-off. A power-off is always safe, because nothing of the session is installed on the decoy and everything NAILS puts in /run is gone at power-off. On the decoy, the cleanup is nails recover, run from a shell with history off (set +o history). The command reference lists every exit code.

Read the hint marked with an arrow under each failing check. It names the fix. check runs the same pre-flight checks as activate, changes nothing and exits with code 4 while a check fails. Some of the checks that refuse activation:

  • platform fails on a host NAILS does not support, such as Fedora, Arch or a Debian host without the merged /usr. A later Debian or Ubuntu release that NAILS has not been tested on only warns. See Requirements.
  • hidden-volume fails when the hidden root is not a volume of its own, as the next section describes.
  • backend-entries fails while a line in the decoy’s /etc/crypttab or /etc/fstab opens or mounts the hidden volume or a device it is stored on. No flag overrides it. Remove the line long before the next use and open the volume by hand each time.
  • decoy-persistence fails on a persistent journal or an active rsyslog, syslog-ng or auditd. Make the journal volatile and remove or disable those daemons, as Prepare the decoy describes. --accept-persistent-decoy accepts the risk instead, and the decoy’s journal then records the soft-reboots and the activation command.
  • swap fails on active disk swap. Only RAM-backed zram is allowed, and --accept-persistent-decoy does not override this check.
  • nixos-build-target fails on a NixOS decoy whose /etc/nixos holds a flake when no hidden flake is given, because activation never builds the decoy’s own flake. The guide to your first session shows how to write a hidden flake.
  • space fails when the hidden volume has less free space than minimum_space_mb.

The hidden root “is not a volume of its own”

Section titled “The hidden root “is not a volume of its own””

init, check and activate, like the other commands that write to the hidden volume, refuse a hidden root that is not storage of its own before they write or probe anything, with exit code 4. The message reads “<root> is not a volume of its own”, names the reason and ends with “Nothing was written.” A folder, a btrfs subvolume or bind mount of a filesystem the decoy uses, a volume stacked on the decoy’s storage or swap and an unencrypted image file on a decoy filesystem are all refused, and so is a root with a second filesystem mounted over it. --no-preflight does not skip this. An encrypted container file on a decoy filesystem is accepted with a warning. That filesystem’s journal and copy-on-write snapshots can keep old copies of the parts of the container that changed, so a whole device or a partition the decoy does not use is better.

Move the hidden volume to storage of its own, such as an external drive prepared on another computer. The hidden volume page lists what the attach gate refuses. When the command could not read /etc/crypttab or a loop device’s record (storage_unjudged), run it with sudo. --allow-decoy-storage turns the refusal of the decoy’s own storage into a warning, for storage you know the decoy does not use. It never accepts a folder, a covered root or storage that cannot be judged.

Run every command with sudo and the binary’s full path. Without root, activate, deactivate, emergency and recover refuse before they write anything, with exit code 5. A command that cannot read the configuration, which init writes readable by root only, exits with code 3. Both print the full sudo line to run instead.

activate, deactivate and reset-upper ask for confirmation and refuse without a terminal, with the error code needs_confirmation. Add -y to every script, key binding or unit that runs them. emergency never asks.

An unknown key, a value of the wrong type or a syntax error stops activate, check, status and most other commands with exit code 3 and a one-line message that names the file, the line and the column. Because activate stops too, no hidden session can start to fix it in. Correct the file on the separate machine and copy it onto the hidden volume, as Configuration describes, and never edit it on the decoy. deactivate and recover log a warning and carry on with defaults, and emergency reads no configuration at all.

If the activation was killed, crashed or failed after it built /run/nextroot, nails status reports an interrupted entry, exits with code 7 and says to run nails recover. Run it on the decoy. Whatever the build wrote stays in the upper layer on the hidden volume, and the decoy is untouched. If the kernel killed NAILS for lack of memory during activation, treat it the same way. Ctrl+C during activation rolls back by itself.

recover also closes the hidden volume’s encryption mapping when nothing uses it. If the hidden volume is still mounted, it leaves the mapping open and says so. Unmount and close the volume yourself when you are done.

The first entry copies the hidden system’s whole closure onto the hidden volume, which can take several GB and minutes on a slow or encrypted backend. Activation reports the megabytes copied every 15 seconds. Later entries copy only the paths that are new. Activation is refused before it copies anything when the hidden volume lacks the space.

Run nails emergency. It kills every hidden program at once and takes the same way back. If the way back cannot be prepared, it powers off instead. deactivate refuses, for example, when a disk is mounted over the hidden volume, or when the way back cannot make the decoy’s mounts again as the decoy had them.

Switch to a text console with Ctrl+Alt+F2, log in and run nails emergency. If nothing responds, power off. A stuck service can delay an orderly deactivate by up to DefaultTimeoutStopSec, 90 seconds by default. emergency does not wait for it.

If the drive was pulled or reports I/O errors, the session starts failing, and the backend watcher alerts you. Run the RAM copy that activation left in /run: sudo /run/overlay-runtime/control emergency. It needs nothing from the hidden volume. Power off if it does not respond. Backend watcher and disks explains how to let the watcher run the emergency by itself.

A disk mounted over the hidden root, or over one of its parents, takes every file saved under the hidden root from then on. The watcher raises “Disk mounted over a folder in use”, naming the disk and the sudo umount command for its mount point, and nails status shows it under the hidden volume. NAILS itself writes nothing there meanwhile. Until you unmount the disk, deactivate refuses, and emergency powers off if the mounts the way back needs are covered too.

Copy-up or a write in the session fails. Free space inside the session, for example with nix-collect-garbage -d and by clearing caches, or deactivate and run reset-upper on a large path from the decoy. Never delete files out of the upper layer or its work directory by hand. df -h /mnt/data shows the free space.

Do not suspend during a session. Suspend keeps the whole hidden session in RAM, including the open encryption mapping and its key, behind the hidden session’s lock screen. Set the decoy’s lid switch to lock and choose a sleep policy. block keeps the host awake for the session, a closed lid included, and teardown powers off when the host is about to sleep.

The decoy boots normally, because nothing of the session is installed on it and /run is gone. If nails status reports a recorded state that does not match, run nails recover once. What the session wrote stays in the upper layer on the hidden volume.

On Ubuntu 26.04, chrony (time sync) and CUPS (printing) fail in the hidden session, because AppArmor blocks their messages to systemd and to the journal, whose sockets were set up before the switch. The clock is not synchronized and printing times out. Other services that AppArmor confines may fail the same way. After the way back, the decoy lists chrony as failed until its next full reboot, which hints that a soft-reboot happened. Reboot fully after a session to clear it. This is a known issue, and the fault lies outside NAILS. A plain systemctl soft-reboot into another root, without NAILS, breaks chrony the same way.

If nails recover cannot run on the decoy, for example because the hidden volume is gone or the binary does not start, remove what an interrupted activation left behind yourself, in a root shell:

Terminal window
set +o history
# 1. Detach the next root and everything below it.
umount -R /run/nextroot
# 2. Remove the session plan and RAM copy, the transition drop-in and the runtime guards (blk-availability's stop, NixOS's static addresses and the D-Bus units).
rm -rf /run/overlay-runtime
rm -rf /run/systemd/system/systemd-soft-reboot.service.d
rm -f /run/systemd/system/blk-availability.service.d/zz-overlay-runtime.conf
rm -f /run/systemd/system/network-addresses-.service.d/zz-overlay-runtime.conf
rm -f /run/systemd/system/dbus.service.d/zz-overlay-runtime.conf
rm -f /run/systemd/system/dbus.socket.d/zz-overlay-runtime.conf
systemctl daemon-reload
# 3. Confirm nothing is left: findmnt prints no rows and ls reports "No such file or directory".
findmnt -R /run/nextroot
ls /run/overlay-runtime

Then close the hidden volume as usual, with umount and cryptsetup close. If cryptsetup close reports the device busy, a service that started during the activation holds a copy of the next root’s mounts in a mount namespace of its own. grep -l /run/nextroot /proc/*/mountinfo names its processes. Restart that service, or power off. A power-off instead of the steps above does the same, because everything they remove lives in /run.

If the activation built a hidden NixOS system, or its switch failed, power off instead. nails recover also gives the kernel back the decoy’s module loader, hotplug helper and firmware path, which the hidden system’s activation pointed at its store, and removes the session’s unit masks in /run/systemd/system and /run/systemd/system.control. Without that, the decoy loads no module and no firmware, and its swap and boot-loader units stay masked. By hand, only a power-off undoes this.

Do not edit or delete state.json by hand. nails recover resets it once the hidden volume is back.

Issues on GitHub are public and link your account to NAILS. Consider a throwaway account, and never post nails status or nails verify output, real paths, host names, user names or volume labels. Replace them with placeholders. Support is best-effort while NAILS is alpha.

  • Questions: read the FAQ and these docs, and search the existing issues first.
  • Bugs: GitHub issues, with nails --version, your distribution and its release, and the steps that reproduce the problem.
  • Security issues: never in a public issue. See Security.