Troubleshooting
In the hidden session, the way out is always deactivate, then emergency,
then a power-off. A power-off is always safe, because nothing of the session is
installed on the decoy and everything NAILS puts in /run is gone at power-off.
On the decoy, the cleanup is nails recover, run from a shell with history off
(set +o history). The command reference lists
every exit code.
When NAILS refuses
Section titled “When NAILS refuses”nails check fails
Section titled “nails check fails”Read the hint marked with an arrow under each failing check. It names the fix.
check runs the same pre-flight checks as activate, changes nothing and exits
with code 4 while a check fails. Some of the checks that refuse activation:
platformfails on a host NAILS does not support, such as Fedora, Arch or a Debian host without the merged/usr. A later Debian or Ubuntu release that NAILS has not been tested on only warns. See Requirements.hidden-volumefails when the hidden root is not a volume of its own, as the next section describes.backend-entriesfails while a line in the decoy’s/etc/crypttabor/etc/fstabopens or mounts the hidden volume or a device it is stored on. No flag overrides it. Remove the line long before the next use and open the volume by hand each time.decoy-persistencefails on a persistent journal or an active rsyslog, syslog-ng or auditd. Make the journal volatile and remove or disable those daemons, as Prepare the decoy describes.--accept-persistent-decoyaccepts the risk instead, and the decoy’s journal then records the soft-reboots and the activation command.swapfails on active disk swap. Only RAM-backed zram is allowed, and--accept-persistent-decoydoes not override this check.nixos-build-targetfails on a NixOS decoy whose/etc/nixosholds a flake when no hidden flake is given, because activation never builds the decoy’s own flake. The guide to your first session shows how to write a hidden flake.spacefails when the hidden volume has less free space thanminimum_space_mb.
The hidden root “is not a volume of its own”
Section titled “The hidden root “is not a volume of its own””init, check and activate, like the other commands that write to the hidden
volume, refuse a hidden root that is not storage of its own before they write or
probe anything, with exit code 4. The message reads “<root> is not a volume of
its own”, names the reason and ends with “Nothing was written.” A folder, a
btrfs subvolume or bind mount of a filesystem the decoy uses, a volume stacked
on the decoy’s storage or swap and an unencrypted image file on a decoy
filesystem are all refused, and so is a root with a second filesystem mounted
over it. --no-preflight does not skip this. An encrypted container file on a
decoy filesystem is accepted with a warning. That filesystem’s journal and
copy-on-write snapshots can keep old copies of the parts of the container that
changed, so a whole device or a partition the decoy does not use is better.
Move the hidden volume to storage of its own, such as an external drive prepared
on another computer. The hidden volume page lists
what the attach gate refuses.
When the command could not read /etc/crypttab or a loop device’s record
(storage_unjudged), run it with sudo. --allow-decoy-storage turns the
refusal of the decoy’s own storage into a warning, for storage you know the
decoy does not use. It never accepts a folder, a covered root or storage that
cannot be judged.
A command asks for root
Section titled “A command asks for root”Run every command with sudo and the binary’s full path. Without root,
activate, deactivate, emergency and recover refuse before they write
anything, with exit code 5. A command that cannot read the configuration, which
init writes readable by root only, exits with code 3. Both print the full
sudo line to run instead.
A script gets exit code 2
Section titled “A script gets exit code 2”activate, deactivate and reset-upper ask for confirmation and refuse
without a terminal, with the error code needs_confirmation. Add -y to every
script, key binding or unit that runs them. emergency never asks.
The configuration is rejected
Section titled “The configuration is rejected”An unknown key, a value of the wrong type or a syntax error stops activate,
check, status and most other commands with exit code 3 and a one-line
message that names the file, the line and the column. Because activate stops
too, no hidden session can start to fix it in. Correct the file on the separate
machine and copy it onto the hidden volume, as
Configuration describes, and never edit it
on the decoy. deactivate and recover log a warning and carry on with
defaults, and emergency reads no configuration at all.
During activation
Section titled “During activation”Activation was interrupted
Section titled “Activation was interrupted”If the activation was killed, crashed or failed after it built /run/nextroot,
nails status reports an interrupted entry, exits with code 7 and says to run
nails recover. Run it on the decoy. Whatever the build wrote stays in the
upper layer on the hidden volume, and the decoy is untouched. If the kernel
killed NAILS for lack of memory during activation, treat it the same way. Ctrl+C
during activation rolls back by itself.
recover also closes the hidden volume’s encryption mapping when nothing uses
it. If the hidden volume is still mounted, it leaves the mapping open and says
so. Unmount and close the volume yourself when you are done.
The first activation on NixOS is slow
Section titled “The first activation on NixOS is slow”The first entry copies the hidden system’s whole closure onto the hidden volume, which can take several GB and minutes on a slow or encrypted backend. Activation reports the megabytes copied every 15 seconds. Later entries copy only the paths that are new. Activation is refused before it copies anything when the hidden volume lacks the space.
In the hidden session
Section titled “In the hidden session”deactivate refuses or fails
Section titled “deactivate refuses or fails”Run nails emergency. It kills every hidden program at once and takes the same
way back. If the way back cannot be prepared, it powers off instead.
deactivate refuses, for example, when a disk is mounted over the hidden
volume, or when the way back cannot make the decoy’s mounts again as the decoy
had them.
The hidden session hangs
Section titled “The hidden session hangs”Switch to a text console with Ctrl+Alt+F2, log in and run nails emergency. If
nothing responds, power off. A stuck service can delay an orderly deactivate
by up to DefaultTimeoutStopSec, 90 seconds by default. emergency does not
wait for it.
The hidden volume failed
Section titled “The hidden volume failed”If the drive was pulled or reports I/O errors, the session starts failing, and
the backend watcher alerts you. Run the RAM copy that activation left in /run:
sudo /run/overlay-runtime/control emergency. It needs nothing from the hidden
volume. Power off if it does not respond.
Backend watcher and disks
explains how to let the watcher run the emergency by itself.
A disk is mounted over the hidden volume
Section titled “A disk is mounted over the hidden volume”A disk mounted over the hidden root, or over one of its parents, takes every
file saved under the hidden root from then on. The watcher raises “Disk mounted
over a folder in use”, naming the disk and the sudo umount command for its
mount point, and nails status shows it under the hidden volume. NAILS itself
writes nothing there meanwhile. Until you unmount the disk, deactivate
refuses, and emergency powers off if the mounts the way back needs are covered
too.
The hidden volume is full
Section titled “The hidden volume is full”Copy-up or a write in the session fails. Free space inside the session, for
example with nix-collect-garbage -d and by clearing caches, or deactivate and
run reset-upper on a large path from the decoy. Never delete files out of the
upper layer or its work directory by hand. df -h /mnt/data shows the free
space.
Suspend and resume
Section titled “Suspend and resume”Do not suspend during a session. Suspend keeps the whole hidden session in RAM,
including the open encryption mapping and its key, behind the hidden session’s
lock screen. Set the decoy’s lid switch to lock and choose a
sleep policy. block keeps the host awake
for the session, a closed lid included, and teardown powers off when the host
is about to sleep.
After a session
Section titled “After a session”Power loss or a crash
Section titled “Power loss or a crash”The decoy boots normally, because nothing of the session is installed on it and
/run is gone. If nails status reports a recorded state that does not match,
run nails recover once. What the session wrote stays in the upper layer on the
hidden volume.
Ubuntu 26.04 lists chrony as failed
Section titled “Ubuntu 26.04 lists chrony as failed”On Ubuntu 26.04, chrony (time sync) and CUPS (printing) fail in the hidden
session, because AppArmor blocks their messages to systemd and to the journal,
whose sockets were set up before the switch. The clock is not synchronized and
printing times out. Other services that AppArmor confines may fail the same way.
After the way back, the decoy lists chrony as failed until its next full reboot,
which hints that a soft-reboot happened. Reboot fully after a session to clear
it. This is a known issue, and the fault lies outside NAILS. A plain
systemctl soft-reboot into another root, without NAILS, breaks chrony the same
way.
Cleaning up by hand
Section titled “Cleaning up by hand”If nails recover cannot run on the decoy, for example because the hidden
volume is gone or the binary does not start, remove what an interrupted
activation left behind yourself, in a root shell:
set +o history
# 1. Detach the next root and everything below it.umount -R /run/nextroot
# 2. Remove the session plan and RAM copy, the transition drop-in and the runtime guards (blk-availability's stop, NixOS's static addresses and the D-Bus units).rm -rf /run/overlay-runtimerm -rf /run/systemd/system/systemd-soft-reboot.service.drm -f /run/systemd/system/blk-availability.service.d/zz-overlay-runtime.confrm -f /run/systemd/system/network-addresses-.service.d/zz-overlay-runtime.confrm -f /run/systemd/system/dbus.service.d/zz-overlay-runtime.confrm -f /run/systemd/system/dbus.socket.d/zz-overlay-runtime.confsystemctl daemon-reload
# 3. Confirm nothing is left: findmnt prints no rows and ls reports "No such file or directory".findmnt -R /run/nextrootls /run/overlay-runtimeThen close the hidden volume as usual, with umount and cryptsetup close. If
cryptsetup close reports the device busy, a service that started during the
activation holds a copy of the next root’s mounts in a mount namespace of its
own. grep -l /run/nextroot /proc/*/mountinfo names its processes. Restart that
service, or power off. A power-off instead of the steps above does the same,
because everything they remove lives in /run.
If the activation built a hidden NixOS system, or its switch failed, power off
instead. nails recover also gives the kernel back the decoy’s module loader,
hotplug helper and firmware path, which the hidden system’s activation pointed
at its store, and removes the session’s unit masks in /run/systemd/system and
/run/systemd/system.control. Without that, the decoy loads no module and no
firmware, and its swap and boot-loader units stay masked. By hand, only a
power-off undoes this.
Do not edit or delete state.json by hand. nails recover resets it once the
hidden volume is back.
Reporting a problem
Section titled “Reporting a problem”Issues on GitHub are public and link your account to NAILS. Consider a throwaway
account, and never post nails status or nails verify output, real paths,
host names, user names or volume labels. Replace them with placeholders. Support
is best-effort while NAILS is alpha.
- Questions: read the FAQ and these docs, and search the existing issues first.
- Bugs: GitHub issues, with
nails --version, your distribution and its release, and the steps that reproduce the problem. - Security issues: never in a public issue. See Security.