Reboot Proofing¶
⚠ Moved OS-side (2026-07-05). Per ADR-0036, the tools target the boring persistent systemd host and know nothing about overlay wipes; everything on this page is now an Offline Lab OS conformance concern (duty list:
OS-CONFORMANCE.mdin the umbrella repo). Boot-time restore is the generic, idempotentappctl recover(replacesrehydrate/restore-apps.serviceas the mechanism); the OS ships one boot unit that calls it after/datamounts.
This page describes what state survives a reboot, what survives a slot switch (RAUC A/B OTA), and how each piece is handled.
Storage layout¶
The disk has three relevant partitions:
| Partition | Mount | Survives reboot | Survives slot switch |
|---|---|---|---|
| rootfs-a / rootfs-b | / (read-only) |
yes (unchanged) | yes (read-only OS image) |
| overlay.ext4 (p3) | /mnt/overlay |
no (upper wiped on every boot) | no (upper wiped on every boot) |
| data.ext4 (p4) | /data |
yes | yes |
The initramfs sets up an overlayfs combining the read-only rootfs with a per-slot upper directory from the overlay partition:
lowerdir = rootfs-a (read-only)
upperdir = /overlay/a/upper ← wiped on every boot; only explicitly restored files survive
workdir = /overlay/a/work
merged = / ← what the running system sees
The initramfs clears the slot upper directory on every boot (rm -rf /overlay/<slot>/upper)
before mounting the overlay. Only files explicitly restored from /data (currently: machine-id)
survive across reboots. Everything else written to /etc at runtime is gone on next boot.
What currently survives slot switches¶
All persistent device state lives on /data/, which survives slot switches unchanged:
| State | Location | Populated by |
|---|---|---|
| SSH host key | /data/config/ssh/hostkey |
bootconf.service |
| WiFi credentials | /data/config/wifi/wpa_supplicant.conf |
bootconf.service |
| Admin authorized_keys | /data/home/admin/.ssh/authorized_keys |
bootconf.service |
| App data and config volumes | /var/lib/appctl/apps/ |
appctl at install time |
| App state (file-per-record JSON) | /var/lib/appctl/state/ |
appctl |
| App images | /var/lib/appctl/images/ |
appctl |
| Resource baseline | /var/lib/appctl/resources.json |
offlinelab-resources.service |
| appctl config | /etc/appctl/appctl.conf |
appctl |
Network configuration (eth0, usb0, wlan0 static files) is in the read-only rootfs and is identical across both slots. No action needed.
What does not survive slot switches¶
machine-id¶
systemd generates a machine-id on first boot and writes it to /etc/machine-id via
the overlayfs (slot upper). A slot switch starts with an empty upper, so systemd
generates a new machine-id, which breaks journal continuity and any software
using the machine-id for device identity.
Fix: store machine-id in /data/config/system/machine-id. The initramfs copies
it to the slot upper at boot before switch_root, ensuring consistency across slots.
Flow:
1. First boot: systemd generates machine-id, writes to /etc/machine-id (overlay upper).
2. persist-machine-id.service runs on shutdown: copies /etc/machine-id to /data/config/system/machine-id (skipped if destination already exists).
3. Every subsequent boot: initramfs copies /data/config/system/machine-id to /overlay/<slot>/upper/etc/machine-id before switch_root.
The file in /data/config/system/machine-id is the authoritative copy. The overlay
entry is always populated from it, never generated fresh after first boot.
portablectl attachments¶
portablectl attach creates unit symlinks in /etc/systemd/system/ and drop-ins in
/etc/systemd/system/<unit>.d/. These live in the overlay upper and do not survive a
slot switch.
Fix: a boot-time service runs appctl rehydrate which re-attaches
all apps recorded in state/packages/. See T27/T36. This covers both first boot on a new
slot and normal boot after a power cycle.
Hostname¶
The default hostname (offlinelab) is static in the rootfs and survives slot switches
unchanged. Per-device hostname customization is out of scope until configctl (T46).
If an admin changes the hostname at runtime, it goes into the overlay upper and is lost
on slot switch. This is accepted behaviour for now.
Implementation tasks¶
| Task | What |
|---|---|
| T21 | persist-machine-id.service + initramfs machine-id copy |
| T27 | restore-apps.service / appctl rehydrate boot unit |
What the overlay upper may contain at runtime¶
For completeness, here is the full list of things that accumulate in the overlay upper during normal operation. All of these are wiped on every reboot (intentional; the overlay always starts clean):
| Path | Written by | Needs /data persistence? |
|---|---|---|
/etc/machine-id |
systemd PID1 on first boot | yes (T21) |
/etc/systemd/system/<app>.* |
portablectl attach | yes (rehydration T27/T36) |
/etc/portables/<app>.conf |
portablectl attach | yes (rehydration T27/T36) |
/etc/hostname |
hostnamectl (if used) | no; static default is fine for now |
/etc/localtime |
timedatectl (if used) | no; default timezone is fine for now |
All other /etc files are static in the rootfs and are not written at runtime.