Syn_OS ISO Build — Deep Dive

Syn_OS ISO Build — Deep Dive

Originally written: 2026-04-25 during v41 build (snapshot at stage 03, 17m31s, 425 crates) Last major update: 2026-05-13 — refreshed for v60.0.0 “Sun & Salt” ship + v60.1 follow-up sprint Build oracle: the build-oracle node — Intel i5-3337U Ivy Bridge (2c/4t @ 2.7 GHz), 11.4 GiB RAM, EndeavourOS rolling Current target: Syn_OS v60.0.1 (canonical rebuild in flight from bad81c9c) v60.0.0 reference build: sha256 d27ae5b73d56f444a14876cbc0f861bc6d0ceec57e0e52d96526a53e8954ca5a, 27 GB, shipped 2026-05-13 Author: Claude Opus 4.7 (1M context), session with Ty Limoges (the operator)

---

TL;DR

Building a custom ISO is a chain of 41 sequential stages. The first ~3 stages compile the kernel and Rust workspace. The middle stages assemble an Arch Linux root filesystem, install your custom binaries, and configure desktop/security/AI subsystems. The final stages compress that root into a squashfs image, wrap it with bootloaders (BIOS + UEFI), sign the kernel for Secure Boot, validate the result (including a QEMU boot test that reaches synos login:), and publish SBOMs. End artifact: a single bootable ISO around 4-30 GB depending on profile. The v60.0.0 master ISO is 27 GB; the live login takes ~125s UEFI / ~197s BIOS in QEMU.

This doc covers what each stage actually does, how the system underneath behaves while it runs, and which tools sit at every layer. It’s also a reference for the failure modes and self-healing patterns this codebase has accumulated across the v34 → v60 codesprint range.

---

1. The Pipeline at a Glance

41 stage scripts under fruit/iso/iso-build/scripts/stages/. They run sequentially (config: enable_parallel = false) under build.sh, which is the orchestrator. Top-level launcher is master-generation-run-kit.sh which adds pre-flight checks, auto-spawns a watchable terminal monitor (v60.1+), and chains into build.sh.

Group	Stages	Purpose
Pre-flight	00-branding-assets, 00-preflight, 00a-mount-sanity, 01-dependencies	Verify host has required tools, no half-mounted leftovers, branding artifacts present
Kernel + Rust modules	02-kernel, 02b-synos-rust-modules, 02c-cradle-capture	Build Linux 6.19 kernel with 17 custom syscalls (469-485) + 11 Rust kernel modules, capture v43 cradle for installed-system identity
Userspace Rust	03-rust-crates	Compile entire 160-crate workspace in release mode (mold linker + sccache + tiered codegen-units per P2/P3)
Base system	04-arch-base, 05-rootfs	Pacstrap minimal Arch into airootfs, rsync custom airootfs trees (v60.1: 3 overlay sources — branding-staging, fruit/assets/airootfs, fruit/iso/archiso/airootfs)
Desktop + boot	06-desktop-environment, 06b-pxe-server, 07-system-configuration, 07a-bootloader	Cinnamon + LightDM with G14.5 session-wrapper, GRUB/systemd-boot, fstab, MOK keys
Security	08-apparmor-policy, 09-security-tools, 10-kernel-hardening	AppArmor profiles (enforce mode), Tier-1 security tools + Calamares + paru AUR helper (v60.1 T3), sysctl hardening
AI + Hive	11-alfred-daemon, 11b-hive-boot, 11c-attest-setup, 11d-federation, 11e-raas, 11f-license-gate, 12-ai-models, 13-ebpf-framework	ALFRED daemon, hive bootstrap, attestation, federation, RaaS, license gate, ML model staging, eBPF framework
Content	14-grimoire-platform, 15-container-integration, 16-system-optimization-and-utilities, 16b-k3s-runtime, 17-multi-distro-environments, 17a-blank-slate, 17e-rootfs-validation-gate	GRIMOIRE 71 lab Dockerfiles (with v60.1 stub generator), Distrobox, k3s, slim profile prep, 14-assertion validation gate (A1-A14)
Image assembly	18-optimization, 18a-squashfs, 19-iso-assembly, 20-postflight	Final airootfs compress, squashfs creation, ISO bootloader wrap (with fallback initramfs per Q4), post-build cleanup
Validation + publish	21-automated-testing, 22-validation-qa, 23-patch-generation, 24-publish-sbom	Integration tests, QEMU boot matrix incl. UEFI/BIOS reaching synos login at <240s, delta patches, CycloneDX SBOM

The orchestrator writes an “atomic checkpoint” after each successful stage to fruit/iso/iso-build/scripts/.checkpoints/<stage-name>.checkpoint. On re-run, completed stages skip in milliseconds. This is what enables incremental rebuilds — fix one bug, re-run, and only the affected downstream stages run.

1.1 Two airootfs trees, one previously dead (v60.1 fix)

The build pipeline has THREE separate airootfs/overlay sources, all consumed by stage 05:

Source	Purpose	History
build/branding-staging/	Per-build branding assets generated by stage 00	Live since v34
fruit/assets/airootfs/	Static branding/identity overlays	Live since v30 Arch pivot
fruit/iso/archiso/airootfs/	Service units, sbin tools, configs, Q-polish	Dead for ~6 months until v60.1 — see Section 7.5

The fruit/iso/archiso/airootfs/ tree was technically committed but the build pipeline NEVER rsync’d it into the rootfs. As a result 31 load-bearing files (13 sbin tools incl. synos-health-check, synos-tailscale-bootstrap, synos-fido2-enroll, 7 systemd services, Q1/Q2/Q5 polish, journald-persist tmpfile, sudoers/udev rules) were silently absent from every ISO since the Arch pivot. Fixed in v60.1 commit cffa5e13 — stage 05 now mirrors the fruit/assets/airootfs rsync hook for the archiso airootfs tree. Always check fruit/iso/archiso/STATUS.md before assuming an airootfs file “ships just because it’s committed.”

---

2. Stage 03 — The Rust Workspace Compile (the long one)

This is where most of the wall-clock time goes. The script invokes:

cargo build --workspace --release \
  --features synos-build-profile/profile-master \
  --target-dir build-target \
  --jobs 4

2.1 What --release actually means

The release profile (in Cargo.toml and .cargo/config.toml) sets:

Knob	Value	What it does
opt-level	3	Maximum LLVM optimizations: aggressive inlining, loop unrolling, vectorization, escape analysis
codegen-units	1	LLVM sees the entire crate as one unit. Disables parallelism inside a crate but enables better cross-function optimization
lto	true (fat)	After all crates compile, the linker re-runs LLVM optimization across the entire binary boundary. Single-threaded, slow, huge runtime payoff
panic	abort	No unwind tables. Smaller binaries, no stack-trace-on-panic recovery
strip	symbols	Debug symbols stripped from binaries
linker	clang	Use Clang as the linker driver (better LTO support than GCC)
link-arg=-fuse-ld=lld	—	LLD as the actual linker (faster than GNU ld for LTO loads)

opt-level=3 + codegen-units=1 + lto=true is the holy trinity for runtime performance. It also makes the build 2-4x slower than the dev profile. For an OS shipping a daemon that runs 24/7 (ALFRED), this tradeoff is correct.

2.2 What rustc is doing per crate

For each of the ~600 compilation units (160 workspace crates + ~440 transitive deps), rustc walks through these phases:

1. Parse → AST (tens of milliseconds, fast)
2. Macro expansion → run proc macros, expand derive(Serialize), tokio::main, etc. (variable, can dominate for macro-heavy crates)
3. HIR/THIR/MIR lowering → three intermediate representations of decreasing abstraction (medium)
4. Borrow checking → the famous one. Verifies ownership, lifetimes, mutability rules (medium-fast)
5. MIR → LLVM IR → emit LLVM bitcode (medium)
6. LLVM optimization → at opt-level=3 with codegen-units=1, LLVM optimizes the entire crate as a single module. This is where 50-80% of compile time goes per crate. Inlining, loop transforms, SIMD vectorization, dead code elimination
7. LLVM codegen → bitcode → x86_64 machine code
8. Crate output → .rmeta (metadata) + .rlib (compiled library) into build-target/release/deps/

Then for binary crates (alfred, synos-ops, grimoire-daemon etc.) one extra step:

9. Link → with lto=true, the linker (lld) collects all .rlib bitcode for the entire dep tree and runs LLVM optimization a SECOND TIME across crate boundaries. Single-threaded. Per binary. Often takes 1-3 minutes for a heavy binary like alfred.

This is why stage 03 has a long “looks like nothing’s happening” tail — the parallel rustc workers all finish, then lld runs LTO sequentially per binary. CPU usage drops to 1 core pegged.

2.3 The -sys crates and why GCC is also running

Several Rust crates wrap C/asm libraries:

• aws-lc-sys — AWS-LC cryptography
• `r
• ing` — ring crypto
• pqcrypto-* — post-quantum (kyber, dilithium, falcon, sphincsplus, etc.)
• sha2-asm — hand-written x86 assembly SHA-2
• zstd-sys — zstd compression
• tract-linalg — linear algebra for ML inference

Each of these has a build.rs that invokes cc / gcc to compile C/asm sources at build time. That’s why you see cc1 (GCC’s frontend) in the process list eating its own ~50% CPU. Those C compiles happen in parallel with rustc workers.

2.4 Why we ran into sccache trouble this session

~/.cargo/config.toml sets rustc-wrapper = "sccache" globally. sccache is a compiler cache that wraps rustc invocations: identical cargo invocation + identical inputs = pull cached object from disk instead of re-compiling.

When invoked under sudo, two things broke:

1. Root cargo discovered the user’s ~/.cargo/config.toml via cargo’s hierarchical config walk (CWD upward through .cargo/config.toml files).
2. sccache server is owned by the build user. When root tried to connect to its IPC socket, the connection failed → “Failed to read response header / failed to fill whole buffer” → cargo exit 101.

Fix for this build: CARGO_BUILD_RUSTC_WRAPPER= (empty) overrides the config. Permanent fix: don’t run sudo build.sh — invoke as user, and the script elevates internally per-stage via sudo only where needed.

---

3. The System Layer Underneath

3.1 The process tree right now

PID    RSS      CPU%   COMMAND
40456  ~50 MB   0%     bash (build.sh orchestrator)
42352  ~400 MB  2%     cargo (workspace builder)
42365  ~178 MB  7%     sccache (idle — bypassed for this build)
59249  ~377 MB  64%    rustc (one of up to 4 workers)
62383  ~55 MB   54%    cc1 (GCC compiling C in a build.rs)
37150  ~5 MB    0%     bash (sudo keepalive, refreshes timestamp every 50s)
31496  1.6 GB   97%    updatedb (mlocate cron job — competing for cycles)
9617   ~346 MB  15%    claude (me, watching)

Cargo and bash are barely consuming anything. They orchestrate. The actual heavy lifting is rustc (compile work) and cc1 (C compile work). When stage 03 finishes you’ll see a brief drop to a single rustc + lld process per binary as LTO runs sequentially.

3.2 CPU scheduling under oversubscription

With --jobs 4 cargo runs up to 4 rustc workers in parallel. Add cc1 invocations from -sys crate build.rs scripts plus background system load (updatedb, cinnamon, claude, Xorg, etc.) and runnable processes routinely exceed 7-9. On 4 hardware threads (2 physical cores × 2 SMT lanes), that’s about 2x oversubscription.

This sounds bad. It isn’t. Why:

• A rustc worker spends meaningful time waiting on memory loads, disk I/O for source files, and pipe communication with the LTO linker. While it waits, the kernel scheduler runs another thread.
• vmstat showing id=0% (zero idle CPU) with wa=1% (almost zero I/O wait) means the CPU is fully utilized but not blocked on disk. That’s the ideal shape for a compute-bound workload.
• Load average over 1 min was ~8.26, over 5 min ~6.89, over 15 min ~4.64. The climbing curve is normal — early stage 03 was still pulling cached deps; mid-stage is now full-tilt compile.

The downside of oversubscription is context-switch overhead. vmstat showed cs ~3000-4000/sec — modern Linux handles this easily. The -j4 choice fits the i5-3337U well; bumping to -j8 would just add overhead without gain (only 4 hardware threads).

3.3 Memory math

Linux memory accounting is famously confusing. The numbers that matter:

Field	Meaning
MemTotal	All physical RAM the kernel sees
MemFree	Pages literally untouched. Always small on a healthy system.
Cached	File system page cache — files mmap’d or recently read, kept in RAM in case anyone wants them again. Reclaimable instantly.
Buffers	Tiny: block I/O metadata cache
MemAvailable	The number you actually care about: how much an app can allocate without forcing swap
Active(anon)	Process working sets actively in use
Inactive(anon)	Process pages not recently touched — candidate for swap-out
SwapUsed	Pages evicted from RAM to swap (ZRAM or disk)
si / so (vmstat)	Swap-in / swap-out rate right now (KB/s). This is the activity signal

Snapshot from this build:

MemTotal:    12.1 GB    (your 11.4 GiB rounded)
MemFree:      0.8 GB
Cached:       2.5 GB
MemAvailable: 7.1 GB    ← real headroom
SwapUsed:     1.1 GB
si=0 so=0              ← no swap I/O happening right now

The 1.1 GB of swap is “out and quiet” — kernel decided some long-idle pages from background services (containerd, fail2ban, cinnamon settings) weren’t worth keeping in RAM during the build. Those pages stayed put on swap; nothing’s paging back in. That’s healthy.

Rule of thumb: when si/so rates climb above 100 KB/s sustained, you’re thrashing. When they’re 0, you’re fine even if SwapUsed is non-zero. Watch the rate, not the total.

The build’s resident working set is roughly:

• cargo: ~400 MB
• 4 × rustc: ~250-500 MB each = 1-2 GB peak
• cc1 instances: 50-150 MB each
• Filesystem cache holding intermediate .rmeta/.rlib: ~2 GB

Total active footprint: ~3-4 GB. On 11 GB you’re nowhere near pressure.

3.4 I/O patterns

For a release Rust build:

• Heavy reads during dep resolution (cargo loads ~600 Cargo.toml files), then again as rustc reads source files
• Sustained writes of intermediate .rmeta and .rlib artifacts to build-target/release/deps/
• After compile, bursty heavy writes of final binaries

vmstat showed bi=88-260 KB/s (block in) and bo=0-144 KB/s (block out). For an SSD that’s nothing — sub-1% of bandwidth. Most of cargo’s “I/O” is satisfied from page cache because the kernel already has the source files and intermediate artifacts hot.

Stage 18a (squashfs) is the I/O-heavy stage to watch. Compressing 25 GB of airootfs with zstd L6 will sustain 50-200 MB/s reads from the build tree and 30-80 MB/s writes to the output ISO. That’s where tmpfs /tmp size matters.

3.5 ZRAM vs disk swap

EndeavourOS typically configures ZRAM — a compressed RAM-backed swap device. When the kernel evicts pages, they get compressed and stored in a ZRAM block device instead of going to disk. Pros: 2-4x effective RAM expansion, no SSD wear. Cons: pages still take RAM (just less). On this build, ZRAM is doing its job — 1.1 GB of swapped pages probably compressed to ~400-500 MB of actual RAM.

---

4. Critical Tools Deep Dive

4.1 mkarchiso (the airootfs composer)

mkarchiso is the upstream Arch Linux ISO builder. Syn_OS uses it as the foundation for stages 04 and 19.

What it does:

1. Reads a profile dir (fruit/iso/archiso/) containing airootfs/ (the root filesystem template), packages.x86_64 (package list), pacman.conf, efiboot.cfg, syslinux.cfg, etc.
2. Pacstraps the package list into a temporary chroot
3. Copies airootfs/ overlay into the chroot (your config files override package defaults)
4. Runs mkinitcpio inside the chroot to generate the initramfs
5. Composes a squashfs of the chroot
6. Wraps the squashfs with bootloaders (BIOS via syslinux/isolinux, UEFI via systemd-boot or GRUB)
7. Generates the final ISO with xorriso

Syn_OS’s stages don’t use mkarchiso end-to-end — they break it apart so each phase can be checkpointed and resumed. Stage 04 does the pacstrap, stage 18a does the squashfs, stage 19 does the ISO assembly.

4.2 mkinitcpio + KMS modules

mkinitcpio builds the initramfs — a tiny root filesystem that the kernel boots into BEFORE mounting your real root. Its job: load whatever modules are needed to find and mount the real root, then switch_root into it.

For a live ISO, the real root is the squashfs inside the ISO. To mount that, the initramfs needs:

• Storage drivers: loop, squashfs, cdrom, sr_mod, isofs, iso9660, ahci, ata_piix
• Paravirt drivers (for VMs): virtio_blk, virtio_scsi, virtio_pci, virtio_gpu
• KMS framebuffer drivers (so Plymouth boot splash shows up): bare metal i915 amdgpu nouveau, VMs bochs qxl vmwgfx cirrus-qemu
• Device-mapper for overlay: dm_snapshot, dm_mod, nbd

The kernel 6.19 module rename caught us this session:

• bochs_drm (kernel ≤6.18) → bochs (kernel 6.19+)
• cirrus (kernel ≤6.18) → cirrus-qemu (kernel 6.19+)

Without the rename, mkinitcpio warned “module not found” and silently shipped an initramfs without QEMU std/cirrus KMS support. Result: VM cells boot to a black screen because Plymouth has no framebuffer to draw on. This is the L1 boot-hang signature that hounded us for days. Fixed in commit ba212232.

The initramfs HOOKS list determines load order. Critical sequence:

HOOKS=(base udev plymouth modconf kms archiso archiso_loop_mnt
       archiso_pxe_common archiso_pxe_nbd archiso_pxe_http
       block filesystems keyboard)

plymouth MUST come before filesystems so the splash shows during the long squashfs mount. kms MUST come early so DRM device exists for Plymouth to draw on. archiso_* hooks come from the archiso package and handle the squashfs-mount-and-overlay magic.

4.3 mksquashfs + zstd

squashfs is a read-only compressed filesystem. The whole airootfs (~25 GB uncompressed for master profile) gets squashed into a single file (typically 2-8 GB compressed).

Syn_OS uses zstd compression at level 6 (configured in iso-config.toml as compress: zstd L6). Tradeoff curve:

• Level 1: fast (200+ MB/s), 30-40% compression
• Level 6: balanced (~80 MB/s), 50-55% compression
• Level 22: max (~10 MB/s), 55-60% compression

Level 6 is the sweet spot for ISO builds — squashfs creation takes 20-40 min on this hardware, output ISO is small enough.

mksquashfs is single-threaded for the high-level composition but uses multiple threads for compression (-processors $(nproc)). Expect heavy CPU + heavy I/O during stage 18a.

4.4 GRUB / systemd-boot / isolinux + Secure Boot

The ISO needs to boot on three different scenarios:

• Legacy BIOS → isolinux (the El Torito boot record on the ISO)
• UEFI standard → systemd-boot or GRUB (in the EFI System Partition embedded in the ISO)
• UEFI Secure Boot → signed shim → signed grubx64.efi → signed kernel → signed initrd

Stage 19 (iso-assembly) sets up all three. Stage 07a (bootloader) generates the bootloader configs, including the gfxpayload setting that bit us in L1 (UEFI must be text to avoid OVMF GOP handoff race).

For Secure Boot, stage 07a auto-generates a dev MOK keypair (synos-mok-dev.{priv,pem,der}) via openssl if absent. Release builds override via $SYNOS_MOK_DIR. The kernel + initrd get signed by stage 19’s secureboot-sign.sh. Shipping uses AUR shim-signed (Microsoft-signed shim that loads our MOK-trusted GRUB).

4.5 AppArmor

Stage 08 loads AppArmor profiles into the chroot’s /etc/apparmor.d/. AppArmor is mandatory access control — defines per-binary allowed capabilities, file paths, network access, signal targets.

Syn_OS ships with AppArmor in enforce mode (vs upstream Arch which defaults to complain mode). Enforcement enabled in v41 wave 10. Stage 08 has a critical guard against the L21 AppArmor bug — it skips *.md|*.txt|*.rst|README*|CHANGELOG*|LICENSE*|.* files when copying into /etc/apparmor.d/ because apparmor_parser would try to parse markdown bullets (*) as profile syntax and abort the entire batch load.

---

5. The Hardware Constraint

The build oracle is an i5-3337U Ivy Bridge from ~2013:

• 2 physical cores, 4 SMT threads, 2.7 GHz max
• 11.4 GiB RAM, no swap partition (ZRAM only)
• 917 GB SSD (LUKS + btrfs)

This shapes every choice in the build pipeline:

• --jobs 4 matches hardware threads. More wouldn’t help.
• enable_parallel = false (sequential stages) — can’t run two heavy stages at once on 2 cores
• compress: zstd L6 — L22 would be better compression but stage 18a would take 4+ hours on this CPU
• codegen-units=1 + lto=true — slow to compile but correct for a ship-quality binary on 2-core hardware
• Build targets cpu_target = "haswell" (one gen newer than the host) — the binary will run on most x86_64 hardware from the last decade
• Docker is not used for ISO builds on this hardware (would add overhead) — stages run directly on the host

The warm-spare infrastructure (Ansible, the GPU node) exists because if this oracle dies mid-build, you need failover. v41 wave 8 shipped that.

---

6. The Self-Healing Toolkit

fruit/iso/iso-build/scripts/lib/selfheal.sh provides 8 helpers stage scripts can invoke:

Helper	Purpose
sh_preflight_env()	Verify required env vars + tool presence
sh_preflight_dns()	Confirm DNS resolution works (chroot inherits)
sh_preflight_mirrors()	Validate pacman mirror reachability
sh_preflight_cleanup()	Remove stale lock files, half-mounted leftovers, orphan processes
sh_preflight_kconfig_deps()	Check kernel config dependency chain (catches CONFIG_MODVERSIONS=y blocking CONFIG_RUST=y type bugs)
sh_stage_assert_outputs()	Post-stage assertion that expected output files exist
sh_timeout_chroot()	Wraps chroot commands with timeout to prevent hangs
sh_run_preflight_suite()	Runs all preflights at stage start

These were added in Batch L2 of the v34 codesprint, after a 10-attempt build campaign exposed 14+ latent landmines. The pattern: every failure mode that ever burned us got a preflight check.

Critical-artifact allowlist (added in Phase D/E): even if a stage’s checkpoint exists, the orchestrator re-runs it if a critical output (e.g., synos-hive-controller binary) is missing from the rootfs. Prevents “stage marked complete but output deleted” silent failures.

---

7. Failure Modes Seen This Session (case study)

We hit three nested issues in this build attempt. Documenting because they’re representative of the class of failures the orchestrator has hardened against.

Failure 1: sccache wrapper unreachable under sudo

Symptom: cargo exit 101 in 7s, “Failed to read response header / failed to fill whole buffer” Root cause: invoked sudo build.sh → root cargo walked up CWD to $HOME/.cargo/config.toml → discovered rustc-wrapper = "sccache" → tried to use the user-owned sccache server socket → IPC failed Fix: sudo env CARGO_BUILD_RUSTC_WRAPPER= ./build.sh (override wrapper for this invocation) Permanent fix: never run sudo build.sh — invoke as user, stages elevate internally

Failure 2: build-target ownership pollution

Symptom: cargo build actually succeeded, but stage 03 still exits 1. Error: “Cargo target directory contains 19,845 root-owned files” Root cause: prior sudo cargo runs left root-owned files in build-target/. Stage 03’s preflight has an explicit guard that fails if any root-owned files exist (anti-pattern protection). Fix: sudo chown -R "$USER:$USER" build-target growth/output growth/development Don’t fix: build/iso/ — that’s intentionally root-owned (it’s the staged airootfs that ships with root ownership inside the squashfs)

Failure 3: lock file inaccessibility

Symptom: stage 03 fails in 3s with “Failed to acquire checkpoint lock for stage: rust-crates” Root cause: growth/development/output/iso-locks/ directory and existing lock files were owned by root from prior sudo runs. User-mode build couldn’t write a new lock. Fix: same chown sweep as above (specifically growth/development/output/iso-locks/)

All three were instances of the same anti-pattern: running the orchestrator as root. The script tells you not to in plain English, but the temptation is real because some internal stages need elevation. The right answer: invoke as user + maintain a sudo timestamp keepalive for unattended runs.

The keepalive pattern

For long unattended builds where internal sudo calls might exceed the timestamp_timeout (default 5-15 min):

# refresh once interactively
sudo -v

# start detached keepalive
setsid bash -c 'while sudo -n true 2>/dev/null; do sleep 50; done' \
  </dev/null >/tmp/keepalive.log 2>&1 &
disown

Uses sudo -n true (not -n -v) because -v fails under setsid (needs tty in some PAM configs). The keepalive exits cleanly when sudo cache becomes invalid (e.g., user runs sudo -k to revoke).

---

7.5 Failure Modes Caught v42 → v60 (later case studies)

The v41 failures in §7 were all top-level orchestration mistakes. The v42 → v60 codesprint surfaced a deeper class of bugs that lived inside individual stages or in shared infrastructure. Each one corresponded to a memory entry under the project’s development memory.

F1: Dead-code airootfs tree (v60.1 root cause discovery)

Already covered in §1.1. The lesson is methodological — when a directory looks “live” (named correctly, tracked, populated) but its contents never appear in the squashfs, the bug isn’t in the file contents but in the absent rsync hook. Diagnose by extracting the shipped squashfs (unsquashfs -l airootfs.sfs | grep <expected_path>) and diffing against the repo tree. If the diff shows every file in repo but missing in squashfs, the source dir is orphaned. Memory: feedback_airootfs_dead_code.md.

F2: Chronic stage exit-1 across stages 22 / 14 / 15

Symptom: orchestrator’s wait $_stage_pid 2>/dev/null || stage_exit=$? catches exit code 1 every single time across 11+ builds (stage 22), 8+ (stage 14), 7+ (stage 15). Yet the stage log shows success "Stage X completed" and log "Final ..." both printed cleanly, and every validator returns PASS.

Root cause: set -euo pipefail was killing the script BETWEEN the final success log line and the explicit return 0. The log() function in lib/common.sh ends with if [[ "$LOG_TO_FILE" == "true" ]] ...; fi — when LOG_TO_FILE was somehow false in that context, the [[ ... ]] test returned 1, the if with no branch taken returned 1 as the function’s exit status, and set -e killed the script before return 0 ran.

v60.1 fix: set +e immediately before the final success + log + return 0 trio in main(), plus a belt-and-suspenders trailer that explicitly exit 0 if _main_rc=0. Treats the symptom rather than the root cause (the bash semantics of if ... fi returning the test’s exit status when no branch is taken) but is robust. Memory: see commit cffa5e13.

F3: Host /dev/pts torn down by chroot cleanup propagation

Symptom: after a build runs, NO terminal emulator on the build oracle can allocate a pty. xterm exits with not enough ptys, kitty shows a Python traceback in tabs.py:add_tabs_from_session, xfce4-terminal opens then immediately exits. mount | grep devpts returns empty, ls /dev/pts returns empty. Kernel /proc/sys/kernel/pty/max=4096, /proc/sys/kernel/pty/nr=2 (plenty), so resource exhaustion is NOT the issue.

Root cause: stages used mount --bind /dev "$ROOTFS/dev" and mount --bind /dev/pts "$ROOTFS/dev/pts" with the default shared propagation type. When the chroot’s cleanup phase ran umount -l "$ROOTFS/dev/pts", the umount PROPAGATED BACK through the peer group to the host’s actual /dev/pts mount, removing it entirely. /dev/ptmx survived (it’s a static node), but with no devpts mounted, posix_openpt(3) returns ENOENT.

v60.1 fix (bad81c9c): all chroot bind mounts now use mount --rbind followed immediately by mount --make-rslave, in lib/common.sh and stage 18a-squashfs.sh. rslave propagation lets mounts flow FROM host but blocks propagation BACK TO host. Plus master-generation-run-kit.sh:spawn_build_monitor() now also auto-remounts /dev/pts as belt-and-suspenders if it’s missing at build start.

Manual recovery if you hit this on an old build: sudo mount -t devpts devpts /dev/pts -o gid=tty,mode=620,ptmxmode=0666.

F4: sed & matched-text reference corrupts branding templates

Symptom: A12 17e gate FAIL — codename rendered as "Sun @SYNOS_CODENAME@ Salt" instead of "Sun & Salt" in /etc/synos/brand-meta.toml, /etc/motd, etc. Caused 2 entire canonical rebuilds (defective + setsid retry) to be discarded.

Root cause: sed 's|@SYNOS_CODENAME@|${codename}|g' — when ${codename}="Sun & Salt", sed sees s|...|Sun & Salt|g and & is a back-reference to the matched text. So @SYNOS_CODENAME@ got replaced with Sun @SYNOS_CODENAME@ Salt (the literal ”&” became the matched text again).

Fix: escape & (and \ and the delimiter) in the sed replacement before substitution:

local _cn_esc="${codename//\\/\\\\}"  # backslash first
_cn_esc="${_cn_esc//&/\\&}"           # then ampersand
_cn_esc="${_cn_esc//|/\\|}"           # then pipe (if used as delim)
sed "s|@SYNOS_CODENAME@|${_cn_esc}|g" ...

Applied in stage 00-branding-assets.sh and stage 07-system-configuration.sh. Memory: feedback_sed_ampersand_in_codename.md.

F5: du -sk dir/ | cut -f1 under pipefail with root-owned subdirs

Symptom: cleanup-build-dir.sh exits 1 silently in the orchestrator’s pre-build cleanup, even though du errors were redirected to /dev/null. Caused multiple aborted launches.

Root cause: under set -o pipefail, the rightmost non-zero exit code wins. du -sk dir/ 2>/dev/null returns non-zero when it encounters subdirs it can’t read (root-owned gnupg keyring, credstore, sudoers.d). The cut -f1 succeeds. But pipefail propagates du’s non-zero, and the entire substitution before_kb=$(du -sk ... | cut -f1) fails under set -e.

Fix: trail with || true and default-to-zero:

before_kb=$(du -sk "${BUILD_DIR}" 2>/dev/null | cut -f1 || true)
before_kb="${before_kb:-0}"

Memory: feedback_du_pipefail_root_owned.md.

F6: systemd-run --user cgroup teardown kills disowned builds

Symptom: setsid bash build.sh & disown should keep the build alive after the launching shell exits. But when launched via systemd-run --user --scope bash build.sh, the build dies the moment the launching terminal closes — even with disown and PR_SET_PDEATHSIG defenses.

Root cause: systemd-run --user creates a transient user-scope cgroup. When the user-session that owns the scope ends (terminal closed, ssh disconnected), systemd tears down the cgroup, which sends SIGKILL to ALL processes in it including children — disown doesn’t help because cgroups don’t honor PID-tree disownership.

Fix: use bare setsid bash -c 'exec bash build.sh > log 2>&1' </dev/null >/dev/null 2>&1 & disown. setsid creates a new process group + session detached from the parent’s cgroup. Memory: feedback_setsid_for_iso_build.md.

F7: nats-server AUR-only on Arch

Symptom: stage 04-arch-base reports pacstrap success but the rootfs has no nats-server binary. Alfred federation event bus then logs “client error: nats: IO error” every reconnect cycle on first boot.

Root cause: nats-server is NOT in extra/community repos on Arch — it’s AUR-only. pacstrap silently no-ops on unavailable packages without erroring.

Fix: fruit/iso/iso-build/scripts/helpers/install-nats-server.sh — downloads upstream static binary from GitHub release nats-server-v2.10.22-linux-amd64.tar.gz, installs systemd unit + config + creates nats user, force-creates the wants symlink. Invoked from stage 07 (D15 fix). Memory: feedback_nats_server_aur_only.md.

F9: cleanup_overlayfs returning non-zero in no-op path

Symptom: stage 22 was flagged FAILED by orchestrator even though main() returned 0 cleanly — pre-v60.1 of F2 root cause investigation.

Root cause: cleanup_overlayfs() in lib/common.sh ends without an explicit return 0. When ENABLE_OVERLAYFS != "true", the function returns 0 immediately via early return. But when overlayfs IS enabled and no mount exists, neither if branch fires, and the function returns the exit code of the last [[ ]] test (typically 1 — “not present”). The EXIT trap captures $? at trap entry, but if cleanup_overlayfs is the last command before the trap fires, its rc=1 leaks.

v43.4 fix: capture $? AT TRAP ENTRY into _synos_trap_rc, then exit "$_synos_trap_rc" at trap end. Plus init_common is supposed to install that trap — discovered in v60.1 to not actually be invoked in any stage script, so the trap never gets set. Real fix is upstream of F2.

F10: GRIMOIRE lab Dockerfiles miss build context

Symptom: 0 of 71 lab Docker images can be built. Every Dockerfile fails at COPY templates/ /home/researcher/lab/templates/ with "templates": not found.

Root cause: every lab directory has a complete lab.json (description, objectives, tools, prereqs) and a Dockerfile, but templates/ was never created in the repo and no README.md was generated. The labs were content-shelled — Dockerfiles + metadata only, no actual build context.

v60.1 fix (277a429f): fruit/iso/iso-build/scripts/helpers/generate-lab-stubs.sh reads each lab.json and produces:

• README.md (lab name, description, objectives, tools, prereqs from JSON)
• templates/HINTS.md (placeholder)
• templates/.gitkeep

The generated README/HINTS templates were committed. Lab content (actual challenge scripts, target binaries, CTF prompts) remains a GRIMOIRE content sprint.

Additionally, the 40 grimoire/lab-*:latest image registry pulls always fail (no upstream registry yet). Bundler now checks docker image inspect after pull failure and accepts locally-tagged images. Plus helpers/prebuild-grimoire-labs.sh lets the operator pre-build a starter set of 15 labs before the build (~30-45 min on 2c/4t).

F11: Bash ((counter++)) returns 1 when counter starts at 0 under set -e

Symptom: generate-lab-stubs.sh exits silently after one iteration; counter scripts kill themselves.

Root cause: ((expr)) returns 1 if the expression EVALUATES to 0. With count_total=0, ((count_total++)) does post-increment: the expression value is the old value (0), so rc=1. Under set -e, the script dies.

Fix: use count=$((count + 1)) (assignment form returns 0) instead of ((count++)). Or ((count++)) || true. Pure-arithmetic post-increment is a classic bash + set -e footgun.

---

8. Post-Build Validation Pipeline

Once Build Status: completed shows up:

ISO=$(ls -t growth/output/iso/Syn_OS-v41.0.0-master-*.iso | head -1)

# 1. VM boot matrix — boots ISO across (BIOS|UEFI) × (std|qxl|virtio-gpu|vmware|cirrus)
./fruit/iso/iso-build/scripts/testing/vm-test-matrix.sh "$ISO"

# 2. Boot benchmark — N iterations, captures dmesg milestones (kernel init, userspace, login)
./fruit/iso/iso-build/scripts/testing/vm-benchmark.sh "$ISO" --iterations 3

# 3. Interrogator — host-side ISO inspection (~280 LOC)
just iso-interrogate ISO="$ISO"

The interrogator verifies:

• ISO SHA matches .sha256 and publish-manifest.json
• GPG signature valid
• El Torito + UEFI boot paths present
• airootfs.sfs extracts cleanly
• Kernel version matches expected (6.19-synos-ai)
• cosign signatures on kernel modules valid
• alfred-daemon, synos-progression, memguard, tail-slayer, build-downstream binaries present
• SLSA provenance attached
• SBOM present and valid CycloneDX
• Public SECURITY.md exists and isn’t a symlink
• INTEGRITY_MANIFEST.toml total_labs == 241
• Curtain v2 artifacts (apparmor/seccomp/tier.toml) shipped

vm-test-matrix emits PASS/WARN/FAIL per cell as a markdown report. vm-benchmark emits JSON + markdown. Both write to growth/output/testing/.

---

9. Glossary

• airootfs — Arch ISO root filesystem template. Contents become the running root after squashfs mount.
• archiso — Arch’s ISO build framework (provides mkarchiso, hooks like archiso_loop_mnt)
• build-target — cargo’s intermediate artifact directory (this project sets it to build-target/ at workspace root)
• checkpoint — small file written after a stage succeeds, used by orchestrator to skip on resume
• codegen-units — how many parallel LLVM IR generation tasks per crate. 1 = better optimization, slower compile
• El Torito — the ISO boot record format that lets a CD-R image boot on legacy BIOS
• initramfs — minimal initial root the kernel boots into before mounting the real root
• KMS — Kernel Mode Setting. Direct kernel-managed framebuffer (vs userspace VBE/legacy)
• LTO — Link-Time Optimization. Re-running LLVM optimizer at link time across crate boundaries
• MOK — Machine Owner Key. Per-machine Secure Boot trust anchor for non-Microsoft kernels
• mkarchiso — Arch’s high-level ISO builder. Composes airootfs + bootloaders into final ISO
• OVMF — Open Virtual Machine Firmware. UEFI implementation for QEMU
• pacstrap — install pacman packages into a target dir (used to build chroots)
• rustc-wrapper — cargo config that wraps every rustc invocation. sccache uses this
• sccache — Mozilla’s compiler cache. Hashes inputs, caches outputs. Big speedup on repeated builds
• shim — small Microsoft-signed bootloader that loads a non-Microsoft signed bootloader. Foundation of Linux Secure Boot
• squashfs — read-only compressed filesystem. The whole live OS lives in one squashfs file inside the ISO
• systemd-boot — simple UEFI bootloader (formerly gummiboot). Lighter than GRUB
• xorriso — modern ISO authoring tool (mkisofs successor). What stage 19 ultimately calls
• ZRAM — compressed RAM-backed swap. Pages get compressed in RAM instead of going to disk

---

10. Quick Reference Commands

# Launch build (as user, never sudo at top level)
./fruit/iso/iso-build/scripts/build.sh --profile master --release

# Resume from a specific stage
./fruit/iso/iso-build/scripts/build.sh --profile master --resume-from 03

# List all stages
./fruit/iso/iso-build/scripts/build.sh --list-stages

# Pre-flight readiness
just iso-audit

# Live build monitoring
tail -F /tmp/synos-rebuild/latest.log

# Filtered (signal only)
tail -F /tmp/synos-rebuild/latest.log | grep --line-buffered -E "STAGE|FAILED|\[ERROR\]|completed in|BUILD COMPLETE"

# Health snapshot anytime
pgrep -af "build.sh" | grep -v claude && tail -3 /tmp/synos-rebuild/latest.log | sed 's/\x1b\[[0-9;]*m//g' && free -h | head -2

# Clean up root-owned pollution from accidental sudo build
sudo chown -R "$USER:$USER" build-target growth/output growth/development
# Don't chown build/iso/ — that's intentionally root-owned

# Sudo keepalive for unattended builds
sudo -v
setsid bash -c 'while sudo -n true 2>/dev/null; do sleep 50; done' </dev/null >/tmp/keepalive.log 2>&1 &
disown

# Stop everything (worst case)
pgrep -f "build.sh|keepalive.sh|cargo build" | xargs -r sudo kill

---

11. Reference Snapshot — v60.0.0 shipped (2026-05-13)

The v60.0.0 master ISO completed its canonical run in 9h 47min on this oracle:

Stage	Duration	Notes
02-kernel	3h 15min	Custom 6.19-synos-ai compile, the long pole
03-rust-crates	8min	160 crates with mold linker + sccache (P2/P3 paid off — was 17min+ in v41)
04-arch-base	107s	pacstrap, often warm-cached
06-desktop-environment	~7min	Cinnamon + Xfce + lightdm
09-security-tools	2h 56min	BlackArch repo + Tier-1 tools + Calamares + paru
18a-squashfs	2h 11min	zstd L6 compression of ~30 GB rootfs
19-iso-assembly	11min	xorriso + isolinux + GRUB + EFI image (now includes fallback initramfs per Q4)
22-validation-qa	28min	QEMU boot test reached synos login: at 125s UEFI / 197s BIOS, all 17e A1-A14 PASS

Final artifact:

• Syn_OS-v60.0.0-master-20260513-x86_64.iso (27 GB, 27,205 MB)
• sha256: d27ae5b73d56f444a14876cbc0f861bc6d0ceec57e0e52d96526a53e8954ca5a
• Shipped to a direct-write USB on /dev/sdb, sha256 verified after copy

12. The Build Right Now (v60.0.1 rebuild in flight — 2026-05-13)

• Source commit: bad81c9c (v60.1 sprint head — airootfs resurrection + chronic exit-1 fixes + pty teardown root cause + auto-monitor hook)
• Stage: 02-kernel in flight at time of writing (kernel compile, ~3hr)
• Launcher: master-generation-run-kit.sh --profile master --confirm-yes via setsid (NOT systemd-run; see F6)
• Log: growth/output/iso-build-logs/master-generation-master-20260513T175620Z.log
• Auto-spawned monitor: kitty terminal “synos-v60.1-rebuild RAW TAIL” (v60.1 hook in master-generation-run-kit.sh now does this for every build)
• /dev/pts: mounted (verified post-cleanup; F3 fix in place)
• Disk: 260 GB free (build/ was purged to 0 before launch)
• 71 GRIMOIRE labs: stubbed via generate-lab-stubs.sh (Dockerfiles can now build; lab content still v60.2)
• Expected v60.0.1 ISO produces:
  • 13 sbin tools (synos-health-check, synos-tailscale-bootstrap, synos-fido2-enroll, …, synos-tpm-seal) — first time shipping
  • 7 service units (boot-telemetry, firstboot-health-probe, etc.) — first time shipping
  • Q1 firstboot-health-probe JSON at /var/log/synos/firstboot-health.json — programmatic ship-grade verdict
  • Q2 persistent journald via /var/log/journal/
  • Stage 22 returning 0 cleanly (no more chronic exit-1)

Next major monitor event: kernel compile finish → stage 02b-synos-rust-modules start. Watch the auto-spawned kitty window for live tail.

---

Originally written autonomously by Claude Opus 4.7 during the v41 build (2026-04-25). Refreshed 2026-05-13 with v60.0.0 ship reality + the v42 → v60 failure mode catalog (§7.5). Verified against current code state in $HOME/Syn_OS/ (HEAD bad81c9c). For corrections or additions, edit in place — this doc is the working reference, not an immutable artifact.