# Development Guide How this repo works, how to add features, and what to keep in mind when editing. --- ## Architecture ### Installation Phases ``` install.sh │ ┌───────────┼───────────┐ ▼ ▼ ▼ preinstall.sh bin/* postinstall.sh (packages) → /usr/local/bin (config + services) ``` | Phase | Script | Responsibility | |-------|--------|----------------| | Pre | `preinstall.sh` | System packages, apt repos, global binaries (yt-dlp) | | Install | `install.sh` | Copies `bin/*` → `/usr/local/bin/` (chmod 755), all `lib/*.sh` → `/usr/local/bin/` (chmod 644) | | Post | `postinstall.sh` | User config (SSH keys, PATH, bash completion), systemd services | Each phase is independent and runs only if the corresponding script exists. ### Directory Layout | Directory | Purpose | Installed To | |-----------|---------|-------------| | `bin/` | Daily-use CLI tools and wrappers | `/usr/local/bin/` | | `apps//` | Optional desktop app installers | run on demand | | `lib/` | Shared libraries: `common.sh` (helpers), `flags.sh` (feature flags), `notify.sh` (multi-platform alerting), `registry.sh` (shared query API for POS tool metadata headers), `entertainment-lib.sh` (entertainment scheduling + last-run state), `entertainment-plugin-lib.sh` (message-safe plugin helpers), `scheduler-lib.sh` (system scheduler), `user-timers-lib.sh` (shared systemd user timer machinery), `config-ui.sh` (interactive config UI), `menu-lib.sh` (category-neutral menu primitives: guard/looping menu/filter picker/prompt), `share-lib.sh` (share-suite domain probes/listings + compat shims to menu-lib) | sourced at build time | | `config/` | Gitignored user config files | `~/.config//` (via postinstall) | | `entertainment/` | Public-API plugins for the entertainment module | `/usr/local/bin` (via install.sh Phase 2) | | `compose/` | ScaleTail templates (git submodule) | `/usr/local/share/linux_post_install/scale-tail` | | `systemd/` | Systemd unit files | `/etc/systemd/system/` (via postinstall) | ### The `pos` CLI `bin/pos` is a smart dispatcher. It scans its own directory for executable `pos-*` files and uses variable-length argument matching: ``` pos docker compose up jellyfin → tries pos-docker-compose-up-jellyfin (not found) → tries pos-docker-compose-up (not found) → finds pos-docker-compose → runs with args "up jellyfin" ``` All non-interactive commands log to `~/.local/share/linux_post_install/logs/`. `pos help ` shows a tool's help, e.g. `pos help communication telegram-sender` (all words joined with dashes → `pos-communication-telegram-sender --help`). `pos ` or `pos --help` shows a category's subcommands (derived from the `pos--*` filenames in `bin/` — no script execution, so it works even for root-only/interactive tools like `system-firewall`). **When adding a command, `bin/pos` itself has one thing to keep in sync:** - **The usage text** (`usage()` function) — the CATEGORIES block is **auto-derived** from the `pos-*` filenames in `bin/` (no manual edit, can't drift). The EXAMPLES block is the only hand-maintained part: add a line there only if you want the tool showcased in `pos --help`. - **`INTERACTIVE_CMDS`** (space-separated list above the dispatch loop) — commands that **read stdin** (password prompts, selection menus: `system-firewall`, `media-mp4`, `system-backup`, `share-usb-server`, `communication-telegram-listener`) must be added here. Everything else is piped through `tee` for logging, which would hang or swallow an interactive prompt. sudo's own password prompt is unaffected — it reads from `/dev/tty`. Trade-off: it's all-or-nothing **per script** — adding a flag-style tool with *any* prompting subcommand (e.g. `share-usb-server --share`) means *every* subcommand of that script skips output logging (e.g. `pos share usb server --ls` loses the `tee` log too). ### Shared Library (`lib/common.sh`) Sourced by most scripts. Key functions: | Function | Purpose | |----------|---------| | `log "msg"` | Green `[+]` status message | | `warn "msg"` | Yellow `[!]` warning | | `err "msg"` | Red `ERROR:` + exit 1 | | `ok "msg"` | Green `OK` prefix | | `section "title"` | Cyan-bordered section header | | `step N T "msg"` | Numbered step header | | `run cmd` | Executes command, respects `$DRY_RUN` | | `spawn "msg" cmd` | Animated braille spinner + elapsed time | | `timer_start` / `timer_stop` | Elapsed time tracking | | `confirm "prompt" [default]` | y/n prompt; Enter accepts the default (`y` when omitted) | --- ## Adding a New CLI Tool Start from the template: `cp templates/pos-tool.sh bin/pos-- && chmod +x bin/pos--`. ### 1. Create the script ```bash #!/usr/bin/env bash set -euo pipefail source "$(dirname "$0")/../lib/common.sh" usage() { cat < EOF exit 0 } case "${1:-}" in -h|--help|"") usage ;; esac # --- script logic --- ``` **Conventions:** - Shebang: `#!/usr/bin/env bash` - Strict mode: `set -euo pipefail` - `--help` flag: accept `-h` / `--help` via `case` pattern - **Deps guards run before `--help`:** `command -v &>/dev/null || err "… (install )"` lines sit at the top of the script, **before** the `-h|--help` case — so `--help` also errors when a dependency is missing. This matches every existing deps-gated tool; keep it that way. - Exception — tools with **no required deps** (every check degrades gracefully): `pos system health` probes binaries at runtime (`if command -v systemctl; then …`) and needs no guard. The lint (`scripts/lint-conventions.sh`) only enforces guard-before-help for lines that are actual guards (`command -v … ||`, `if ! command -v`, `command -v … \` continuation), never for graceful-degradation probes. If you add a tool like this, keep all checks optional and note it in `usage()`. - Shared library: always source `common.sh` for colors, logging, spinners - Exit codes: `0` success, `1` error - No shared lib? Inline fallbacks: ```bash log() { echo "[+] $*"; } warn() { echo "[!] $*"; } err() { echo "ERROR: $*" >&2; exit 1; } ``` If you skip `common.sh`, `make gen` adds the tool to the "Scripts that do NOT source common.sh" list in `DOC/AGENT_Context_Project.md` automatically. ### 2. Make it discoverable - The dispatcher auto-discovers executable `bin/pos-*` files — no registration needed. The file **must be executable** (`chmod +x`, committed as mode `100755`); the dispatcher and `install.sh` skip non-executables. - `pos --help` (and bare `pos `) is derived from the `pos--*` filenames too — a new tool appears in its category's help automatically, with no registration (see [The `pos` CLI](#the-pos-cli)). - **Add the `# POS:` header** (single source of truth for the docs) right after the shebang/strict-mode lines: ```bash # POS: — one-line description rendered by `make gen` # POS_FLAGS: --flag1 --flag2 # ONLY for flag-style tools # POS_SUBCMDS: sub1 sub2 # ONLY for multi-command tools # POS_DEPS: binary1 binary2 # Optional: runtime deps (space-separated binary names) # POS_EXAMPLES: pos | Description # Optional: usage examples ``` The description feeds the dispatch table, bin tree and file table in `DOC/AGENT_Context_Project.md`; `POS_FLAGS` feeds flag completion and `POS_SUBCMDS` feeds subcommand completion in `completions/pos.bash` (both update via `make gen`). `POS_DEPS` lists runtime binary names that `command -v` would check — use when the tool requires specific binaries beyond what `preinstall.sh` installs. `POS_EXAMPLES` provides curated usage examples (one per line, pipe-delimited `command | description`) shown in `pos tree` and future help views. Both are optional and degrade gracefully when absent. `make gen` only reads the text after the first `— ` — the ` ` words before it are convention-only (for nested tools, keep the full path there, e.g. `# POS: communication telegram-listener — …`). - **Category-less vs categorized:** most tools are `bin/pos--`. Use category-less `bin/pos-` (e.g. `pos-config`, `pos-tree`) only for dispatcher/dev-level commands that fit no category — they dispatch and document like any tool but show with an empty category in the generated tables. - Nested tools (e.g. `bin/pos-communication-telegram-listener`) are auto-detected from filenames: the trailing segment (`listener`) is offered as a subcommand of the parent tool (`communication-telegram`) in `pos --help` and tab-completion, instead of appearing as a flat sibling (`telegram-listener`). The flat dash-form (`pos communication telegram-listener`) still dispatches. - Optionally add an EXAMPLES line in `bin/pos` `usage()` to showcase the tool in `pos --help`. - If the command **reads stdin** (prompts/selection), add it to `INTERACTIVE_CMDS` in `bin/pos` — see [The `pos` CLI](#the-pos-cli). ### 3. Add system dependencies Add package names to the `PACKAGES` array in `preinstall.sh`: ```bash PACKAGES=( ... your-package ) ``` **Not in apt?** If the dependency ships as a manual installer (no package — e.g. `usbsrv`, the USB Redirector server), do **not** put it in `PACKAGES` (that would break `preinstall.sh`). Instead, add a `command -v || err "… install from "` guard in the tool itself and note the manual install in `usage()`/`DOC/POS.md`. ### 4. Config files (if needed) Two kinds of config, don't mix them up: - **Machine defaults shipped by the installer:** place the file in `config/` and add copy logic to `postinstall.sh`. If it contains secrets, add to `.gitignore` and document in `DOC/`. - **Runtime tool config set by the user:** `~/.config/linux_post_install/.env` with `chmod 600`. Load it with env-var precedence (flags > environment > file). Patterns: `pos-docker-compose` (`compose.env`), `pos-communication-telegram-sender` (`telegram.env`, edited via `pos config telegram` — token masked), and the shared ones below. Never store tokens in the repo. - **Canonical loader (new tools, D-D):** `load_env_file ` in `lib/config-ui.sh` — the one shared KEY=VALUE loader (reads `^[A-Z_]+=` lines, skips blank + `#` comments, strips trailing CR and one pair of quotes, exports a key only when the env var is not already set → env always wins over the file; loaded keys are appended to the global `LOADED_ENV_KEYS`). A bare basename resolves under the XDG-aware `CONFIG_DIR`; a full path is used as-is. Missing/unreadable file = quiet no-op; the loader never creates files and never chmods (chmod-600 stays with `cfg_write`/the tool's own writers). Source it with the standard fallback chain: `source "$(dirname "$0")/../lib/config-ui.sh" 2>/dev/null || source "$(dirname "$0")/config-ui.sh"`. Legacy `load_system_env()` in `lib/common.sh` is functionally identical and stays for its existing three callers (`pos system health`, `pos system backup`, `pos media sync`) — new tools should use `load_env_file`. One deliberate exception: `pos-docker-compose`'s `load_global_config` still `source`s `compose.env`, because that file is shell-executed (variable expansion, file-beats-defaults layering, unquoted `config set` writes) — don't convert it to `load_env_file`. - `system.env` — shared "system" settings loaded via `load_system_env()` in `lib/common.sh` (currently `BACKUP_SERVICE_ROOTS`, `BACKUP_USB_ROOT`, `BACKUP_MOUNT_BASE`, `BACKUP_USB_BYID`, `HEALTH_BACKUP_MAX_AGE_DAYS`, plus `USB_MOUNT_BASE`/`USB_BYID`/`MEDIA_SYNC_SOURCE`/`MEDIA_SYNC_DEST` for `pos media sync`). Env already exported wins over the file. - `notify.env` — alerting platform selection (`NOTIFY_PLATFORM=telegram,matrix`), read by `lib/notify.sh`. ### 5. Add SSH keys (if needed) Place public keys in `config/authorized_keys` (one per line). `postinstall.sh` reads this file automatically. ### 6. Update the docs - `DOC/POS.md`: add the command to the section table + a detail block (commands, behavior, configuration). This is the one hand-written doc. - `DOC/HOWTO.md` index row + a hands-on section in `DOC/howto/.md` (recipes + troubleshooting) for user-facing tools. - `DOC/AGENT_Context_Project.md` generated sections (bin tree, dispatch table, no-common.sh list, line-count table) and the `completions/pos.bash` flags block are produced by `make gen` — do **not** hand-edit between the `GEN:START`/`GEN:END` markers. Hand-maintained, not gen-checked: the line-count rows **above** the filetable marker (non-`pos-*` files only — bump a row's count when that file's length changes) and the "Common Tasks for Agents" table (add a row for the new tool). - `AGENTS.md` Quick facts: update if a structural fact changed (new category, new convention). - Root `README.md`: only if the `pos` category list in the help text changes. - Move the finished task to the `Done` section of `AGENT_TODO.md` (dated) in the same commit. ### 7. Test ```bash chmod +x bin/your-tool bash -n bin/your-tool shellcheck bin/your-tool ./bin/your-tool --help env-seam review: every path the tool writes must be `VAR="${VAR:-default}"`-guarded — grep for config writes without a `:-` guard: `grep -nE '>\s*(\$HOME|/etc)' bin/your-tool lib/your-lib` (each hit needs the seam; prove it with `VAR=/tmp/x bin/your-tool …` + assert the real path is untouched) bin/pos help # confirm dispatch works bin/pos --help # confirm category listing includes the new tool (first tool in a new category) make gen # regenerate doc tables + completion flags make check # full self-consistency gate (syntax, exec bits, doc/code sync, smoke) make lint # convention gate (scripts/lint-conventions.sh) — must end 0 FAIL, 0 WARN ``` `make check` + `make lint` (0 FAIL / 0 WARN) are the definition of done. `make check` is also run as a pre-commit hook once you've run `make hook`; `make lint` is not part of the hook — run it yourself. Pushing to Gitea re-runs all four gates on the live Actions runner (see `CI: Gitea Actions Gate` below) — a red run is a merge-blocker. ### Testing tools that need root / systemd / missing deps `make check` only proves syntax, exec bits, doc sync and dispatch — not behaviour. For tools that need `sudo`, systemd, or binaries absent from the dev box (samba, usbsrv, …), test them end-to-end with two patterns: - **Env-overridable paths.** Anything that touches a system config location gets an env override whose default is the real path — the seam that lets the tool be exercised against temp files. Precedents: `FLAGS_DIR` (`lib/flags.sh`), `SMB_CONF` (`bin/pos-share-smb-server`, default `/etc/samba/smb.conf`), `EXPORTS_FILE` (`bin/pos-share-nfs-server`, default `/etc/exports`), `SMB_CREDS_DIR`/`UNIT_DIR` (`bin/pos-share-smb-client`, and `UNIT_DIR` again in `bin/pos-share-nfs-client`), `USER_SYSTEMD_DIR` (`bin/pos-network-download`, `bin/pos-communication-{telegram,matrix}-listener`, `lib/scheduler-lib.sh` — write it as `${USER_SYSTEMD_DIR:-${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user}`), the scheduler's `SCHEDULE_DIR`/`SCHEDULE_STATE_DIR`/`SCHEDULE_LOG_DIR`/`SCHED_LEGACY_ENV` (`lib/scheduler-lib.sh`), and the USB layer shared by `pos system backup` + `pos media sync`: `USB_MOUNT_BASE` (default `/media`, keeps `BACKUP_MOUNT_BASE` as an alias) and `USB_BYID` (default `/dev/disk/by-id`, keeps `BACKUP_USB_BYID` as an alias) — both guarded as `VAR="${VAR:-${BACKUP_…:-default}}"` in `lib/usb-lib.sh` so existing config lines keep working. Pick a short tool-specific name and don't advertise it in `usage()` — it's a test seam, not user-facing. **Gotcha (session-learned):** a `VAR="${XDG…:-…}"` without the leading `VAR:-` *overrides* the seam — the stub run then silently writes to the real `$HOME` path and every assertion passes while the bug hides. The override must be written first, then tested with `VAR=/tmp/x …` and a check that the real path is untouched. - **Stub PATH.** Create a temp dir with fake binaries, then run the tool with `PATH="$stubs:$PATH"`: fake `sudo` → `exec "$@"`; fake `systemctl`/`smbcontrol`/`mount.cifs` → echo their args; fake `testparm` → `cat` the file back (so validation passes); fake `systemd-escape` → print a fixed name. Assert on output **and** exit codes — happy path plus each failure path (`err` sets rc=1). - **Interactive prompts** (`read … -test/`: `stubs/` + `run-tests.sh` with a `check "desc" "expected" "$actual"` helper and a pass/fail count), run them, then leave them in `/tmp`. Only the *pattern* above is worth keeping in the repo. (CI — `.gitea/workflows/lint.yml`, live act_runner — runs the *static* gates `make gen`+`git diff --exit-code`/`make check`/`make lint` on every push/PR; it does not run behaviour suites.) **Stub-harness gotchas (session-learned, `pos system backup` USB-detection suite).** Each red check means exactly one assumption — in the tool *or* the harness — is wrong; keep the diagnosis cheap by copying the run's `out.log` to a per-test file and asserting on artifacts (`fake_state`, `.gpg` on disk, `sends.log`, exit code), then deciding which side lied: - **Layout: the harness must live *outside* the sandbox it wipes.** If `fresh()` does `rm -rf "$TEST_DIR"`, the runner and the stubs cannot live inside `$TEST_DIR` or they get deleted mid-run. Prefer siblings: `/tmp/opencode/-run.sh` + `/tmp/opencode/-stubs/` + `/tmp/opencode/-test/` (sandbox wiped per test). - **Field separators: never `IFS=$'\t'` on JSON-derived data.** `read` treats IFS *whitespace* specially and collapses consecutive delimiters, so an empty JSON field (null `tran`, null `mountpoint`) shifts every following column and detection silently misfires. Emit a non-whitespace separator from jq (`… | join("\u001f")`) and read with `IFS=$'\x1f'`. - **`!` is not a command: `check "x" "$@"` with `! grep …` runs `!` as a binary (rc 127).** For "nothing present" prefer `test -z "$(grep … )"` — note `grep -qv` on an *empty* file still exits 1 (zero lines selected), so it fails the "nothing was written" case. - **`read -rp` prompts vanish when stdin is a pipe.** Bash suppresses the prompt text when stdin isn't a tty, so never assert on prompt strings in piped runs — assert on the side effect. - **Fakes that shell out must call the *real* binary, not the stub on PATH.** A fake `gpg` that ran `cp` picked up the corrupting fake `cp` and broke the tool's encryption step instead of the USB-copy step under test. Resolve the real one: `real() { for d in /usr/bin /bin; do [ -x "$d/$1" ] && { printf '%s' "$d/$1"; return; }; done; }` then `"$(real cp)" …`. - **Scope failure-injection fakes, don't make them global.** A "chmod fails" flag hit *every* `chmod` the tool runs (it chmods the archive too) and aborted the run before the section under test. Match the target instead: only fail for args under the seam path (e.g. `[[ "$target" == "$BACKUP_MOUNT_BASE/"* ]]`). - **Fixtures must genuinely exercise the branch.** A fixture builder that hardcodes `"tran":"usb"` means the "TRAN empty → cross-check" path never runs and its test passes for the wrong reason. Check the discriminating field actually varies (add a `_notran` builder, a card-reader `sata` fixture, etc.). --- ## Adding an Entertainment Plugin The `entertainment` module routes public-API data to the configured notify platforms via the single runner `pos entertainment send ` (`bin/pos-entertainment-send`). Auto-triggering is config-driven: `ENABLED` in `entertainment.env` holds `plugin, interval` pairs; the tools `pos entertainment config|enable|disable|status` (`bin/pos-entertainment-*`) reconcile the schedule. Shared logic (ENABLED parsing, plugin lookup, last-run state, scheduler sync) lives in `lib/entertainment-lib.sh` — sourced by the `pos-entertainment-*` tools (never by plugins). The scheduler is **systemd user timers** — the only backend (requires a reachable user systemd manager); the timer machinery itself is shared with the system scheduler via `lib/user-timers-lib.sh`. ### 1. Create the plugin Drop an executable script in `entertainment/.sh` with a `# POS_PLUGIN: ` marker (this is what makes it a plugin — the installed runner lists plugins by this marker, not by `.sh` files, since `/usr/local/bin` is shared with other tooling). Declare every config key the plugin reads with `# POS_KEYS: (required|optional)` lines right after it — `pos entertainment config` prints these in its Keys section and uses them to warn/not-warn on `config set`. Plugins may source `lib/entertainment-plugin-lib.sh` — message-safe helpers (config load, dep guards, JSON fetch with retry) that never write to stdout. Template: ```bash #!/usr/bin/env bash set -euo pipefail # POS_PLUGIN: myplugin # POS_KEYS: MYPLUGIN_URL (required) # POS_KEYS: MYPLUGIN_TAG (optional) source "$(dirname "${BASH_SOURCE[0]}")/../lib/entertainment-plugin-lib.sh" 2>/dev/null \ || source "$(dirname "${BASH_SOURCE[0]}")/entertainment-plugin-lib.sh" 2>/dev/null \ || source "$(dirname "$0")/../lib/entertainment-plugin-lib.sh" 2>/dev/null \ || source "$(dirname "$0")/entertainment-plugin-lib.sh" plugin_load_config plugin_require MYPLUGIN_URL tag="${MYPLUGIN_TAG:-}" json="$(plugin_http_json --key '.title' "${MYPLUGIN_URL}${tag:+/?tag=$tag}")" printf 'Title: %s\n' "$json" ``` **Contract:** plugins are **self-contained** — do **not** source `lib/common.sh` or `lib/entertainment-lib.sh`. Their `log`/`warn`/`ok` helpers print to **stdout**, and the runner captures stdout as the message to send (helper chatter would be sent to the notify platforms). All stdout is the message; errors go to stderr and exit nonzero. `lib/entertainment-plugin-lib.sh` is the *only* lib plugins may source (it defines just `plugin_*` and writes nothing to stdout). Plugins must be non-interactive (no prompts) — the module is designed for systemd user timers. ### 2. Config (if needed) Read runtime values from `~/.config/linux_post_install/entertainment.env` (chmod 600, env precedence) via `plugin_load_config` — same pattern as `telegram.env`. Example: `weather.sh` uses `WEATHER_LAT`/`WEATHER_LON`. Declare each key with a `# POS_KEYS:` header line (see step 1) so `pos entertainment config` lists it and `config set` recognizes it. ### 3. Deps `curl` and `jq` are already in `preinstall.sh` PACKAGES (and are what `plugin_have curl`/`plugin_have jq` check). Anything else: guard with `plugin_have ` and, if apt-available, add to PACKAGES. ### 4. Done Plugins are not `pos-*` tools, so `make gen`/`make check` don't scan them — verify with `bash -n entertainment/.sh` and a live `pos entertainment send --print` run. Document the plugin in `DOC/POS.md`'s entertainment plugin table. --- ## Adding an Optional App Start from the template: `cp templates/app.sh apps//.sh`. ### 1. Create the installer ```bash #!/usr/bin/env bash set -euo pipefail source "$(dirname "$0")/../../lib/common.sh" install_myapp() { command -v myapp &>/dev/null && { log "myapp already installed"; return 0; } spawn "Installing myapp" sudo apt install -y myapp } uninstall_myapp() { command -v myapp &>/dev/null || { log "myapp not installed"; return 0; } spawn "Removing myapp" sudo apt purge -y myapp spawn "Cleaning up dependencies" sudo apt autoremove -y } case "${1:-}" in uninstall) uninstall_myapp ;; *) install_myapp ;; esac ``` Place it in `apps//.sh`. It auto-appears in the picker — no registration needed. **Categories:** `browsers`, `development`, `media`, `networking`, `remote-access`, `system`, `utilities` **Docs:** add a row to the catalog table in `DOC/APPS.md` (name, category, purpose, install method). ### 2. Conventions - Idempotent: check `command -v` (or `flatpak list` / file existence) before installing **and** uninstalling - Every app **must** provide an `uninstall_()` function and dispatch on `uninstall` via the `case` above — `apps/install.sh --uninstall` depends on it - APT packages → `sudo apt install -y` inside `spawn`, remove with `sudo apt purge -y` + `sudo apt autoremove -y` - Repo-based apps (apt repo added at install) → also remove the `.list` file and keyring in uninstall - Official scripts → `curl ... | sh` inside `spawn` - Flatpak → `flatpak install -y flathub ` inside `spawn`, remove with `flatpak uninstall -y ` - `.deb` files → download to temp, `sudo apt install -y ./file.deb` inside `spawn` - File/AppImage installs → remove the installed files, symlinks, and desktop entries in uninstall - `usermod` for groups → print re-login reminder --- ## Editing an Existing Tool 1. Find the script in `bin/` 2. Understand its contract (args, output, exit codes) 3. Make the change — keep it idempotent 4. Update `DOC/POS.md` (or the relevant doc) if behaviour changed 5. Run `shellcheck` on the modified file --- ## Convention Lint Gate `scripts/lint-conventions.sh` is the automated convention gate — it encodes the rules in this document so drift is caught by the machine, not the next audit. Run it with `make lint` (or `./scripts/lint-conventions.sh`). FAIL = definite violation (fix it before committing), WARN = manual review needed. Exit code is non-zero when any FAIL exists. Check classes (all heuristic-based; heredocs, `${...}` brace-counting, `while` loop stdin, `/dev/tty` reads and `command -v` fallbacks are excluded): - **Shebang / strict mode** (FAIL) — every shell file (`bin/*`, `install.sh`, `preinstall.sh`, `postinstall.sh`, `features/*`, `apps/*`, `templates/*`, `scripts/*`) starts with `#!/usr/bin/env bash` and has `set -euo pipefail` (libs are sourced, so they're exempt). - **Exec bits** (FAIL) — `bin/pos-*` and `entertainment/*.sh` committed as `100755` (`chmod +x`). - **`# POS:` header** (FAIL) — every `bin/pos-*` carries it with the em-dash separator (`# POS: — `); a header past line ~6 is a WARN. - **`-h|--help`** (FAIL) — every `bin/pos-*` handles it via `case`. - **Deps guards before help** (FAIL) — the first real guard (`command -v X … || err`, `if ! command -v X …`, multi-line `\` continuation) must sit before the `-h|--help` dispatch, so help errors on a box missing the dependency. Graceful-degradation probes (`if command -v X; then …`) are not guards. - **Top-level `local`** (WARN) — `local` at brace-depth 0 outside a function is invalid bash. - **stdin ⇄ `INTERACTIVE_CMDS`** (FAIL) — a tool that reads stdin must be in `INTERACTIVE_CMDS` in `bin/pos` (else the logging tee swallows/hangs the prompt); every entry must also have a matching `bin/pos-` tool. - **DOC/POS.md coverage** (WARN) — each `bin/pos-*` referenced in `DOC/POS.md`. - **Entertainment plugins** (FAIL) — must carry `# POS_PLUGIN:` and must NOT source `lib/common.sh` (stdout is the message). - **Apps** (FAIL) — each `apps/*` script has `uninstall_()` and an `uninstall` dispatch case. - **Systemd units** (WARN) — `TimeoutStopSec=` and `[Install] WantedBy=`. - **Legacy wrappers** (FAIL/WARN) — `bin/wr-*`, `mp3`, `mp4`, `vbox`, `ssh-load-all` must forward to `pos` (FAIL if not); >12 lines or a `case` statement is a WARN (thin forwarder only). - **Secrets** (WARN) — literal `…TOKEN=/…SECRET=/…KEY=…` assignments are flagged for manual review (env guards, config reads and runtime generation are excluded). - **Env seams** (WARN) — writes to `/etc/`, `$HOME`, `/usr/local` are flagged unless guarded (`command -v` or `|| echo`), i.e. the write needs a `VAR="${VAR:-path}"` test seam. If a rule is genuinely wrong for a new case (as happened with graceful-degradation probes in system-health), refine the heuristic — never weaken it — and note the change in `MAINTENANCE.md`'s lint section. --- ## CI: Gitea Actions Gate `.gitea/workflows/lint.yml` re-runs the four gates on every `push` and `pull_request`: `make gen`, `git diff --exit-code` (gen drift), `make check`, `make lint`. A red run is a merge-blocker; runs are visible under Gitea → Actions. - **Runner** — act_runner v0.6.1 (`linux-post-install`, labels `ubuntu-latest` → job image `node:20-bullseye`) is registered on the Gitea host and always on: compose project `~/srv/gitea/runner/` (`docker compose up -d`, `restart: unless-stopped`), standalone next to the ScaleTail gitea compose. - **Gotchas (session-learned):** - act_runner's `run.sh` `cd`s into `/data` and only reads the config when the `CONFIG_FILE` env var is set — the compose service must pass `CONFIG_FILE=/config.yaml`, not just mount the file. - The job container can't resolve `gitea.skink-platy.ts.net` by itself; pin it with `container.options: "--add-host gitea.skink-platy.ts.net:100.111.241.54"` in `config.yaml`. - Registration tokens are one-time use; the token lives in `runner/.env` (chmod 600) and is burned after the first registration. - Inspecting runs via sqlite: Gitea's status enum is runnerv1-consistent — **1 = success, 2 = failure** (not the old 0/1/2/3 scheme). - **Deterministic generators** — any script whose output is committed (gen docs, completions) must sort in byte order: plain `sort` collates differently per locale, and the CI container tripped exactly this (category-less tool keys like `pos-config` start with `|`, which collated after letters under that locale, reordering the generated tables). `scripts/gen-docs.sh` sets `export LC_ALL=C`; keep that in mind for any new generator. - **Checking green without SSH** — the workflow reports its own outcome as a lightweight git tag: `ci-ok/` on success, `ci-fail/` on failure (pushed with the job's automatic `GITEA_TOKEN`; the workflow only triggers on `push` to `main`, so tag pushes don't re-trigger it). Check from the dev box with plain git — `scripts/ci-status.sh [--wait] []` (reads the tags via `git ls-remote`, exit 0/1/2 = green/red/pending). No SSH to the runner, no API tokens. - **Limits** — CI proves the *static* gates only; it never runs behaviour suites (stub harnesses stay throwaway in `/tmp`). --- ## Best Practices ### Alerting To notify on events, source the shared helper instead of calling a platform tool directly: ```bash source "$(dirname "$0")/../lib/notify.sh" 2>/dev/null || source "$(dirname "$0")/notify.sh" notify_send "Backup completed" notify_send "**disk full**" --markdown ``` `notify_send` is deliberately dependency-free (defines only itself, so it never clobbers a tool's own `log`/`warn`/`err`) and **silent-fails**: if no platform is configured it warns and returns 0, never breaking the caller's flow or exit code. Source it opt-in in any tool that should alert; for failure alerts use `trap 'notify_send "..." ERR'`. **Multi-platform routing:** `notify_send` delivers to every platform listed in `NOTIFY_PLATFORM` (env or `~/.config/linux_post_install/notify.env`, default `telegram`, comma-separated to send to all). Adding a new platform (e.g. Matrix/Synapse) means creating a `bin/pos-communication-` tool that implements the **sender contract**: ```bash pos-communication- send [--markdown] # exit 0 on delivery ``` then listing it in `NOTIFY_PLATFORM`. Platform keys map to tool names via `notify_sender_name()` in `lib/notify.sh` — the telegram platform key stays `telegram` but its tool is `pos-communication-telegram-sender`. `pos-communication-telegram-sender` already follows this (`--markdown` is an alias for `--parse-mode markdown`). No changes to `lib/notify.sh` are needed for a new platform. ### Confirmation prompts `confirm()` rule: Enter accepts the displayed default; destructive call sites pass explicit `'n'`. ### Idempotency Check before creating, use `>>` with grep guards, don't overwrite user configs. ### Systemd units Every unit a tool writes (or `systemd/` ships) sets `TimeoutStopSec=5s` so a stuck process can't stall a reboot for the 90s systemd default. Long-polling daemons (listeners) also `trap TERM INT` in their loop so a stop returns in well under a second — the unit timeout is the backstop. Keep `KillMode=` explicit (`control-group`) on the daemons. A oneshot job running at shutdown is SIGKILLed 5s after stop begins — fine, `Persistent` timers re-run it next boot. Existing installs keep the old unit files until the tool rewrites them (re-run the `enable` path), so template changes need a regeneration step on live boxes. ### Managed Config Blocks To let a tool own a slice of a user/system config file (e.g. Samba shares in `/etc/samba/smb.conf`) without clobbering hand edits, delimit the tool's section with start/end marker lines and rewrite only that slice: ``` # >>> pos-managed share: [media] path = /mnt/hdd # <<< end pos-managed share ``` - Idempotent upsert: one `awk` pass drops the existing block (or nothing if absent), then append the new block; removal uses the same `awk` with only the slice dropped. - The block-deletion guard matters: `$0 == s {inblock=1}` … `$0 == e && inblock == 1 {inblock=0; next}` — without the `inblock == 1` check, deleting one block also eats the end markers of other blocks further down the file. - Validate before writing: run the config's own checker on a temp copy (`testparm -s` for Samba), then apply with `sudo cp`; hot-reload instead of restarting (`smbcontrol smbd reload-config`). - Precedent: `bin/pos-share-smb-server`. ### Error Handling ```bash set -euo pipefail command -v docker &>/dev/null || { echo "docker not found"; exit 1; } [[ -n "${1:-}" ]] || { echo "Usage: my-tool "; exit 1; } ``` ### Portability Targets **Debian** and **Ubuntu**. Use `apt`, assume bash at `/usr/bin/env bash`, check tools with `command -v`. ### Dry-run Support Scripts support `--dry-run`. Use the `run()` helper: ```bash run() { if [ "$DRY_RUN" -eq 1 ]; then log "(dry-run) $*" else "$@" fi } run sudo apt install -y git ``` ### Security - Never hardcode secrets — put them in `config/` (gitignored) or, for runtime tool config, `~/.config/linux_post_install/.env` - `chmod 600` for sensitive files - Mask secrets in `config` output (see `pos-communication-telegram-sender`'s `mask_token`) - Validate input before shell commands - Use `sudo` only where needed ### Naming - CLI tools: `bin/pos--` - Legacy wrappers: `bin/wr-*` - App installers: `apps//.sh` - Features: `features/.sh` - Lowercase with hyphens --- ## Features & Flags `features/` holds scripts the user is likely to customize (e.g. `autostart.sh`). Unlike `bin/` (synced on every install), features are installed on demand and **never overwritten without asking**. ### Adding a Feature Start from the template: `cp templates/feature.sh features/.sh`. 1. Create `features/.sh` following the CLI tool template (shebang, `set -euo pipefail`, `--help`). 2. Nothing else is registered — `./install.sh --feature` auto-discovers it, copies it to `/usr/local/bin/`, asks before overwriting an existing file, and sets its flag. 3. If the feature backs a systemd service, gate the service on the flag in `postinstall.sh` (see below). ### Flag System System-wide flag store at `/usr/local/share/linux_post_install/flags/` (presence = set, content = optional value). Sourced via `lib/flags.sh` (or the installed `/usr/local/bin/flags.sh`): ```bash source "$(dirname "$0")/lib/flags.sh" 2>/dev/null || source "$(dirname "$0")/flags.sh" flag_set autostart # green flag flag_set app "2.1" # green flag with a value flag_is_set autostart # test (0/1) — the primitive consumers use flag_value app # → "2.1" flag_list # names of all set flags flag_clear autostart ``` Writes use `run` + `sudo`, so they respect `--dry-run`. CLI equivalents: `flag-reader`, `flag-set`, `flag-clear`. **Example — service gated on a flag** (in `postinstall.sh`'s systemd loop): ```bash if [ "$svc_name" = "myapp.service" ] && ! flag_is_set myapp; then warn "myapp feature not installed — skipping myapp.service" continue fi ``` --- ## Working with Systemd Create `systemd/.service` — `postinstall.sh` copies it to `/etc/systemd/system/` and enables it automatically. ```ini [Unit] Description=My Service After=network-online.target Wants=network-online.target [Service] Type=simple ExecStart=/usr/local/bin/your-script.sh Restart=on-failure RestartSec=10 [Install] WantedBy=multi-user.target ``` --- ## Working with Config Files 1. Place the file in `config/` 2. Add copy logic to `postinstall.sh`: ```bash if [ -f config/your-config.conf ]; then mkdir -p "$HOME/.config/your-app" cp config/your-config.conf "$HOME/.config/your-app/your-config.conf" chmod 600 "$HOME/.config/your-app/your-config.conf" fi ``` --- ## Docker Compose / ScaleTail The installer clones [ScaleTail](https://github.com/tailscale-dev/ScaleTail) templates to `/usr/local/share/linux_post_install/scale-tail/` — 119+ self-hosted services with a Tailscale sidecar pattern. Each service gets a `tail-xxxxx.ts.net` URL via `network_mode: service:tailscale`. ### Config Strategy — Three Layers Values cascade from least to most specific: ``` Template .env (per-service defaults from ScaleTail) ↓ Global config (~/.config/linux_post_install/compose.env) ↓ Per-service .env (/srv//.env) — created on first deploy, NEVER overwritten ``` On first `pos docker compose up `: 1. Template `.env` is copied to `/srv//.env` 2. Matching keys from global config are filled in 3. If `TS_AUTHKEY` is still empty, you're prompted to enter it 4. After that, the per-service `.env` is **never touched** — not even by `update` ### Layout | Path | Purpose | Mutability | |------|---------|------------| | `/usr/local/share/linux_post_install/scale-tail/services//` | ScaleTail templates (git repo) | Read-only | | `~/.config/linux_post_install/compose.env` | Your global defaults | Edit via `config set` or `config edit` | | `/srv//` | Active deployment | Per-service `.env` preserved forever | ### Key Commands | Command | Behaviour | |---------|-----------| | `pos docker compose up ` | Deploys to `$SERVICES_BASE//`, creates `config/` + `data/`, generates `.env` from global defaults | | `pos docker compose down ` | Stops the stack | | `pos docker compose update` | `git pull` templates + refreshes `compose.yaml` for all deployed services (`.env` untouched) | | `pos docker compose config set K=V` | Sets a global default in `~/.config/linux_post_install/compose.env` | | `pos docker compose config show` | Displays current global config and `SERVICES_BASE` | | `pos docker compose config edit` | Opens global config in `$EDITOR` | ### Global Config Keys | Key | Required | Default | Purpose | |-----|----------|---------|---------| | `TS_AUTHKEY` | Yes | — | Tailscale auth key for sidecar networking | | `TZ` | No | `Europe/Amsterdam` | Timezone for services | | `DNS_SERVER` | No | `9.9.9.9` | Custom DNS server | | `SERVICES_BASE` | No | `/srv` | Root directory for all deployments | --- ## Commit Guidelines - Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:` - Explain *why*, not just *what* - One logical change per commit ``` feat: add pos-disk-usage for monitoring disk space fix: pos-network-ip fails when no default route exists docs: add example output for pos-network-scan ``` --- ## Useful Commands ```bash # Syntax check a single script bash -n bin/my-script # ShellCheck linting shellcheck bin/my-script # Check all scripts for f in bin/* apps/*/*.sh lib/common.sh install.sh preinstall.sh postinstall.sh; do bash -n "$f" || echo "FAIL: $f" done # Init submodule git submodule update --init # Pull latest ScaleTail templates git submodule update --remote compose/scale-tail # Test install in Docker docker run --rm -it -v $PWD:/repo ubuntu:22.04 bash # inside: cd /repo && ./install.sh # Test app installers ./apps/install.sh ./apps/install.sh --all ./apps/install.sh docker vscode ```